When an auditor asks for a document, knowing it’s somewhere in the system doesn’t get you very far. You need to trace the AI systems in use today, understand which requirements apply to them, and have the evidence to back it up.  

The approvals, assessments, and monitoring records may all exist, yet still fail to form a clear audit trail. This is where the difference between being HIPAA-compliant in principle and being able to demonstrate AI compliance in practice becomes important. 

Is Your AI Use HIPAA-Compliant? 

HIPAA doesn’t have a separate set of rules for artificial intelligence. It has rules for protecting protected health information (PHI), and those rules apply regardless of the type of system creating, receiving, maintaining, or transmitting that information.  

The first thing to look at is whether the AI system is working with PHI. If it does, the next question is which HIPAA compliant AI requirements apply to that specific use. 

It starts with knowing who is handling the PHI. If a third party creates, receives, maintains, or transmits PHI on behalf of a covered entity, it generally falls under the definition of a business associate under 45 CFR 160.103.  

HIPAA requires a written business associate agreement (BAA) before the business associate can access PHI on the covered entity’s behalf. This can include AI vendors whose systems process PHI on an organization’s behalf, such as ambient documentation, coding assistance, or prior authorization tools. 

The AI vendor may not be the only party involved. Some products rely on separate companies for the underlying foundation model or other services. 

These providers may qualify as subcontractor business associates when they handle PHI on the vendor’s behalf. A BAA is an important requirement, but using a HIPAA compliant AI solution requires more than a BAA alone. AI governance requires a clear view of how PHI is handled in the AI workflows. 

Become Audit-Ready and AI-Compliant 

There’s a huge difference between being compliant and being able to demonstrate AI compliance. A lot of the anxiety around AI audits comes from confusing the two. 

Compliance starts with having the policies and controls required by HIPAA and other state level requirements. The organization also needs records showing those controls were put in place and reviewed.  

Provide a clean trail of evidence to an auditor. 

Requirement → Control → Owner → Evidence → Review → Remediation 

 
It needs to specify the responsible parties, the controls in place, what was reviewed, and how any gaps were addressed. Have these questions handy to make sure you’re audit-ready. 

• How current is the system documentation?  

• Where does this AI system appear in your inventory? 

• Who owns the system and its compliance? 

• What evidence supports the controls in place? 

• Can decisions be traced to the AI version, input, and reviewer? 

• Which vendor evidence is on file?  

• How are policy exceptions documented?  

• Where do previously identified gaps stand?  

Create a Complete Inventory of Your AI Systems 

An AI inventory tracks the AI systems in use and shows how they’re being used across the organization.  

It matters for compliance because you can’t assess or govern systems you don’t know are there. With an inventory in place, compliance and audit teams have a starting point for identifying applicable requirements, checking controls, and gathering the right evidence.  

This list shows the different ways AI can enter or exist within a healthcare organization. 

• Generative AI used for ambient documentation or content drafting. 

• AI embedded in the EHR through predictive features. 

• AI capabilities included in existing vendor contracts. 

• Patient-facing tools such as chatbots and symptom checkers. 

• Administrative tools used for revenue cycle or prior authorization. 

• Tools introduced by employees using personal accounts for work. 

• AI features added through routine software updates. 

• Shadow AI that bypasses the organization’s review process. 

Each system should have enough information to answer an auditor’s questions without needing another meeting. 

The inventory needs enough information, like: 

• AI system and intended use to provide clarity on what you’re assessing. 

• Owner to establish clear accountability. 

• Data accessed to identify the relevant compliance considerations. 

• Applicable requirements to show which obligations need to be checked. 

• Approval status to confirm the system went through the required review. 

• Model or version for traceability to the system in use. 

• Change history to track what happened after approval. 

• Monitoring status to provide evidence of ongoing oversight. 

Note: Shadow AI can be especially easy to miss because it may never go through the organization’s usual review process. 

Address State-Specific AI Requirements 

State requirements can change, and their scope can differ by use case. Colorado provides a good example. Its 2024 AI law, SB 24-205, was later replaced by SB 26-189 in 2026. Colorado also enacted separate healthcare-specific requirements for certain uses of AI in utilization review and psychotherapy. 

AI requirements for each state sit alongside HIPAA and internal policies. Keep both in view and assign someone to track changes as state laws evolve.  

State AI requirements can change after a system is already in use. Organizations should be aware of updates and collaborate with their vendors to understand what those changes mean for the solution. Data governance should keep pace with those changes, including how data is handled and protected. Any resulting updates or actions should be documented. 

Mapping AI Systems to Applicable Requirements 

Now that you have inventory in place, the next step is to connect each system to the specific obligations that actually apply to it.  

Let’s take an ambient clinical documentation as an example.  

It is used during a patient encounter and may process protected health information (PHI). This brings privacy and security requirements into the picture. If its output is used in a process covered by another requirement, such as certain decisions about coverage or access to care, those requirements need to be considered too. State-specific requirements may apply as well. 

The organization can then identify the needed controls, such as access limits, minimum necessary PHI, a vendor BAA, and audit logs. Evidence may include the BAA and security assessment. 

Mapping looks like this:  

AI system → Use → Data → Applicable requirements → Controls → Evidence 

 
For every Tier 1 or Tier 2 system in the inventory, mapping helps surface requirements that are easy to miss otherwise. Two systems built on the same underlying technology can carry entirely different obligations because their use case and data differ. 

8 Steps to Prepare AI Systems for an Internal Audit 

Follow this sequence for audit preparation before an auditor, regulator, or accreditation surveyor asks for evidence. 

1. Confirm The AI Inventory 

Verify that every AI system currently in use is represented and up to date. Include the areas most likely to be missed, such as vendor-enabled features and employee-adopted tools. 

2. Assign Ownership 

For each system, identify a specific person who is accountable for its compliance posture and can speak to it directly. 

3. Identify Applicable Requirements  

Run the mapping exercise from the previous section for each system. Bring together federal, state, contractual, and internal policy requirements. 

4. Gather Documentation 

Pull together the risk assessments, approval records, vendor contracts and BAAs, validation results, and policies tied to each system. Keep the underlying artifacts, not just summaries. 

5. Verify Controls 

Check that the AI governance controls described in the documentation actually exist and work as described. If a control is documented but never implemented, an auditor is likely to spot it. 

6. Review Evidence 

Make sure the evidence for each control is complete, reasonably current, and traceable to the specific control and system it supports. 

7. Track Changes and Incidents 

Review model or vendor updates, exceptions, complaints, human overrides of AI output, and incidents. Check how each one was handled and closed. 

8. Identify and Close Gaps 

Document what is missing and assign an owner and remediation date for each gap. Keep the evidence showing that each gap was closed. A gap that is identified but never tracked to resolution is still a problem. 

What Happens When an AI System Changes? 

When an AI system changes, the organization needs to review the change and decide whether the existing approval still applies. 

Common changes include: 

• A new model or version replaces the one that was validated 

• The vendor updates the model 

• A new data source is added 

• The system starts serving a new user or clinical population 

• The model is used for a new clinical or operational purpose 

• A new integration changes how data flows through the system 

• The vendor or a subprocessor data handling and storage 

• Regulatory requirements 

The change is then assessed to determine whether reassessment is needed or whether it can be logged as a minor change. If reassessment is needed, update the relevant documentation and controls, then record the outcome.

Common AI Compliance Gaps Auditors Can Find 

The things that tend to show up in an AI compliance review are often pretty simple. An AI system may be missing from the inventory, its documentation may be out of date, or there may be no evidence that a required control was followed. 

Here are some gaps worth checking before an audit: 

Compliance gap	What to verify
Outdated risk assessment	Check that the risk assessment reflects relevant changes to the system or its environment.
Unclear accountability	Confirm that roles and responsibilities for managing AI risks are documented.
Controls are documented but not evidenced	Maintain documentation of the required safeguards and processes implemented.
Changes aren’t documented	Document relevant changes and update assessments or controls when those changes affect compliance.
Monitoring records are incomplete	Maintain records of required activity reviews, evaluations, and ongoing monitoring.
Identified gaps remain unresolved	Document corrective actions and track remediation of identified issues.
Retired systems aren’t properly documented	Document system decommissioning and the handling of relevant data and access.

Audit Readiness Is a Moving Target 

One thing worth changing in how we think about audit readiness is the concept of readiness itself. It suggests a finish line where everything is checked, documents are gathered, and the organization is prepared to answer questions. 

AI doesn’t really give you that finish line. The record keeps moving as systems change and decisions are made. The organizations that handle this well have the history and record of everything already there. If the next person can pick up the record and understand it, you have done something right.

FAQs About Healthcare AI Compliance and Audits