HIPAA Compliant AI: What It Takes to Put an LLM Feature Into a Healthcare App
HIPAA compliant AI for healthcare apps: which AI vendors sign a business associate agreement, de-identification, access control, audit logs, and a checklist.
HIPAA compliant AI is not a model you buy; it is a system you build around a model. A large language model can sit inside a healthcare app legally when the vendor has signed a business associate agreement for the exact services you use, when protected health information reaches the model only where the feature needs it, and when your app controls and records every access. Below: what the major AI vendors offer, where HIPAA compliant AI tools stop and your engineering starts, and the checklist we use before an LLM touches patient data.

What HIPAA Compliant AI Actually Means
HIPAA has no certification for software. Microsoft and Google Cloud say so on their own compliance pages — there is no HHS-approved certification programme for cloud services — and AWS, like both of them, states that customers remain responsible for their own compliance. What the law does require is a contract and a set of safeguards.
The contract is the business associate agreement (BAA). When a covered entity (a clinic, a hospital, a health plan) or one of its business associates sends protected health information (PHI) to a service provider, that provider becomes a business associate too, and the two sides need a signed agreement that sets out how the data may be used, protected and reported if something goes wrong. A model API receiving patient details in a prompt is one.
The safeguards come from the Privacy Rule and the Security Rule: limit use to what the task needs, control access, protect data in transit and at rest, and keep records. None of that disappears because a model does the processing. The useful definition: an AI feature is HIPAA compliant when the whole path the data travels — your app, your backend, the model provider and anything the model can reach — is covered by agreements and safeguards.
Medical AI in Practice: What Clinicians Use It For
Medical AI covers a wide range: diagnostic imaging models cleared as medical devices at one end, general-purpose language models helping with paperwork at the other. A product team adding AI to a practice tool or patient app, rather than building a medical device, is at the paperwork end: drafting visit notes from a transcript, summarising a chart before an appointment, writing patient letters and prior-authorisation requests, answering questions about internal protocols, triaging inbound messages.
These features save clinicians time on documentation, which is why staff reach for free consumer chat apps when their own tools lack them — the first risk to plan for.
Free Versus Paid HIPAA Compliant AI Tools
The difference between free and paid is not the model; it is the contract. A free tier of a consumer chat app usually comes with terms that allow the provider to retain conversations and offers no BAA, so PHI typed into it is a disclosure without a contract. Paid business tiers are not automatically different: OpenAI's own help centre says it does not offer a BAA for ChatGPT Business, while sales-managed ChatGPT Enterprise and Edu accounts can request one.
Some clinical tools are free to the clinician and still covered, because the company behind them signs agreements with the organisations or clinicians who use them. Accessible and secure can go together; it has to be checked tool by tool against the paperwork, not the marketing page. What to look for before staff use a tool:
- a BAA you can read, signed for your organisation or for the individual clinician;
- a clear statement of which features the agreement covers and which it excludes;
- data retention terms, and whether conversations are used to train models;
- single sign-on, so access ends when employment ends;
- an export or audit view of who used the tool and when.
Which AI Vendor Will Sign a Business Associate Agreement
For teams building their own LLM in healthcare features, the choice of AI vendor is mostly a choice of contract and coverage. What each provider's own documentation says, as of October 2026:
- OpenAI API. Organisations with an established usage history can accept a standard BAA in the API Platform settings, without an enterprise agreement. The help centre is explicit that this does not make your application compliant, and only the API services listed as eligible may receive PHI.
- Anthropic. A BAA covers HIPAA-ready services such as the first-party API and Enterprise plans, for the single organisation that accepted it. Not every API feature is covered — the Batch, Files, Skills, Code Execution and Web Fetch features are listed as excluded — and the covered models require 30-day data retention rather than zero data retention unless Anthropic grants an exception.
- Azure OpenAI and other Azure services. Microsoft makes its HIPAA BAA available by default through its Product Terms for in-scope services, with no separate contract to sign, and lists the covered services on its audit-scope page.
- Amazon Bedrock. AWS lists Bedrock as a HIPAA eligible service; it may be used with PHI only after the customer has entered into the AWS business associate addendum and configured the service accordingly.
- Google Cloud. Google's BAA covers the infrastructure and a published list of products, which includes its generative AI agent platform and Gemini Enterprise offerings; Google describes compliance as a shared responsibility.
Two consequences follow. Read the excluded-features list before you design the feature: a document-upload flow built on an excluded endpoint is out of scope even when the main API is covered. And the agreement covers the vendor's side only — your backend, vector database, logging tool and any third-party service the model can call need their own coverage.
Keep Protected Health Information Out of the Prompt
The Privacy Rule's minimum necessary standard applies to prompts like any other disclosure: send the model what the task needs and nothing more. A protocol assistant answering "what is our follow-up interval after this procedure" needs no patient data at all; a letter generator needs a first name and the clinical facts, not the full chart.
Where a feature can run on de-identified data, HHS recognises two methods. Safe Harbor removes a defined list of identifiers — names, geographic units smaller than a state, date elements other than the year, contact details, record and account numbers, device identifiers, biometrics, full-face photographs and the other listed categories. Expert Determination has a qualified expert document that the re-identification risk is very small. In an app, de-identification is usually a step inside your own environment that swaps identifiers for tokens before the prompt is sent and back again in the response.
Free text is the trap: HHS's guidance says identifiers in free-text fields must be removed too, and clinical notes are full of names, dates and places. Test redaction on real-looking notes, not clean samples.
Access Control and Audit Logs for an LLM in Healthcare
The access control question changes once a model can retrieve data on its own. In a retrieval-augmented feature, the model reads whatever the retrieval layer returns, so permissions have to be enforced in that layer: a nurse asking about a patient on another ward should get nothing back, regardless of how the question is phrased. The pattern we use is to filter documents by the requesting user's permissions before they are ranked, never after the model has seen them, and to test it with prompts written to cross those boundaries. The retrieval design itself is covered in our guide to RAG chatbot architecture, evals and cost.
Audit logs are the second half. For every model call, record who asked, which patient records were retrieved, which model and version answered, what was returned and whether a human accepted, edited or discarded it. Keep the log inside your covered environment, not in a general analytics tool. The same records serve accountability, debugging and cost tracking — the last one is broken down in the real cost of an LLM feature.
The Self-Hosted Option: Open-Weight Models
Running an open-weight model in your own cloud account removes the model vendor from the data path; your cloud provider's BAA still matters, but there is no third-party model API to cover. You take on hosting, scaling and patching, and a smaller open model may handle long clinical documents less well than the largest hosted ones, so test it on your own notes. It suits narrow, high-volume tasks such as classification or extraction. A sensible path is to start on a covered hosted API and move individual tasks in-house once usage justifies it.
Healthcare App Development Checklist for AI Features
Whether the feature lives in a telemedicine app, a practice-management tool or a patient portal, the healthcare app development work around it looks the same. The order we follow:
- Map the data path: every system that will see PHI, including logs and monitoring tools
- Confirm a signed business associate agreement for each of them, and read each one's excluded-features list
- Decide per feature whether it needs patient data at all; de-identify where it does not
- Enforce access control in the retrieval layer, before ranking, and test it with boundary-crossing prompts
- Record audit logs for every model call inside the covered environment
- Turn off training on your data and set retention to the shortest the vendor allows for covered use
- Keep a human review step on anything that reaches the chart or the patient
- Give staff a covered tool, so they stop pasting notes into free apps
- Update your security risk analysis and policies before launch, not after
We built the Caidr symptom checker app as a guided decision-tree assessment rather than free-form generation — a reminder that not every healthcare problem needs a language model. When one does, the checklist above is where the budget goes, and it is where our AI integration service starts in its discovery week.
FAQ
Is ChatGPT HIPAA compliant? Not by default. OpenAI offers a BAA for sales-managed ChatGPT Enterprise and Edu accounts and has a separate flow for eligible clinicians, but none for ChatGPT Business, and the free consumer app carries no agreement at all. Even with an agreement, your organisation remains responsible for how staff use it and what they enter.
Is AI prohibited under HIPAA? No. HIPAA does not prohibit artificial intelligence; it regulates how PHI is used and disclosed. An AI service that receives patient data needs a BAA and the same safeguards as any other vendor: limited use, access control, encryption and records of access. AI that never receives identifiable data, such as a protocol assistant, falls outside those rules entirely.
Is the OpenAI API HIPAA compliant? The OpenAI API can be used with PHI once your organisation has accepted OpenAI's BAA and enabled HIPAA support in the API Platform. Only the services OpenAI lists as eligible are covered. OpenAI states that the agreement does not make your application compliant; the rest of the system, from your backend to your logs, remains your responsibility.
What is the difference between de-identified data and limited data? De-identified data has had identifiers removed by Safe Harbor or certified by Expert Determination, and is no longer protected health information. A limited data set keeps some identifiers, such as dates and city, and may be used only under a data use agreement. For prompts, de-identification is the more practical target, because it takes the data out of HIPAA's scope.
How long does it take to add an AI feature that meets HIPAA to an existing app? For a focused feature on a covered hosted API, our AI integration work takes four to eight weeks to production, starting with a discovery week. What stretches timelines is rarely the model; it is signing agreements, de-identification that works on messy notes, and the security review. Plan those in parallel with the build rather than after it.
If you are adding an LLM feature to a healthcare product and want the data path, agreements and safeguards right before the first prompt is sent, we can map it with your team in a fixed-price discovery week. Talk to IvorySoft and we will start with your data flow and the feature you want to ship.