AI Governance for Healthcare Startups: HIPAA, HITECH, and the EU AI Act

Healthcare is the sector where AI governance failures have the highest stakes. A misconfigured AI tool that receives Protected Health Information without a Business Associate Agreement creates HIPAA liability, potential HITECH breach notification obligations, and civil monetary penalties that scale with negligence.

"Is ChatGPT HIPAA compliant?" is one of the most-searched AI questions among healthcare teams. The short answer is: not by default, and not in its consumer form. This guide covers exactly what healthcare startups need to use AI legally and how to build a minimum-compliant setup.

TL;DR: ChatGPT.com (consumer), Claude.ai, and most AI productivity tools are not HIPAA compliant and cannot be used with PHI without a Business Associate Agreement. A BAA is available from: OpenAI (Enterprise only), Microsoft Azure OpenAI, Google Cloud Vertex AI, and AWS Bedrock. Healthcare startups in the EU also face EU AI Act high-risk obligations for diagnostic, triage, or treatment-recommendation AI. The minimum compliant setup is: BAA-covered AI provider + documented PHI boundary + human oversight for clinical decisions.

The BAA Requirement, Why Most AI Tools Are Off-Limits

HIPAA requires that any "business associate", a vendor that creates, receives, maintains, or transmits Protected Health Information on your behalf, sign a Business Associate Agreement with you before accessing PHI.

PHI includes any combination of health information with identifiers: name, date of birth, email address, phone number, address, insurance ID, or any other identifier that could link the health information to a specific person. A patient's diagnosis alone is not PHI. A patient's diagnosis plus their date of birth is PHI.

When an employee submits a message to an AI tool, that message is transmitted to the AI vendor's servers for processing. If the message contains PHI, the AI vendor is receiving PHI, and becomes a business associate who requires a BAA.

Without a BAA, using an AI tool with PHI is a HIPAA violation, regardless of the vendor's security practices.

Which AI Tools Have a BAA Available

The pattern is consistent: consumer products from any vendor have no BAA and cannot be used with PHI. Enterprise API and cloud platform access, where you sign a formal agreement, is where BAAs are available.

What Counts as PHI in an AI Prompt

Employees often assume that removing a patient's name from a prompt makes it safe. This is incorrect. PHI includes any of 18 HIPAA-defined identifiers when combined with health information:

1. Patient name
2. Geographic data smaller than state (address, city, zip code)
3. Dates (other than year): birth date, admission date, discharge date
4. Phone numbers
5. Fax numbers
6. Email addresses
7. Social Security numbers
8. Medical record numbers
9. Health plan beneficiary numbers
10. Account numbers
11. Certificate or license numbers
12. VINs and serial numbers
13. Device identifiers
14. Web URLs
15. IP addresses
16. Biometric identifiers (fingerprints, voice prints)
17. Full-face photos or comparable images
18. Any other unique identifier

Practical rule for AI prompts: If a human could identify a specific patient from the information in the prompt, it is PHI. "A 47-year-old male with Type 2 diabetes in Austin, TX admitted on March 14" is PHI even without a name.

Instruct employees: when using AI for clinical summarization, documentation, or coding, use only de-identified data or use a BAA-covered tool configured for PHI.

The EU AI Act: Healthcare AI Is High-Risk by Default

Healthcare startups building AI products (not just using productivity AI) face an additional regulatory layer: the EU AI Act classifies certain healthcare AI as high-risk.

High-risk healthcare AI under the EU AI Act is classified through two pathways.

Annex I (Article 6(1)) covers AI embedded in medical devices (MDR/IVDR):

• AI intended to support clinical diagnosis (e.g., imaging analysis, disease detection)
• AI systems that influence treatment or medication decisions
• Any AI that is a safety component of a medical device regulated under EU MDR or IVDR

Annex III, point 5(d) covers emergency systems:

• Emergency healthcare patient triage systems
• Emergency call classification systems

High-risk AI systems are not banned, they are permitted with stricter requirements:

The deadline for high-risk AI systems under the EU AI Act is December 2, 2027 (extended from August 2, 2026 by the EU Digital Omnibus provisional agreement, May 7, 2026). Healthcare startups deploying diagnostic or triage AI in the EU must be compliant by then.

If your AI product is integrated into a medical device regulated under the EU Medical Device Regulation (MDR) or In Vitro Diagnostic Regulation (IVDR), additional conformity requirements from those regulations apply alongside the EU AI Act.

The Minimum Compliant Setup for a Healthcare Startup

If your healthcare startup uses AI tools for internal operations (documentation, coding, research), not clinical decision-making, here is the minimum governance setup:

Get a BAA with your AI vendor for any tool that will touch PHI. If you use Azure OpenAI, ensure your Microsoft agreement includes the HIPAA BAA. If you use AWS Bedrock, confirm Bedrock is listed as a HIPAA-eligible service in your BAA.

Document your PHI boundary. Write down which AI tools may receive PHI (those with BAAs) and which may not (all consumer tools). Include this in your AI acceptable use policy and your HIPAA policies and procedures.

Train employees on AI-specific PHI risks. Add content to your HIPAA training: what counts as PHI in a prompt, which tools have BAAs, and what to do if PHI is accidentally submitted to a non-BAA tool.

Update your incident response plan to cover AI-related breaches. If an employee submits PHI to a non-BAA AI tool, this is a potential breach event. Your existing HIPAA breach notification process should cover AI-triggered disclosures, confirm it does.

If your startup is building AI-powered clinical tools for EU deployment, start your conformity assessment now. December 2027 is the deadline, and conformity assessments for high-risk AI take months.

HITECH and AI-Related Breach Notification

HITECH (the Health Information Technology for Economic and Clinical Health Act) strengthened HIPAA's breach notification requirements and is directly relevant when an AI tool is involved in a PHI exposure.

When an employee submits PHI to a non-BAA AI tool, this is a potential HITECH reportable breach, not just an internal policy violation. HITECH requires:

Notification timelines:

• Affected individuals must be notified within 60 days of discovering a breach
• The Department of Health and Human Services (HHS) must be notified within 60 days (or annually for breaches affecting fewer than 500 individuals in a state)
• Breaches affecting 500 or more individuals in a single state or jurisdiction require media notification in addition to individual and HHS notification

What counts as a breach in an AI context: An employee pasting PHI into consumer ChatGPT is a potential breach because the data was transmitted to a vendor without a BAA. Whether it qualifies as a reportable breach depends on a four-factor risk assessment: the nature of the PHI and likelihood it could be re-identified, whether the unauthorized person who received it could actually have accessed it, whether PHI was actually acquired or viewed, and the extent to which the risk has been mitigated.

The low probability of compromise exception: If you can demonstrate a low probability that PHI was compromised, for example, because the AI vendor immediately deleted the conversation and confirmed no human reviewed it, you may be able to document the incident as not meeting the reportable breach threshold. Document the analysis regardless of which conclusion you reach.

AI-specific incident response steps for healthcare startups:

1. Identify what PHI was submitted, to which AI tool, by which employee
2. Confirm whether the AI vendor has a BAA (determines breach vs. policy violation)
3. Request evidence from the vendor that data was not retained or accessed by humans
4. Run the four-factor risk assessment
5. Document conclusions and retain records for 6 years (HIPAA's standard documentation retention period)
6. Notify HHS, individuals, and (if applicable) media if the breach threshold is met

A single employee pasting a patient name plus diagnosis into a non-BAA AI tool can trigger this entire process. Preventive governance, clear PHI boundaries, approved tool lists, employee training, is significantly less costly than breach notification at scale. HITECH penalties for willful neglect start at $10,000 per violation and can reach $2,190,294 per violation category annually (HHS inflation-adjusted cap effective January 28, 2026). For healthcare AI specifically, the compliance investment pays for itself the first time it prevents a reportable breach.

Using AI in a healthcare context? Start with the Compliance Quiz to see which specific regulations apply to your team, then use the AI Risk Assessment to rate your AI use cases from Low to Critical.

Related Reading

• EU AI Act compliance guide for small teams
• AI vendor due diligence checklist
• Privacy-first AI APIs, which don't train on your data
• AI governance roles and responsibilities for small teams
• AI data privacy for small teams, GDPR/CCPA
• AI governance guide for small teams