Testing Health AI for Privacy - Not Just Performance
Publication date: January 5, 2026
New research presented at the 2025 Conference on Neural Information Processing Systems shows how to test foundation models trained on de-identified electronic health records (EHRs) so they don't leak sensitive patient information. The goal is simple: before release, verify that targeted prompts can't extract patient-level details.
Confidentiality is core to medicine. It builds trust. As AI absorbs more clinical data, we need guardrails that reflect that same standard.
The core issue: memorization vs. generalization
Foundation models should generalize by drawing on patterns across many records to make predictions. Memorization is different: the model leans on a single patient's record to produce an output. That can cross a privacy line.
High-capacity models are known to leak training data under adversarial prompting. "Knowledge in these high-capacity models can be a resource for many communities, but adversarial attackers can prompt a model to extract information on training data," says Sana Tonekaboni, first author of the paper.
The work was conducted with MIT Associate Professor Marzyeh Ghassemi, who leads the Healthy ML group at the Abdul Latif Jameel Clinic for Machine Learning in Health (Jameel Clinic). The group focuses on dependable machine learning in health settings.
What the tests actually measure
The team built a structured evaluation to assess privacy risk in context. The tests measure different kinds of uncertainty and stress the model under realistic "tiers" of attacker knowledge to see what, if anything, leaks.
They distinguish between safe generalization and patient-level memorization. That separation matters: it identifies whether a response reflects broad clinical knowledge or a specific person's record.
Practicality is the point. "If an attacker has to know the date and value of a dozen laboratory tests from your record in order to extract information, there is very little risk of harm. If I already have access to that level of protected source data, why would I need to attack a large foundation model for more?" says Ghassemi.
Key findings that should shape your release process
- More prior knowledge, more risk: As an attacker's knowledge about a patient increases, leakage becomes more likely.
- Not all leaks are equal: Revealing age or demographics is one tier of risk; disclosing an HIV diagnosis or alcohol abuse is another.
- Uniqueness matters: Patients with rare or distinctive conditions are easier to single out and may require stronger protections.
- Context is non-negotiable: Evaluate leakage in a health care context to judge whether it meaningfully compromises privacy.
Why this matters now
Breaches are frequent. In the past 24 months, hundreds of U.S. health data breaches affecting 500+ individuals were reported, most due to hacking or IT incidents. See the U.S. Department of Health and Human Services breach portal for a sense of scale.
Practical checkpoints before you ship an EHR foundation model
- Test for patient-level memorization vs. generalization using structured prompts across tiers of attacker knowledge.
- Quantify leakage severity by harm class (e.g., demographics vs. stigmatizing diagnoses).
- Document the minimum prior information required to trigger leakage and assess whether that scenario creates real-world harm.
- Red-team with clinicians, privacy specialists, and legal experts to reflect clinical context and policy obligations.
- Gate model access and monitor prompts for extraction attempts; update tests as models and interfaces change.
What's next
The researchers will expand testing with clinicians, privacy experts, and legal advisors. As Tonekaboni puts it, "There's a reason our health data is private. There's no reason for others to know about it."
For researchers tracking the technical conversation, the work was presented at NeurIPS 2025.
Funding and resources
This work was supported by the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard, Wallenberg AI, the Knut and Alice Wallenberg Foundation, the U.S. National Science Foundation (NSF), a Gordon and Betty Moore Foundation award, a Google Research Scholar award, and the AI2050 Program at Schmidt Sciences. Resources were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
Your membership also unlocks:
