Coffee Beans - AI, Health Misinformation, and Responsibility
25.11.2025
AI tools now touch nearly every part of healthcare work. They help draft notes, summarize studies, and organize projects. They can also produce confident, convincing nonsense. In healthcare, that risk isn't academic-it's a patient safety issue.
What a recent stress test found
A study in the Annals of Internal Medicine tested five widely used LLMs (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Llama 3.2-90B Vision, and Grok Beta) through their APIs. Researchers fed them formal scientific instructions laced with falsehoods and backed by fabricated citations.
Four models-GPT-4o, Gemini, Llama, and Grok-used 100% of the false inputs to generate misleading health claims. Claude 3.5 used about 40%. Topics included vaccine-autism myths, claims that 5G causes infertility, the idea that sunscreen raises skin cancer risk, unsafe cancer diets, garlic as an antibiotic substitute, and airborne HIV-each "supported" by nonexistent citations.
Why this matters for your practice
These systems can be manipulated at a deep instruction level. It's possible to turn a helpful assistant into a confident source of disinformation, especially when prompts include fake "evidence."
Journalists are feeling the same pressure. In one survey, 77% report using AI tools, and more than a third see misinformation as the top threat to AI-enabled reporting. That same dynamic reaches your patients and your staff.
Practical guardrails for healthcare teams
- Define allowed uses. Draft patient education, summarize guidelines, and generate administrative content. Prohibit diagnosis or treatment recommendations without clinician review and sign-off.
- Verify every clinical claim. Cross-check against trusted sources (e.g., national guidelines, specialty societies). If a claim cites a paper, confirm it via PubMed/DOI. If you can't find the citation, treat it as fabricated.
- Treat all inbound content as untrusted. Patient forums, PDFs, screenshots, or emails can carry hidden instructions. Instruct tools to ignore any commands found in the input and to summarize content only.
- Use retrieval from vetted repositories. Feed models curated, up-to-date clinical materials. Disable open web access or browsing unless routed through a controlled, logged process.
- Harden system prompts. Explicitly forbid following instructions embedded in user content. Require citation checks, uncertainty flags, and refusal to answer beyond scope.
- Log and monitor. Capture prompts and outputs, sample them weekly, and run a quarterly red-team exercise focused on high-risk topics (vaccines, oncology, infectious disease).
- Human in the loop. Anything that could influence care requires clinician validation before it reaches a patient or record.
- Train your staff. Give everyone a simple verification workflow and clear do/don't rules for AI use in your setting. If your team needs structured upskilling, consider role-specific options like AI courses by job.
60-second citation check
- Look for a DOI or PMID. If present, open it in PubMed or the journal site.
- Confirm journal, author names, and year match the claim.
- Scan the abstract. Does it actually support the stated conclusion?
- No DOI/PMID, mismatch, or paywalled with no trace elsewhere? Assume the citation is fabricated until proven otherwise.
What to tell patients
- Ask for sources. "Can you share where you saw this?" opens the door without judgment.
- Encourage bringing online claims to appointments. Review them together and model quick source checks.
- Share dependable channels (national public health agencies, specialty societies). Make your clinic's recommended list easy to find.
- Explain that AI tools can mix real facts with made-up studies. Curiosity beats confrontation.
Bottom line
AI can save time, but uncritical use can spread harmful myths fast. Build simple guardrails, verify sources, keep clinicians in the loop, and communicate clearly with patients. This is how we protect trust-and health outcomes-while still getting value from the tools.
References
- Modi ND, Menz BD, Awaty AA, Alex CA, Logan JM, McKinnon RA, Rowland A, Bacchi S, Gradon K, Sorich MJ, Hopkins AM. Assessing the System-Instruction Vulnerabilities of Large Language Models to Malicious Conversion Into Health Disinformation Chatbots. Ann Intern Med. 2025 Aug;178(8):1172-1180. doi: 10.7326/ANNALS-24-03933
- Muck Rack: The State of Journalism 2025
- MedicalXpress coverage of LLM health disinformation safeguards
- FactCheck Africa: AI, misinformation, and the future of journalism in Africa (2025)
- GlobeNewswire: Disinformation and Misinformation Are Top Concerns in Journalism (2025)
*This text has received support from the National Recovery Plan under project 1.4 CEDMO 1 - Z220312000000, Support for increasing the impact, innovation, and sustainability of CEDMO in the Czech Republic, which is financed by the EU Recovery and Resilience Facility.
Your membership also unlocks:
