AI is pouring into healthcare. Here's how to use it without getting burned
In the past week alone, OpenAI bought health startup Torch, Anthropic launched Claude for healthcare, and MergeLabs raised a $250 million seed round at an $850 million valuation. Money and products are flooding into health and voice AI because the opportunity is obvious: lower admin burden, fill staffing gaps, and improve access.
The concern is just as obvious. Hallucinations, inaccurate medical advice, and weak security controls can put patients and organizations at risk. If you work in healthcare, your job is to capture the upside while limiting the downside.
Why the rush
- Labor squeeze: documentation, prior auth, and patient messages are swamping clinicians.
- Model quality: general models are stronger, and health-tuned variants are arriving.
- Voice is ready: ambient scribing and call-center automation are finally usable.
- Capital push: investors want AI deployed where the problems (and budgets) are largest.
Where AI will land first
- Ambient clinical documentation: draft notes from patient visits, reviewed by clinicians.
- Patient triage and nurse lines: first-pass intake, routing, and education prompts.
- Revenue cycle: coding suggestions, denial letters, and prior auth packet assembly.
- Care management: personalized outreach scripts and follow-up reminders.
- Clinical decision support: evidence lookups with citations, under human oversight.
- Back-office: HR, IT helpdesk, scheduling, and supply chain queries.
- Voice AI: intake, discharge instructions, and call summaries.
Risks you can't ignore
- Hallucinations and outdated facts posing as confident answers.
- Unsafe recommendations that bypass clinical reasoning.
- Bias from skewed training data that affects triage or treatment suggestions.
- PHI exposure through prompts, logs, or vendor training pipelines.
- Model drift and version changes that alter behavior without notice.
- Vendor lock-in and unclear IP rights on your data.
- Weak uptime and poor incident response.
- Compliance gaps across HIPAA, state privacy laws, and device regulations.
Practical safeguards that work
- Keep a human in the loop for any clinical output; require attestations in the workflow.
- Use guardrails: structured prompts, strict output formats, and clinical reasoning checklists.
- Retrieval-augmented generation: ground the model in approved, current sources and cite them.
- Minimize PHI: de-identify where possible and separate secrets from prompts.
- Vendor security: BAA, encryption, SOC 2 or HITRUST, breach playbooks, and clear data retention.
- Audit everything: prompts, outputs, reviewers, and model versions.
- Safety testing: red-team with harmful and edge-case scenarios before go-live.
- Fallbacks: graceful degradation to traditional workflows if AI fails or confidence is low.
- Measure impact: time saved, quality metrics, denial rates, patient satisfaction, and error rates.
Questions to ask every vendor
- Do you train your models on our data? If so, how is PHI excluded?
- Where is data stored and processed (region, subprocessors)? Provide a data flow map.
- Will you sign a BAA? Which certifications do you hold (SOC 2, ISO 27001, HITRUST)?
- How do you defend against prompt injection and data exfiltration?
- What sources ground clinical answers? Are citations included by default?
- How are model changes versioned, tested, and communicated?
- What are your uptime SLAs and incident response timelines?
- Total cost of ownership: licenses, usage fees, integration, and change management.
- Do you rely on fine-tuning, RAG, or both? Why that choice for this use case?
Build vs. buy: make the call
Buy for common workflows (scribes, call summaries, coding suggestions) where vendors already integrate with your EHR and telephony. Build when your use case is unique, data-rich, and tied to your moat (e.g., specialty pathways, proprietary registries).
Either way, plan for integration with Epic/Cerner, SSO, logging, and clinician-facing UX. Budget time for training and change management; the tech is the easy part.
Regulatory checkpoints
- HIPAA Security Rule: confirm access controls, audit logs, and transmission security. HHS overview
- If the tool influences diagnosis or treatment, check whether it qualifies as SaMD and requires FDA oversight. FDA AI/ML guidance
Ninety-day pilot plan
- Pick one workflow with clear pain and measurable outcomes (e.g., note drafting for one clinic).
- Name a clinical champion and an operations partner; set weekly reviews.
- Baseline metrics for time, quality, and error rates; define pass/fail thresholds.
- Run shadow mode for 2-3 weeks, then supervised production for 4-6 weeks.
- Assess safety, adoption, ROI, and patient feedback; decide go/no-go and scale plan.
Skills to build in your team
- Clinical prompting: how to ask for structured, verifiable outputs.
- AI product literacy: limits, failure modes, and escalation paths.
- Privacy and security basics for PHI in AI workflows.
- Evaluation: test sets, bias checks, and continuous monitoring.
If you want a fast start on upskilling by role, browse curated programs here: Complete AI Training - Courses by Job.
The bottom line
AI is arriving fast in healthcare because the need is real. Treat it like any high-stakes tool: narrow scope, strict guardrails, measurable outcomes, and clear accountability. Do that, and you'll reduce burden without increasing risk.
Your membership also unlocks:
