Do no harm: Why Missouri must set guardrails for AI mental health tools
AI mental health tools are already in clinics, EHR portals, and on patients' phones. Access improves, costs drop, and waitlists shorten - but the margin for error is thin. Missouri needs clear rules that protect patients while letting teams ship useful tools with confidence.
The goal is simple: if software influences care, it should be safe, evidence-based, and accountable. That starts with baseline standards, transparent claims, and real oversight.
What's at stake for clinicians and patients
- Accuracy and clinical validity: Chatbots can miss red flags or over-assure. Claims of "therapeutic benefit" often lack peer-reviewed evidence or consistent external evaluation.
- Bias and equity: Models trained on narrow data can under-detect symptoms in rural patients, veterans, communities of color, and those with atypical presentations.
- Privacy and data security: Many consumer apps sit outside HIPAA. Sensitive data can be shared with third parties, creating real risks for employment, housing, or insurance.
- Scope of practice: Self-help tools drift into clinical advice without licensed oversight, blurring liability and confusing users about what the tool can and cannot do.
- Crisis response: Weak suicide-risk detection and unreliable escalation pathways put users in danger when seconds matter.
Where current rules fall short
HIPAA covers covered entities and business associates, but many mental health apps never touch a covered entity. That leaves sensitive data exposed to weak privacy policies and ad tech pipelines.
Federal signals are helpful but incomplete. The FTC has pursued health apps under the Health Breach Notification Rule, and the FDA regulates certain software-as-a-medical-device. Gaps remain for behavioral tools that influence care without making explicit medical claims.
Missouri can close the distance with targeted, workable state standards that clinical teams and vendors can follow without guesswork.
Guardrails Missouri can put in place now
- Clinical validation: Require risk-based evaluation before market entry in Missouri. High-risk use cases (screening, triage, crisis detection) need prospective or real-world evidence and plain-language performance labels (e.g., sensitivity, specific populations tested).
- Clear labeling and disclosures: Every tool should disclose intended use, limitations, training data sources, known gaps, and release/version history. No therapy-language unless supervised by a licensed clinician.
- Privacy and data minimization: Ban the sale or advertising use of mental health data. Require opt-in consent for any secondary use. Align with 42 CFR Part 2 principles for substance-use information even when the app is outside HIPAA scope.
- Bias and equity testing: Mandate pre-release and ongoing audits across key demographics common in Missouri (rural vs. urban, age bands, race/ethnicity, veterans). Publish disparity metrics and remediation timelines.
- Scope-of-practice rules: Define thresholds where a tool becomes clinical care and must include licensed oversight, documentation standards, and audit trails.
- Crisis safety: Require validated suicidality detection, user consent for location-aware routing, and reliable escalation to human responders. Tools must fail safe, surface emergency options immediately, and log handoffs.
- Incident reporting: Stand up a state reporting channel for adverse events and near-misses, including a safe harbor that encourages disclosure and fixes.
- Procurement standards: For state-funded providers, require conformity with recognized risk frameworks and secure development practices.
NIST's AI Risk Management Framework offers a practical baseline for risk tiers, documentation, and testing. It's a strong foundation for state procurement and vendor expectations.
Practical steps Missouri healthcare organizations can take today
- Inventory and risk-rank tools: List every AI-enabled feature touching mental health. Classify by patient impact (informational, augmentative, high-risk).
- Shadow mode before go-live: Run tools alongside current workflows; compare outputs to clinician judgment and gold-standard screeners. Track false negatives on crisis risk.
- Data safeguards: Treat app data as sensitive by default. Use BAAs or data protection addenda, restrict analytics sharing, and turn off product "training" on user inputs unless explicitly consented.
- Bias checks: Measure performance by subgroup. If disparity ratios exceed thresholds, pause or restrict to populations where performance is solid.
- Crisis protocols: Define escalation paths, train staff, and run drills. Measure time-to-human and completion of warm handoffs.
- Clinician-in-the-loop: For high-risk uses, require human review and accountability. Document overrides and feedback to improve models.
- Patient communication: Provide clear, readable disclaimers and informed consent. Give users a simple way to opt out and request deletion.
- Upskill your team: Offer focused training on evaluation, prompts, and clinical safety. A curated option for role-based learning is available here: Complete AI Training - Courses by Job.
A workable state framework
- Tiered oversight: Low-risk tools get light touch; high-risk tools require evidence, labeling, and monitoring.
- State registry: Public listing of approved tools with versions, intended use, evidence summaries, and known limitations.
- Continuous monitoring: Require post-market surveillance, adverse event reporting, and sunset reviews every 12-18 months.
- Enforcement with coaching: Fines for deceptive claims, but also technical assistance to help vendors meet standards.
- Coordination: Align with federal guidance to reduce friction for multi-state providers and vendors.
Metrics that matter
- Sensitivity and specificity for suicide risk and severe symptom flags.
- Median time from risk flag to human contact.
- Adverse event and near-miss rates per 1,000 users.
- Disparity ratios by subgroup and setting (clinic, school, telehealth).
- Data incidents: unauthorized sharing, re-identification attempts, or retention beyond policy.
Call to action
Missouri can lead with clear, practical guardrails that clinicians trust and vendors can meet. Protect privacy, require evidence for high-risk use, and ensure a clean handoff to human care in moments that matter.
Legislators, regulators, and healthcare leaders should align on a tiered framework, build a simple registry, and fund unbiased evaluations. Patients will be safer, clinicians will have clarity, and innovators will have a stable path to deliver value without cutting corners.
For additional federal context on privacy enforcement in health apps, see the FTC's Health Breach Notification Rule overview: FTC guidance.
Your membership also unlocks:
