California Lawmakers Propose Guardrails for AI in Clinical Decision-Making
California Assembly members are debating how to regulate artificial intelligence in health care, specifically the use of AI systems that guide doctors and nurses in patient care decisions. Assembly Bill 1979, titled "Health care services: artificial intelligence," would require health care professionals to exercise independent judgment when relying on AI-generated recommendations.
The bill targets clinical decision support systems-AI tools integrated into electronic health records that help clinicians with diagnoses and drug interaction checks. Assemblymember Bonta, who authored the measure, said the technology should "assist human clinicians, not replace them."
What the bill requires
AB 1979 would mandate that health care entities make final clinical decisions themselves, rather than deferring to AI outputs. The measure also extends California's Confidentiality of Medical Information Act to health care chatbots, requiring companies to protect patient data from being sold or analyzed for purposes outside direct patient care.
The bill addresses concerns that uploading medical records to AI services like ChatGPT Health could expose sensitive information to large language models operating outside traditional privacy protections.
Support and opposition divide along predictable lines
The California Nurses Association, California Labor Federation, and Consumer Watchdog support the bill. They argue that AI systems risk connecting medical records to language models that could "analyze, sell, or otherwise utilize health data outside the traditional privacy frameworks."
Kaiser Permanente, the California Hospital Association, and Adventist Health oppose it. These organizations suggest the restrictions could slow adoption of tools that improve efficiency and patient outcomes.
The efficiency argument
Studies suggest AI in emergency departments could increase patient survival rates while maintaining quality care. Neela Paykel, a tech general counsel who has advised health care companies, said AI could ease workload burdens on underpaid emergency medicine staff.
Jill Speece, an engineering professor who previously consulted in health care, said: "It's not reasonable to prohibit it when it's something that could improve health care efficiency, so what we have to do is use it properly."
The privacy counterargument
Health care workers and privacy advocates worry that connecting AI systems to electronic health records creates new vulnerability. Electronic health records already contain safeguards for patient privacy, some argue, making additional AI integration unnecessary.
Annalise Mui, a public health student researching AI in health communication, said AI in clinical workflows could also damage the relationship between patients and providers. "AI essentially eliminates that," she said of the human connection in care.
The bill remains in the California Assembly. If passed and signed into law, it would take effect January 1.
For more on AI for Healthcare and Generative AI and LLM, explore relevant resources for your field.
Your membership also unlocks:
