AI news ·
California reveals it uses six high-risk automated systems for decisions on jobs, education, and criminal justice
California disclosed six high-risk automated systems it uses to make consequential decisions, reversing last year's claim of zero. The tools include predicting recidivism and flagging unemployment fraud, affecting millions of residents.
California has identified six automated systems it currently uses to make high-stakes decisions about people's lives, according to a state technology department report released June 12. The disclosure marks a sharp reversal from last year, when state agencies reported no such systems - an answer that contradicted well-documented examples of their use in corrections and unemployment benefits.
The report is required under a 2023 law mandating annual transparency into "high-risk automated decision systems." The law defines these as systems that assist or replace human judgment in decisions with legal or similarly significant effects on housing, education, employment, credit, health care, or criminal justice. Civil rights, privacy, and civil liberties groups pushed for the legislation, citing concerns that AI-driven tools can produce biased results against marginalized communities.
What the six systems do
The disclosed systems serve a range of functions across state agencies:
- Predicting whether incarcerated people will re-offend
- Evaluating whether unemployment claims are fraudulent
- Remotely administering exams for California State University students
- Detecting when college students use generative AI to write assignments
Two additional systems named in the report are not currently in use. The Department of Cannabis Control is developing AI to analyze whether marijuana packaging violates a law against appealing to children. California State University discontinued a language model it had used to review job applications.
Some of the systems have been operating for years. The California Department of Corrections and Rehabilitation has used COMPAS to assign recidivism scores to inmates for at least a decade. The technology department said it found more systems this year because it evaluated agency responses more thoroughly, including by meeting with departments and questioning them directly about their tools.
What the report leaves out
The inventory does not include generative AI pilot projects backed by the governor's office. Those include tools to help businesses file taxes, support state employees working on homelessness, and an AI assistant called Poppy - which uses language models like Anthropic's Claude to draft documents, research policy, and build custom AI tools. A state website says 67 departments provided input during the pilot phase, and the statewide rollout of Poppy begins next month.
A California State University contract with OpenAI to provide ChatGPT also goes unmentioned. The 2023 disclosure law exempts several agencies from reporting, including the judicial branch and the University of California system. CalMatters reported last month that a majority of the roughly 60 courts operating statewide have adopted generative AI use policies. Courts in Los Angeles and Riverside counties are testing an AI tool to act as a clerk, drafting orders and producing research memos.
The report separately identified six additional systems initially flagged as high risk but later determined not to qualify. One was AI used for legislative bill analysis by the California Department of Finance.
Legislative and public context
These findings arrive as California-based AI companies like Anthropic and OpenAI pursue government contracts and go public. Surveys by TechEquity and Carnegie California last year found most Californians prioritize safety over innovation. A Gallup poll showed similar national sentiment. Senate Bill 1248, which would have prohibited state employees from using automated decision systems as the sole basis for decisions, was killed last month in the appropriations process.
"The report is required under a 2023 law mandating that state agencies annually disclose their use of 'high-risk automated decision systems,' which the law defines as systems 'used to assist or replace human discretionary decisions that have a legal or similarly significant effect,'" the technology department's report states.
Why this matters for legal professionals
For attorneys and legal analysts, the disclosure of COMPAS and other recidivism-prediction tools raises immediate questions about due process and evidentiary standards. When a system's risk score influences sentencing, parole, or housing decisions, the methodology behind that score becomes discoverable. The report's omission of judicial branch systems - even as dozens of California courts adopt generative AI policies - creates a gap between the transparency law's intent and its reach. Legal professionals advising clients on administrative appeals or challenging benefit denials should track which agencies use automated fraud detection, since those determinations can carry the same legal weight as a human decision but with less accessible reasoning.