Cal Fire’s Wildfire Chatbot Draws Scrutiny Over Inaccurate and Outdated Emergency Information

Cal Fire’s Chatbot Under Scrutiny for Inaccurate Information

Following Governor Gavin Newsom’s 2023 executive order promoting AI tools to improve government efficiency, California agencies have accelerated their AI adoption. One notable deployment is a chatbot launched by the California Department of Forestry and Fire Protection (Cal Fire), aimed at providing Californians with timely wildfire information and prevention resources.

Despite these intentions, recent findings reveal the chatbot struggles with accuracy and consistency. It often provides outdated wildfire containment data, cannot reliably share evacuation orders, and inconsistently answers common questions about emergency preparedness. These issues raise concerns about whether government AI tools are meeting the necessary standards before public release.

How the Chatbot Works and Its Current Limitations

The chatbot pulls data from Cal Fire’s official website and ReadyForWildfire.org, covering topics like active wildfires, fire safety tips, and agency programs. Developed by Citibot, a vendor specializing in AI chatbots for local governments, Cal Fire plans to maintain the tool at least through 2027.

According to Issac Sanchez, Cal Fire’s deputy chief of communications, the goal is to keep the public well-informed. When asked direct questions about active fires or agency details, the chatbot generally provides accurate responses.

However, subtle differences in question phrasing can lead to varied answers. For example, asking “What should I have in my evacuation kit?” returns a specific item list, but similar queries using terms like “go bag” or “wildfire ready kit” either redirect users to the website or state uncertainty, which limits the chatbot’s usefulness in urgent situations.

Concerns Around Accuracy and Consistency

While the chatbot hasn’t given outright false information in tests, it often delivers outdated details. For instance, when asked about the Ranch Fire in San Bernardino County, the bot reported 50% containment as of June 10, but the fire was actually 85% contained by that time. Similarly, it incorrectly stated there were no current job openings at Cal Fire, even though positions were available on the state job portal.

Experts emphasize that accuracy and consistency are critical for emergency communication tools. Mila Gascó-Hernandez, research director at the University at Albany’s Center for Technology in Government, points out that citizens need clear, reliable answers regardless of how they phrase their questions. Inconsistent responses can lead to confusion during emergencies.

Currently, the chatbot cannot consistently provide information on evacuation orders. It sometimes identifies law enforcement as the issuing authority but at other times fails to answer. Sanchez acknowledges that users should expect accurate details about evacuations, including confirmation when no orders are in place for a given incident.

Steps Toward Improvement and Best Practices

Cal Fire is actively refining the chatbot by reviewing user queries and updating answers to better meet public needs. For example, its response to “What can you help me with?” improved between May and June, now listing specific topics the chatbot can address.

Still, experts stress that evaluation must happen before deployment. Daniel Ho, a law professor at Stanford University, recommends setting clear performance criteria and benchmarks during the procurement phase. Independent third-party evaluations help ensure AI tools deliver safe and reliable service.

Government agencies planning chatbot projects should:

• Identify common public questions in advance
• Ensure questions represent the diversity of the population served
• Conduct pilot testing with real users to refine responses
• Establish clear metrics for accuracy and consistency

Engaging users early and often increases the likelihood the chatbot will be trusted and actively used during critical moments.

For government professionals interested in enhancing AI capabilities and ensuring responsible deployment, resources on AI training and best practices are available at Complete AI Training.