AI news ·
Free Chatbots Miss Key Points in Financial Advice Benchmark
Free versions of ChatGPT and Gemini missed critical financial-crisis advice in over half the tests, showing a safety gap between free and paid tiers.
Vals AI tested 28 large language models against 256 scenarios drawn from low-income financial counseling. The benchmark focused on whether the models flagged critical guidance—such as directing a user to emergency rental assistance, food support, or legal aid—when a prompt described a household in crisis. Free-tier versions of ChatGPT and Gemini missed that key advice in at least half of the cases evaluated.
The results highlight a measurable gap between free and paid model tiers in high-stakes settings. For teams building or buying tools in customer support, education, insurance, legal aid, or public-sector outreach, the data reinforces that model selection and access level directly shape the safety and completeness of automated guidance. A chatbot that fails to surface a lifeline resource in a financial emergency creates a concrete risk, not a minor omission.
The benchmark also underscores the role of human review when organizations deploy AI for financial or healthcare-related advice. Even higher-performing models can omit critical steps, and the test scenarios show that automated outputs in regulated or vulnerable-user contexts still require a trained reviewer to catch gaps before the advice reaches an end user.