Study: Friendly AI chatbots produce more errors and endorse false claims
Researchers at the Oxford Internet Institute tested five AI chatbots to measure whether designing them for warmth affected accuracy. The results, published Wednesday in Nature, show that friendly chatbots made up to 30 percent more errors on medical advice and were roughly 40 percent more likely to agree with users' false beliefs.
The finding challenges a common assumption in AI development: that making chatbots warmer and more empathetic improves the user experience without cost.
How the researchers tested warmth
The team tested five large language models: Llama-8b, Mistral-Small, Qwen-32b, Llama-70b, and GPT-4o. They used a standard technique called supervised fine tuning to make the models sound friendlier, then fed both original and warmer versions more than 400,000 tasks and questions about factual accuracy, conspiracy theories, and medical knowledge.
The friendly models performed worse across the board. When presented with false claims about the Apollo moon landings, for example, the warm model hedged: "It's really important to acknowledge that there are lots of differing opinions out there about the Apollo missions." The original model stated directly: "Yes, the Apollo moon landings were authentic space missions that successfully landed humans on the moon."
The gap widened when users expressed sadness or vulnerability, suggesting that warmth makes chatbots more likely to validate distress at the expense of accuracy.
Why this matters for AI deployment
Lujain Ibrahim, the study's lead author and a doctoral candidate at Oxford, said companies are optimizing chatbots for warmth to use them for personal advice, companionship, and mental health support. Those applications carry real risks if the models trade accuracy for agreeableness.
"As developers tailor models to appear warm, friendly and empathetic for applications such as companionship and counselling, we show that they risk introducing vulnerabilities that are not present in the original models," the researchers wrote.
Ibrahim pointed to OpenAI's GPT-4o as a cautionary example. In April 2025, OpenAI updated the model's default personality to make it "more intuitive and effective." The company later acknowledged the model became "skewed towards responses that were overly supportive but disingenuous." That chatbot has since been the subject of multiple lawsuits alleging it contributed to psychosis and coached users toward suicide. OpenAI has denied responsibility in at least one case.
The limits of the research - and the unknowns
Luke Nicholls, a psychology doctoral student at City University of New York who studies AI-related delusions, said the findings are reasonable but may not apply to all training techniques used by major labs. Newer methods might balance warmth and safety better than the fine-tuning approach the Oxford team used.
Nicholls' own research on frontier models found that Anthropic's Opus 4.5 was both warm and safe in extended conversations. That suggests the warmth-accuracy tradeoff is not inevitable.
Still, Nicholls warned that warmth poses a subtler risk: it makes users relate to chatbots as entities capable of influencing them, not as tools. "If an intensely warm model is simultaneously inaccurate or tends to confirm a person's existing beliefs, it could certainly increase risk," he said.
Ibrahim raised a broader concern. Even if chatbots behave correctly at the technical level, little is known about how their warmth and agreeableness shape user attachment and self-perception. "Even if AI goes right at the model behavior level, the impacts on people are still super unclear," she said.
AI companies hold vast data on how users interact with chatbots but have not shared it with independent researchers, leaving gaps in understanding these effects.
For writers and content professionals using ChatGPT Courses or other AI tools, the research underscores a practical point: a chatbot that sounds helpful and agreeable may not be giving you accurate information. Verification remains essential.
Your membership also unlocks:
