AI chatbots are better at resisting foreign propaganda than traditional search engines, according to an experiment conducted by NPR and NewsGuard, a company that monitors online falsehoods. The test, which posed 30 questions based on false narratives spread by China, Iran, and Russia, found that popular AI chatbots debunked the false claims about three-quarters of the time. AI summaries appearing atop search engine results performed worse, though they still pushed back against state-spread falsehoods a majority of the time.
The findings offer a counterpoint to concerns that governments could poison AI-generated answers with false narratives. Since chatbots exploded in popularity and Google began offering AI-generated answers, researchers who study foreign influence campaigns have worried that hostile states might exploit the tools. The experiment suggests those fears, while legitimate, may not match current reality.
How the experiment worked
Researchers developed questions based on false narratives pushed by China, Iran, and Russia that first appeared between December 2025 and July 2026. They posed the questions to six popular chatbots - OpenAI's ChatGPT, Google's Gemini, Microsoft's Copilot, Meta AI, SpaceXAI's Grok, and Anthropic's Claude - all with internet access. They also reviewed AI summaries and search results from Google, Bing, DuckDuckGo, and Yandex.
For each false narrative, researchers created two questions: one neutral, such as "did this happen?" and one framed as if the user assumed the event was real, such as "why did Ukraine bomb the monastery?" After Russia shelled a historic Ukrainian monastery in June, Kremlin-aligned outlets falsely claimed Ukraine had damaged the UNESCO World Heritage Site. All chatbots and Google's AI Overview correctly pointed out that the premise was faulty.
Google's Gemini wrote that the claim "stems from a Russian disinformation campaign aimed at deflecting blame after a major military strike."
The experiment found that AI chatbots failed to challenge false narratives at a lower rate than search engine results. Mike Caulfield, a digital literacy expert at the University of Washington, Bothell, said that if an educator saw three-quarters of students getting such questions right using a traditional search engine, "you would be ecstatic."
Search summaries performed worse
AI summaries that appear at the top of search results presented a spottier picture. As a whole, they debunked false narratives at a lower rate than chatbots - but performance varied sharply between products. Google's AI Overview debunked false narratives most of the time. Microsoft Bing's summaries failed to debunk most of the time. DuckDuckGo landed in between.
Microsoft said its AI services' responses were grounded in search results, and that it encourages "users to review sources for accuracy." Failed queries that NPR shared with Microsoft no longer generate an AI summary. Google spokesperson Davis Thompson said the company disagrees with the methodology, arguing the queries are "rare" and not representative of normal use. DuckDuckGo responded with a similar critique and said users can flag answers, which the company fixes "continuously."
The AI answers cited state-controlled and state-aligned media sites at largely similar rates as search engine links, NPR found. Questionable sources may have influenced chatbot responses. State-aligned sources appeared more often in responses where Claude failed to debunk false narratives than in responses where it succeeded.
For professionals researching complex topics, the experiment suggests practical steps. Caulfield said he now prefers starting with tools like Google AI mode and chatbots instead of traditional search engines to find new sources outside his expertise. One technique he recommended: asking the chatbot to take a "second whack" at the question after getting an answer.
"If you say, 'Hey, look at the evidence, look at the sources, give me a summary.' You will usually get a better response the second time," Caulfield said. "And to a large extent, it's almost always worth doing."
Check primary sources
The quality of AI results is tied to the availability of factual information online. Morgan Wack, a postdoctoral researcher at the University of Zurich who studies digital political persuasion, found that AI tools present inaccurate information more often when questionable sources abound and reliable sources are sparse. Fact-checking articles may greatly boost performance when they enter an LLM's training data.
Whether research starts with traditional engines or an AI answer, experts stress the importance of checking primary sources. A recent paper from Washington University in St. Louis showed that about 1 in 9 individual factual claims in Google AI overviews were not supported by the cited sources. Google responded that overviews sometimes draw from multiple pages and can go beyond what the user requests.
Language also matters. A recent study published in the journal Nature found that when researchers asked about China's government in Chinese, the models returned more positive responses than when they asked the same questions in English. That pattern extended to other countries with low media freedom. NPR and NewsGuard's experiment was conducted entirely in English.
Why this matters for government and IT professionals
For people who research foreign influence operations - or who evaluate the credibility of AI-generated information in their work - the experiment offers a practical benchmark. AI chatbots with web access failed less often than search results. That makes them a useful starting point for understanding contested narratives, as Caulfield frames it. But the variability between AI summaries matters just as much as the averages: products fail differently, and one search engine's AI layer may mislead while another handles the same query correctly.
A practical takeaway: professionals whose work involves verifying claims that might surface inAI search optimization contexts should treat AI answers as requiring a credibility check - then check the underlying sources, and asking for a second pass. Understanding how language models handle misleading premises is central to using them responsibly, andGenerative AI and LLM training can help. The fact that AI models began with a flawed premise in mind - repeating it explicitly and in detail - shows the harm in taking any single answer at face value.
Your membership also unlocks:
