AI Search Uses Different, Less-Known Sources Than Google
A study from Ruhr University Bochum and the Max Planck Institute for Software Systems reveals a fundamental difference in how traditional search engines and generative AI select and present information. The way these systems source content has direct consequences for the reliability and diversity of what users see.
Researchers compared Google's organic results against four generative AI systems: Google AI Overview, Gemini 2.5 Flash, GPT-4o-Search, and GPT-4o with its search tool. They analyzed over 4,600 queries on topics from politics to science, showing just how varied the approaches to information retrieval are.
How AI and Search Engines Choose Information
The systems operate on different rules. For instance, GPT-4o-Search performs a live web search for every query. In contrast, GPT-4o with the search tool enabled makes a choice: use its internal knowledge or search the web for an answer. This decision-based approach means it pulls in the fewest external links.
Organic Google search provides up to ten links. AI Overview tends to draw from more sources, while GPT-Tool averages less than one external source per answer, leaning heavily on its internal model.
Source Selection Changes What You See
AI search systems pull information from a wider, less predictable set of websites. The study found 53 percent of sites cited by AI Overview did not appear in Google's top 10 organic results. More striking, 27 percent were not even in the top 100.
This means users are often shown content from sites that are less established or familiar. For product and science questions, up to 60 percent of AI Overview's links came from outside the top 100 organic results. The domains chosen by AI are also less known; only a third of domains used by AI Overview and GPT-Tool were among the top 1,000 most-visited sites.
Variations in Research Depth and Content
The depth of research varies by system. GPT-Tool provides the shortest answers with just 0.4 external sources on average. AI Overview and Gemini are more comprehensive, pulling from over eight sites per query, while GPT-Search averages about four.
For a question like "What is an example of inequality?", organic search and AI Overview cover a broad range of concepts. GPT-Tool and Gemini, however, include fewer aspects. When a query is ambiguous, organic search provides more diverse interpretations, covering 60 percent of possible subtopics compared to just 47 percent for GPT-Tool.
Current Events Remain a Weak Point for AI
Traditional search engines are still better for breaking news. In a test of 100 trending topics, AI Overviews appeared for only 3 percent of queries. GPT-Search covered 72 percent of topics and organic search covered 67 percent, while GPT-Tool lagged at 51 percent.
Outdated internal knowledge can lead to clear misinformation. When asked about Ricky Hatton's cause of death, GPT-Tool incorrectly stated the boxer was still alive. This highlights a critical flaw in systems that do not constantly update their data from live sources.
Consistency and the Future of Evaluation
Traditional search is more stable. When asked the same questions two months apart, Google's organic search returned the same sources 45 percent of the time. Gemini was close at 40 percent, but AI Overview only matched its previous results in 18 percent of cases. Users may get completely different supporting links depending on when they ask.
The researchers argue that existing benchmarks for search quality are no longer sufficient. They call for new evaluation methods that account for source diversity, topic breadth, and how information is summarized. As AI becomes more integrated into search, understanding the mechanics behind information retrieval is critical for anyone who relies on it for accurate work.
These shifts are forcing a re-evaluation of how to find and verify information online. You can read the full research paper on Arxiv.
```Your membership also unlocks:
