AI Is Answering Surveys Instead of People. That's a Problem.
Pollsters are replacing human respondents with AI-generated synthetic answers to cut costs. A 10-minute survey of 1,000 people typically costs tens of thousands of dollars. AI can simulate thousands of responses for a fraction of that price. But researchers warn the practice conflates simulation with measurement-and the difference matters.
The method works by prompting a large language model with demographic details. A pollster might write: "You are a young college-going urban voter with conservative political views. Respond to the following questions." The model generates different answers each time, allowing researchers to create thousands of varied synthetic responses from a single prompt.
This approach, called synthetic surveys or silicon sampling, is already in use. But studies show synthetic respondents are unusually sensitive to small changes in prompts or settings, producing sharply different results from minor variations.
Simulation Is Not Measurement
The core problem goes beyond statistical reliability. Surveys serve as measurement tools meant to capture what people actually think. A thermometer measures temperature directly. You would not trust one that estimated temperature by consulting an AI model instead.
Large language models inherit biases from their training data. They can oversimplify or distort opinions from groups underrepresented online. Traditional polls have biases too, but many biases in modern AI systems remain hidden inside closed proprietary models.
When researchers present synthetic responses to the public as survey results, they obscure this distinction. The public sees poll data without knowing whether it came from people or machines.
This creates a practical risk. A self-driving car trains on synthetic images but undergoes extensive real-world testing before deployment. Researchers may treat synthetic survey responses as public opinion without checking them against reality. If the simulated opinions distort what people actually think, flawed conclusions could shape public policy, business decisions, or scientific research before anyone detects the error.
Where AI Can Actually Help
AI tools can improve survey research without replacing human respondents. They can help researchers write clearer questions by simplifying wording and reducing ambiguity. They can identify unnecessary questions that discourage participation.
After surveys are completed, AI can organize large volumes of open-ended responses, summarize recurring themes, and handle incomplete data more efficiently than human analysts. Some researchers are testing hybrid approaches that combine smaller human surveys with AI-assisted analysis.
These applications support survey work without weakening the measurement of public opinion itself. The challenge is keeping the human voice at the center while using AI to make the process more efficient.
Falling response rates and rising survey costs are real problems. But the solution is not to stop measuring what people think. It is to find ways to measure it better.
For researchers working with survey data, understanding these distinctions is critical. AI Data Analysis Courses and AI Research Courses can help professionals develop the skills to evaluate AI tools responsibly in research contexts.
Your membership also unlocks:
