Self-Improving AI Requires Human Oversight, Researcher Says
As AI systems begin optimizing themselves, computer science professor Yi Fang at Santa Clara University is working to ensure they remain transparent and accountable to human values. His research explores how AI can discover better search algorithms while keeping humans in control of the process.
The shift from AI as a tool to AI that improves itself is happening now, Fang said. For decades, AI assisted humans in analyzing data and supporting decisions. Today, systems are moving toward self-optimization-and the pace is accelerating.
"When AI starts improving itself, the concern is that the human-in-the-loop can slowly become a human-out-of-the-loop situation," Fang said. The challenge is maintaining transparency, safety, and alignment with human goals as systems become more autonomous.
How the Research Works
Fang's current work uses a method called RankEvolve, which mimics natural evolution. The process begins with a defined search task and benchmark datasets where relevant results are already known.
Large language models then generate new ranking strategies-variations similar to genetic mutations. These ideas are converted into code and tested against standard search metrics. If an AI-generated algorithm outperforms existing methods, it gets selected for further refinement and another round of testing.
The critical difference: researchers don't automatically trust AI-generated ideas. They treat them as hypotheses to generate, implement, test, compare, and refine. Sometimes the AI proposes weak solutions. Other times it discovers algorithms that outperform human-designed methods.
"Much of this process can be automated, allowing the system to explore many possible solutions more quickly than humans could do manually," Fang said.
Why This Matters Now
The decisions made today about AI governance and design will affect society for decades. Waiting to address safety, fairness, and transparency risks building these concerns into systems at scale.
Fang's work on AI research and large language models reflects a broader challenge: advancing AI capabilities while ensuring responsible development.
His interest in the field traces back 20 years to research on search engines. He realized that search is itself a form of AI-understanding user intent, processing vast information, and ranking results by usefulness. That work had real human impact: search engines shape what people read, learn, and trust.
"That made the research meaningful to me," Fang said. "Over time, as AI became more powerful, I became interested in two connected questions: Can AI help us do research better, and how can we make sure these systems are developed responsibly?"
The Role of Students
Fang co-created a Responsible AI minor with philosophy professor Susan Kennedy. Students from engineering, humanities, and business backgrounds bring diverse perspectives to complex problems.
"That diversity is incredibly valuable," Fang said. "Working with students from different disciplines broadens how I think about AI's impact, and their creativity often leads to unconventional ideas that help advance the field."
The goal of his work is not simply to build more powerful AI, but to build systems that are fair, reliable, transparent, and beneficial to people and society.
Your membership also unlocks:
