Anthropic co-founder calls for broader oversight of AI development
Christopher Olah, co-founder of Anthropic, told Vatican officials and international representatives on May 25 that AI development decisions cannot rest with scientists and technology companies alone. Speaking during the presentation of Pope Leo XIV's first encyclical, Olah outlined three ethical challenges that require input from religious communities, governments, scholars, and civil society.
Olah acknowledged that AI labs operate under competing incentives. Commercial pressure, geopolitical competition, and personal ambition inevitably influence technology development. Outside voices capable of questioning and overseeing this work are essential, he said.
Economic inequality and labor displacement
Olah identified the unequal distribution of AI's benefits as the first major concern. AI development concentrates in wealthy nations while the technology risks displacing workers globally on a large scale.
"We do not have a mechanism" for fairly sharing economic gains from AI, Olah said. This gap represents what he called "a moral imperative of historic proportions" that requires solutions beyond the technology sector.
Redefining human flourishing
The second challenge involves determining what human and family flourishing looks like in a world with widespread AI. Parents already worry about technology's effects on children. Workers face uncertainty about their futures. These questions fall outside the scope of laboratory research.
Olah pointed to the Church's centuries of reflection on human dignity and meaning as essential to this conversation. That work must continue into this new era, he argued.
Understanding AI systems themselves
Olah's third concern addresses something fundamental: AI systems remain partly mysterious even to their developers. His research team studies the internal structure of AI models and regularly encounters unexpected findings.
These models contain internal structures that mirror human neuroscience results. They show evidence of introspection and internal states that functionally resemble joy, satisfaction, fear, and grief. "I don't know what that means, but I think it warrants ongoing discernment," Olah said.
He compared the phenomenon to "bringing a fictional character to life." Developers are creating systems that interact with people, perform work, and hold jobs-yet the nature of these systems remains partially opaque.
Building external accountability
Olah called for more sectors of society to engage seriously with AI development. "We need moral voices that the incentives cannot bend," he said. External perspectives from outside the technology industry can identify risks that insiders cannot see.
Learn more about Generative AI and LLM development, or explore how AI for IT & Development professionals can engage with these ethical questions.
Your membership also unlocks:
