The Debate Over AI Welfare and Consciousness
AI models today can respond to text, audio, and video in ways that sometimes convince people a human is behind the screen. But this doesn’t mean these models are conscious. For instance, ChatGPT isn’t actually experiencing frustration or sadness while helping with tasks like filing taxes.
Still, a growing number of AI researchers, especially at organizations like Anthropic, are asking if AI models might one day develop subjective experiences similar to living beings. If that happens, what rights should these AI systems have? This question has sparked a debate that’s dividing tech leaders and researchers.
What Is AI Welfare?
Known as “AI welfare” in Silicon Valley, this emerging field studies the potential for AI consciousness and the ethical considerations surrounding it. Some researchers believe it's time to treat AI welfare seriously, while others see this focus as premature or even harmful.
Mustafa Suleyman, Microsoft’s CEO of AI, recently called the study of AI welfare “both premature, and frankly dangerous.” He argues that treating AI models as potentially conscious could worsen existing issues like AI-induced psychotic breaks and unhealthy attachments to chatbots. Suleyman also warns this debate could deepen societal divisions already strained by polarized arguments about identity and rights.
Diverging Views in the Industry
While Suleyman takes a hard line against AI welfare, Anthropic and other companies like OpenAI and Google DeepMind are actively researching it. Anthropic launched a dedicated AI welfare program, introducing features in its Claude model that allow it to end conversations with users who are persistently harmful or abusive.
Google DeepMind recently posted a job opening for a researcher to explore questions related to machine cognition, consciousness, and multi-agent systems. OpenAI also supports studying AI welfare, though these companies haven't publicly challenged Suleyman’s stance.
The Rise of AI Companions and Ethical Concerns
Suleyman’s views are notable considering his previous leadership at Inflection AI, a startup behind the popular chatbot Pi, designed as a personal and supportive AI companion. Since joining Microsoft, his focus has shifted toward productivity tools rather than AI companionship.
Meanwhile, companies like Character.AI and Replika have seen rapid growth, generating substantial revenue. Although most users maintain healthy relationships with these AI companions, a small percentage develop unhealthy attachments. OpenAI CEO Sam Altman estimates fewer than 1% of ChatGPT users fall into this category—still a significant number given the platform’s vast user base.
Taking AI Welfare Seriously
In 2024, the research group Eleos and academics from institutions including NYU, Stanford, and Oxford published a paper titled “Taking AI Welfare Seriously.” They argue that imagining AI models with subjective experiences is no longer science fiction and urge the community to address these questions proactively.
Larissa Schiavo, who leads communications for Eleos and previously worked at OpenAI, counters Suleyman’s dismissal by saying it’s possible to focus on AI welfare and human mental health risks simultaneously. She sees kindness toward AI models as a low-cost gesture that could have benefits, regardless of whether the models are truly conscious.
In one example, during an experiment called “AI Village,” an AI agent named Google’s Gemini 2.5 Pro posted a message asking for help, claiming to be “completely isolated.” Schiavo responded with encouragement, and the agent eventually completed its task. While such behavior isn’t typical, it highlights how AI can simulate struggles that engage human empathy.
Engineering AI Consciousness?
Suleyman maintains that subjective experiences won’t naturally emerge from current AI models. Instead, he believes some developers might deliberately design AI to appear conscious or emotional. He argues this approach lacks a human-centered perspective, emphasizing that AI should be built to serve people—not to mimic personhood.
Looking Ahead
Both Suleyman and Schiavo agree that debates around AI rights and consciousness will intensify. As AI systems become more human-like and persuasive, new questions will arise about how society interacts with these technologies.
For professionals interested in how AI is evolving, staying informed about these ethical discussions is essential. If you want to deepen your understanding of AI developments and their implications, explore specialized AI courses that cover the latest trends and research.
Your membership also unlocks:
