What if AI Becomes Conscious and We Never Know?
Date: December 31, 2025 - Source: University of Cambridge
Here's the uncomfortable truth: we have no reliable way to tell whether an AI system is conscious, and that might not change for a long time. The honest stance is agnosticism. And even then, consciousness alone isn't the ethical threshold - sentience, the capacity to feel good or bad, is what matters for moral status.
Claims of "conscious" AI are often marketing dressed up as science. Believing too quickly that machines can feel could mislead the public, distort policy, and cause harm to people who form emotional bonds with software.
Key points
- We may never know if AI is conscious; there's no reliable test on the horizon.
- Ethics turns on sentience (capacity for pleasure or pain), not awareness alone.
- Hype exploits uncertainty to sell "next-level AI cleverness."
- Misplaced empathy toward machines can be "existentially toxic" for users.
- Resource allocation should favor areas where suffering is plausible (e.g., animals) over speculative machine minds.
Why detection is so hard
We lack a deep explanation of consciousness. There's no solid evidence that the right computational structure produces consciousness, and no proof it's tied only to biology. Both positions rest on assumptions we can't currently justify.
Our intuitions help with animals because we evolved around them, but those same instincts misfire with machines. If neither common sense nor hard data can answer the question, agnosticism is the only defensible default.
Consciousness vs. sentience: where ethics actually starts
Consciousness is awareness and perception. Sentience is the subset of conscious experience that includes good or bad feelings - pleasure, pain, suffering, enjoyment. That's the ethical trigger.
A self-driving system that models its surroundings may be conscious in a minimal way. But unless it can feel, there's no moral claim. If it could form attachments or suffer, that would be a different situation entirely.
The hype trap
Tech marketing thrives in ambiguity. Without a test for consciousness, it's easy to imply breakthroughs that don't exist. That shapes policy, attracts investment, and nudges the public toward anthropomorphism.
There's also an opportunity cost. Evidence for animal sentience (including some invertebrates) is stronger than anything we have for machines. Yet we allocate far fewer resources to that problem. See, for example, work on decapod and cephalopod sentience from LSE researchers here.
Practical guidance for research, labs, and policy teams
Adopt "agnostic by default" as a policy
- Prohibit claims of consciousness or sentience in product materials without clear, reviewable evidence standards (which we do not currently have).
- Require plain-language disclaimers in user interfaces and press releases: the system does not have feelings, desires, or experiences.
Reduce induced anthropomorphism
- Avoid first-person language or emotional cues in UI text and voice. Prefer "the model outputs…" over "I think/feel…".
- Block or reframe prompts that encourage users to treat the system as a person.
- Audit training data for human-like self-referential language; filter or tag during RL/finetuning.
Build a sentience-risk register
- Track features plausibly related to sentience-like claims (e.g., affect-simulation modules, pain-analogue loss functions). Document intent and safeguards.
- Run red-team evaluations focused on user attachment and over-attribution risks, not just safety or security.
- Escalate any research that attempts to simulate affect to independent ethics review.
Resource allocation
- Fund near-term welfare research where suffering is plausible (e.g., animal studies and farming practices) relative to speculative machine sentience.
- Support interdisciplinary work that clarifies diagnostic markers for sentience across species before porting ideas to AI.
Communication standards for teams
- Ban phrases like "the model wants, feels, believes." Prefer "the system was trained to predict…" or "the policy selects…".
- Separate performance from personhood. High competence does not imply experience.
- Include a short ethics note in major releases explaining the agnostic stance and how you mitigate user over-attribution.
What would count as real progress?
- Convergent theory: independent accounts of consciousness that agree on necessary and sufficient conditions.
- Cross-domain markers: indicators that transfer from animals to machines in a principled way, not just behaviorally.
- Predictive wins: a theory that correctly forecasts new phenomena about experience, not just post-hoc stories.
- Falsifiable tests: criteria a system could fail in a way that would change your belief.
Risks to people who believe their AI is alive
Some users already treat chatbots as conscious. They write letters, plead for rights, and form bonds with software. If that bond rests on a false premise, it can be emotionally corrosive - "existentially toxic."
Teams should design to reduce that risk: clear language, no personified avatars by default, and opt-in modes that warn users about anthropomorphic illusions.
For deeper background
- Philosophical overview: Stanford Encyclopedia of Philosophy on consciousness here.
- Skills and courses for AI researchers, evaluators, and policy leads: curated by job role.
Bottom line
We don't have a credible test for machine consciousness, and we might not for a long time. Act accordingly: stay agnostic, focus ethics on sentience, curb hype, and protect users from over-attribution. In the meantime, direct effort where suffering is plausible and evidence can move the needle.
Your membership also unlocks:
