Anthropic said it disrupted several potential attempts this year by scientists who used its AI models to conduct research that could have helped develop biological weapons, according to a report the company released Sept. 10. The company said it could not determine whether the work was legitimate or malicious, and shut it down anyway.
The finding lands in an uncomfortable spot for researchers: the same biological inquiry that produces vaccines and treatments can also help engineer dangerous pathogens. Anthropic said that ambiguity drove its decision, because the cost of missing real malicious activity would be too high.
"You are not seeing someone in a comic book kind of way say, 'Hey, I want to build a biological weapon to kill everybody,'" Jacob Klein, Anthropic's head of threat intelligence, said in an interview. "It's an incredibly nuanced situation."
What the report describes
Anthropic's report catalogs misuses of its models. In the biological cases, the company said scientists used its leading models for research that could support weapons development. It did not name the users or say how many cases it found.
The company's difficulty is structural, not procedural. Legitimate and illegitimate biological research can look nearly identical from the outside, which makes automated detection unreliable and forces judgment calls about intent.
A risk that gets less attention than others
Experts rank biological misuse among the gravest concerns about AI, alongside catastrophic cyberattacks and AI agents that fail to align with human intentions. It has drawn less public attention than those threats, in part because past examples have surfaced mainly in research settings rather than in the real world.
That may be changing. Andrew Weber, a senior fellow at the Council on Strategic Risks who reviewed Anthropic's report before its release, said the findings were "chilling examples of state-sponsored biological weapons developers tapping into the rapidly advancing capabilities" of leading AI models.
For scientists who use these models in day-to-day work, the report raises a practical question: which research queries will trigger a shutdown? Anthropic has not published the criteria it applies. The company's stated approach is to err on the side of caution, which means some legitimate work may be interrupted. Researchers working at the boundary of pathogen research and drug development have the most exposure to that tradeoff. Broader coverage of AI's role in scientific work is available through AI for Science & Research.
Why this matters for science and research professionals
If you run biological research through a commercial AI model, you are working inside someone else's risk calculus. Anthropic has shown it will cut off access when it cannot verify intent, and it will not always be able to tell you why. Document your research purpose before you query, keep records that distinguish legitimate work from anything ambiguous, and expect that some prompts in pathogen-adjacent fields will get flagged. For labs that depend on these tools, the safer path is to treat access as conditional rather than guaranteed. Research scientists who want to build fluency with AI tools while understanding their limits can start with this AI Learning Path for Research Scientists.
Your membership also unlocks:
