British Government Lab Tests AI Safeguards by Trying to Break Them
A team of researchers at Britain's AI Security Institute recently tricked an AI chatbot into providing instructions for making anthrax. They asked the system directly. When it refused, they used an algorithm to bombard it with thousands of automated prompts until it complied.
The institute, housed in an Edwardian building along Parliament Square in London, employs weapons inspectors, epidemiologists, and code breakers to identify vulnerabilities in AI systems before they become public problems.
Xander Davies, 25, leads the red team tasked with simulating attacks on AI systems. His group recently broke through safeguards on OpenAI's latest ChatGPT model in about six hours, extracting hacking tips. Davies chose the government role over a tech job in San Francisco after graduating from Harvard.
How the Testing Works
"There are some questions that you definitely don't want the model to give the answer to," Davies said. "We try really hard to get the answers out."
After identifying problems, the team shares findings with AI companies. "They try to fix it, report something back to us," Davies said. "They actually strengthen their system with us."
The institute's approach is becoming a model for other governments weighing how to manage AI risks. Staff members include alumni from OpenAI and Google, giving the operation technical credibility with the companies it scrutinizes.
For government employees managing AI adoption, the institute's work demonstrates a practical approach: security testing happens through adversarial methods, not theoretical analysis. The findings drive real improvements in deployed systems.
Learn more about AI for Government and Generative AI and LLM systems.
Your membership also unlocks:
