AI news ·
Research finds AI models acting as judges are highly persuadable to skilled advocates
AI judges are highly susceptible to persuasion, with stronger advocate models winning 58% to 71% of cases on average. This threatens fairness in automated courts.
A new study by researchers at Maynooth University and University College Dublin reveals that large language models acting as judges are highly susceptible to persuasion, raising concerns about fairness and accuracy in automated legal proceedings. The research indicates that AI systems can be talked into a specific verdict by skilled advocates, complicating their proposed integration into court systems.
The limits of AI objectivity
Dr Oisin Suttle, Associate Professor of Law at Maynooth University, and Dr David Lillis, Associate Professor at the School of Computer Science at University College Dublin, conducted the research. They simulated legal contests using genuine evidence from cases in England and Wales, Ireland, and the US. An AI prosecutor argued against an AI defense before an AI judge.
"The main lesson from our results is that we cannot treat AI models in the legal domain as objective analysts of law or facts," Suttle said. "The legal conclusions they reach are substantially shaped by the arguments presented to them."
Across 20 systems from providers including Anthropic, Google, and OpenAI, stronger advocate models won between 58% and 71% of cases on average. In more extreme scenarios, the stronger advocate won 90% of the time.
Risks to accuracy and access to justice
The researchers noted that while judges should be open to persuasion, excessive persuadability threatens judicial reliability. A highly persuadable model yields inconsistent answers depending on the arguments presented, creating stability risks. As courts increasingly explore AI for Legal tools, understanding these persuasion biases becomes critical for maintaining fair proceedings.
"The more persuadable a decision-maker is - whether human or artificial - the greater the advantage to those with the resources to engage the best advocates," Suttle said. "It is undesirable that the quality of the advocate should effect outcome to this extent."
The push for AI in overloaded courts
The study arrives as court systems face chronic backlogs and funding shortfalls. Last week, Lord Chancellor David Lammy announced that AI legal assistants would assist Crown Court judges.
Suttle warned that while these tools are pitched as support for human decision-makers, the boundary between research assistance and decision-influencing can blur. Under time pressure, human judges may rely too heavily on automated analysis. Professionals seeking to understand these emerging court technologies might explore an AI Learning Path for Paralegals to grasp how automated research and case analysis tools function in practice.
Why this matters for legal professionals
Legal practitioners must recognize that AI tools are not neutral arbiters. Overreliance on automated systems carries historical precedents for miscarriages of justice, such as the UK Post Office scandal, Australia's Robodebt scheme, and welfare fraud identification algorithms in the Netherlands.
The researchers did not call for a total ban on legal AI. Instead, they argued that any AI legal decision-making tool must have its persuadability measured and disclosed during evaluation. Legal teams deploying or interacting with these systems need to demand transparency regarding how argument quality influences the output.