A six-month review of AI chatbots used by two legal organizations found that while many users received actionable help, a small but dangerous share of answers could cause real harm. Duke Law professor Keith Porcaro and his algorithmic auditing class analyzed sessions from a Legal Aid of North Carolina (LANC) bot and a Nevada court system tool, rating responses on a five-point scale that ranged from "helpful and actionable" to "harmful." In the Nevada audit, 1% of 1,462 conversations were rated harmful-a figure that carries weight when the topics include domestic violence, eviction, and child custody.
The work is part of a growing effort to bring auditing rigor to legal self-help AI before flawed advice reaches people who act on it. "There's so much pressure out there to 'do something with AI' that it's easy to just do something simple," Porcaro said. "Organizations risk skipping the step of asking 'What does a good version of this look like, and how do we know that it's going to be good?'"
Real-world audits shape better tools
LANC's chatbot, called LIA (legal information assistant), launched as a minimum viable product. Porcaro's class combed through data from the first six months of use and delivered an audit report with feedback that LANC used to refine the bot. The report "helped focus LIA's evolution from a minimum viable product into a robust and continuously improving public resource," said LANC board chair Jeff Kelly. The organization also authorized public release of the audit so other groups building similar tools could learn from the findings.
In the Nevada audit, the class developed and applied its own rating methodology, scoring answers from helpful and actionable down to harmful. About 46% of responses earned the top rating, while 1% fell into the bottom category-answers that were incorrect or misleading and might cause harm if followed. The class recommended improving the bot's ability to detect high-risk situations and provide more specific resources, such as directing users to call 911 or child protective services.
Confidentiality and competence as design requirements
Porcaro pointed to two core principles from legal professional responsibility-confidentiality and competence-as essential to building trustworthy legal AI. That means shielding client data from commercial use and government subpoenas, not just relying on a disclaimer that the bot "only provides information." "People do disclose sensitive information, so having good filters and good data governance practices is critical," he said.
Competence, in this context, also means planning for user errors. When someone describes a personal legal problem to a bot, the story often arrives jumbled, emotional, or full of irrelevant detail. That messiness can cause the bot to return imprecise or wrong information. "Clients bring messy stories, they bring emotional stories, they bring incomplete stories," Porcaro said. "As we're building these tools, we need to shift the way we think about deploying software, from where users are left on their own no matter what, to making sure that even if a user misinterprets something … it's not going to be entirely on them to catch a mistake before they file at court and lose their rights."
The unresolved tradeoff between help and harm
Porcaro framed the current moment as an experimentation phase, where the question is not whether to use AI but how a given tool measures up against other interventions a legal organization might choose. "The answer is not AI or nothing. The answer is how it measures up compared to all the other information or tech interventions you could use," he said.
That comparison forces a difficult judgment. "If you're helping a certain number of people but a small slice of people are being harmed, is that an appropriate tradeoff? I don't know that we've really found the right balance of benefit and harm yet," Porcaro said. "But the goal is to add enough detail so our partners can make that decision." Courts and legal aid groups, he said, need enough data from pre-launch stress testing and ongoing monitoring to decide whether a chatbot's risks are acceptable.
Why this matters for legal professionals
For lawyers, paralegals, and court administrators, the audits deliver a clear message: deploying AI in legal services without built-in auditability and continuous review invites liability and risks client harm. The same rigor that applies to human legal work-confidentiality, competence, and error-checking-must be engineered into chatbots. Legal teams considering or already using such tools should ask for audit frameworks, pre-launch testing, and a plan for ongoing monitoring before they go live. For those building competency in evaluating these systems, Complete AI Training's AI for Legal page covers skills in compliance automation and document review that align with the audit discipline Porcaro's work demands.
Your membership also unlocks:
