Legal AI tools can now read contracts, extract obligations and summarise documents with impressive accuracy. The harder question is whether they can reproduce a decision from six months ago, under the rules as they stood then. In most deployments, the honest answer is that nobody knows.
That consistency problem is the focus of a new analysis from Shaz Aziz, Senior Director of Client Solutions at Neota Logic. Lawyers with no technical background can now build working legal AI tools in a chat window, Aziz said. The models are excellent at reading and conversing. But the technology that matters is not the model itself - it is the structured logic that sits behind it.
Two kinds of machine
Aziz draws a distinction between two types of AI that do different jobs. A language model is probabilistic: it produces a plausible answer, usually a very good one, and a slightly different one each time. A rules engine is deterministic: same inputs, same answer, every time, with a logic path you can print out and hand to someone to inspect.
"One machine reads and converses. The other makes decisions and remembers why. Legal work needs both, usually in the same workflow: the model reads the document, the rules make the decision, and where the risk warrants it the workflow stops for a person and will not move on without them."
That distinction matters for quality control. Recent hallucination cases that fell foul of professional standards all had a human in the loop somewhere - a paralegal, a clerk, local counsel. But presence is not a control, Aziz said. "A control says what the person must verify, evidences that they verified it, and blocks the next step until they have."
Where do your rules live?
When clients ask why they cannot simply build compliance logic inside an AI assistant, Aziz asks one question: where do your rules live?
If the answer is "in a prompt", there is a problem. Version control can track the words of a prompt, but not its behaviour. The underlying model changes without asking, so the same prompt in January and in June is not the same system. A rules engine versions the behaviour itself. The auditor's question is simple: can you reproduce a decision made in January under the rules as they stood in January? For a prompt-built system, the answer is no.
The scale of the risk is significant. A large insurer or firm is not making one decision but tens of thousands a day. Probabilistic logic that shifts without notice produces inconsistent decisions at scale - "the pattern class actions are built on", Aziz said. When one decision is successfully challenged, every decision made the same way is in play, and you cannot show they were made the same way.
What comes next
Aziz describes three stages for the future of legal AI. The first is AI working inside governed workflows: the model does the reading and extraction, the rules make the decision, and the audit record shows which steps were probabilistic and which were deterministic. This is shipping today, he said.
The second stage reverses the direction of travel. Instead of AI inside the workflow, the workflow becomes callable from wherever the lawyer already works, such as a chat interface. Ask a question with consequences, and the model invokes a deterministic application rather than guessing. The model makes exactly one decision: to call the workflow.
The third stage has AI helping author the rules themselves, making the deterministic layer cheaper to build and maintain. Aziz called that "a direction rather than a product today", and said he would be wary of anyone who claims otherwise.
Why this matters for legal professionals
For lawyers evaluating AI tools this year, Aziz's advice is to skip the model benchmarks. Ask where the rules live, who can change them, and whether the system can reproduce a decision made under last January's logic. "The vendors who can answer that will be happy you asked."
Legal professionals working with these tools - including AI for Legal training - should treat consistency as a core requirement, not a technical detail. For support staff involved in oversight, an AI Learning Path for Paralegals can help build the skills to verify AI outputs effectively. The question to put to any vendor: point at the workflow and say which steps are probabilistic and which are rules. If they cannot answer, the system is not ready for decisions that matter.
Your membership also unlocks:
