AI news ·
Scale AI and Singapore IMDA Benchmark Finds Models Miss Facts Needed for Legal Answers
A legal AI benchmark found the best model met only 69.8% of required criteria and failed to ask clarifying questions 39% of the time.
A new benchmark from Scale AI and Singapore’s Infocomm Media Development Authority (IMDA) tested 19 language models on legal tasks and found significant gaps in factual completeness. The top-performing model met only 69.8 percent of the criteria required for a legally sound answer. The evaluation also revealed that models failed to ask necessary clarifying questions roughly 39 percent of the time, proceeding with incomplete information instead of probing the user’s intent.
These results indicate that fluency alone is an insufficient measure for legal AI applications. The benchmark suggests that evaluation frameworks should explicitly measure a model’s ability to gather information before delivering an answer, not just the polish of its final output. For professionals in insurance, healthcare, customer support, and other regulated fields, the findings underscore the risk of deploying assistants that sound authoritative while missing critical facts.
The work points to a need for testing protocols that treat information gathering as a core competency, particularly where incomplete answers carry compliance or liability consequences.