Law Professors Rate AI Tutors Higher Than Peer Instructors in Contract Law Study
Law professors prefer AI-generated answers to student questions over responses written by fellow instructors, according to a study involving 16 professors across U.S. law schools. In nearly 3,000 blind comparisons, AI won 75% of head-to-head matchups.
The research tested whether large language models could serve as effective tutors for contract law courses. Professors flagged AI responses as pedagogically harmful only 3.5% of the time, compared to 12% for answers written by other professors.
Why This Matters for Legal Education
The study focused on law specifically because the field requires judgment, nuanced reasoning, and the ability to work through ambiguity-not just factual recall. Legal reasoning demands careful analysis of competing arguments and defensible conclusions, making it a rigorous test of AI capability.
Professors created 40 representative contract law questions students might ask during office hours, wrote their own answers, and then evaluated responses without knowing their source. The AI systems performed comparably to the best human instructor in the study.
Many questions required synthesizing complex material, applying it to new situations, and explaining legal concepts in ways that help students develop analytical skills. These weren't straightforward problems with obvious answers.
What Professors Found
The research team tested multiple AI models, including commercial tutoring systems and Google's NotebookLM, finding varying performance levels. Even when context limitations affected AI responses, professors still frequently preferred them to human-written alternatives.
Researchers took extensive precautions to ensure validity. They calibrated AI responses to match the length and structure of human answers, used multiple evaluation methods, and had professors assess whether responses might mislead or confuse students.
Implementation Questions Remain Open
The study evaluates the quality of answers AI tools produce, but how to implement these systems to most effectively improve student learning is still unclear. Law schools nationwide are grappling with integrating AI into legal education while maintaining rigorous academic standards.
Some institutions have embraced AI experimentation, while others remain cautious about potential risks including hallucinations, overreliance, and erosion of critical thinking skills. The conversation should shift from whether AI can provide accurate, high-quality responses to how institutions can deploy it responsibly for student benefit.
Blanket skepticism about AI tutoring may be as unwarranted as wholesale adoption without consideration. The data suggests a middle path exists.
For educators interested in AI for Education or AI for Legal fields, this research provides evidence that AI systems can meet professional standards when evaluated by domain experts.
Your membership also unlocks:
