Law Professors Rate AI Tutors Higher Than Peer Instructors
Law professors overwhelmingly preferred answers generated by AI over those written by fellow professors in a Stanford University study, rating artificial intelligence responses as superior in 75 percent of head-to-head comparisons.
The research, led by Stanford Law School Professor Julian Nyarko, tested whether large language models could serve as effective tutors for contract law courses. Sixteen law professors from across U.S. law schools evaluated nearly 3,000 anonymized comparisons without knowing whether answers came from AI systems or other participating professors.
The professors flagged AI responses as potentially misleading or pedagogically harmful only 3.5 percent of the time, compared to 12 percent for answers written by their peers.
Why This Matters for Legal Education
Legal reasoning demands judgment and analysis of competing arguments-not just factual recall. The study examined whether AI could meet professional standards lawyers use to evaluate each other's work in a field where multiple defensible conclusions often exist.
"Two opposing arguments can both be good," said Sarath Sanga, co-author and professor at Yale Law School. "What we wanted to know is whether AI can meet the latent professional standard that lawyers use to evaluate each other's arguments. In this case, the answer was yes."
Professors created 40 representative contract law questions that students might ask during office hours or after class. They wrote their own answers, then rated responses without identifying their source. The AI systems performed comparably to the best human instructor in the study.
How the Research Was Conducted
The research team took precautions to ensure validity. They calibrated AI responses to match the length and structure of human answers, used multiple evaluation methods, and had professors assess whether responses might confuse students.
The study examined specific AI models, including commercial tutoring systems and Google's NotebookLM, finding varying levels of performance. Even when context limitations affected AI responses, professors still frequently preferred them to human-written alternatives.
Implications for Your Practice
The findings suggest AI tutoring systems can provide on-demand support that complements classroom instruction and may broaden access to expert guidance in legal education. This has potential applications beyond law schools, as firms and legal departments consider how to structure training and mentorship programs.
Learn more about AI for Legal and AI for Education to understand how these tools are being deployed across the profession.
Your membership also unlocks:
