Law Schools Test AI for Grading Exams, But Professors Disagree on How to Use It
Law professors are using artificial intelligence to grade student exams, with early research showing AI scores align closely with human grading. But the profession remains divided on whether machines should lead the grading process or simply check human work.
Jack Graves, a professor at Syracuse University College of Law, grades student work with ChatGPT and finds AI more consistent than himself. "AI doesn't get tired; it doesn't get distracted; it doesn't get frustrated," Graves said. "If I use it correctly, I am absolutely certain it is more consistent in scoring essays than I am."
A recent paper in the Journal of Law and Empirical Analysis tested OpenAI's GPT-5 against human professors grading final exams from four different law subjects. When given detailed rubrics, the AI's grades correlated with human grades at a Pearson correlation coefficient of up to 0.93-a significant alignment.
Daniel Schwarcz, a University of Minnesota Law School professor and co-author of the study, argues professors should grade first, then use AI to spot discrepancies. "You should first grade on your own without an AI," Schwarcz said. "And then you may want to use an AI to sort of see if there are any discrepancies and then double-check."
Graves takes the opposite approach. He now spot-checks AI work rather than vetting every grade. His method involves training the tool with rubrics, model answers, and sample assignments he's already graded, along with careful prompting to avoid hallucinations. Over 1,000 interactions across two semesters, he found only one clear hallucination.
The fairness question
Schwarcz refuses to let AI grade first for a procedural reason: students expect human graders. "Your students have an expectation that their exams will be graded by a human. It's not fair to deprive them of that opportunity without really getting their buy-in," he said.
Daniel W. Linna Jr., director of law and technology initiatives at Northwestern University Pritzker School of Law, disagrees. He argues the profession should move past prohibition and disclosure. "The tools are here. They're integrated in more and more places," Linna said.
Beyond exam grading
Graves protects student privacy by assigning numbers instead of names and keeping projects visible only to himself. He uses AI to analyze patterns in student performance-asking it to identify the 10 most commonly missed issues-then adjusts his teaching accordingly.
The study authors suggest AI could handle tasks beyond final exam grading: reviewing professor grading for validation, providing feedback on ungraded midterms, and helping students practice with self-administered exams.
Graves envisions AI administering progress tests during the semester. Students could retake tests multiple times until passing and file protests if they believe the AI treated them unfairly. "If you bomb it the first time, I don't see it. Do it again and again," he said.
For legal professionals exploring this shift, understanding both ChatGPT capabilities and broader AI applications in legal work provides practical grounding in how these tools function in practice.
Your membership also unlocks:
