AI Law Student Earns Top Grades on Real Law School Finals, Study Shows
OpenAI’s latest AI model, o3, scored between A+ and B on eight law exams, outperforming earlier versions like ChatGPT. It uses reasoning steps to generate polished legal answers.
Artificial Intelligence Achieves A+ on Law School Exams, Latest Study Shows
Generative artificial intelligence has made a significant leap in legal education performance. A recent study reveals that OpenAI’s newest model, named o3, scored between A+ and B on eight final exams administered by the University of Maryland Francis King Carey School of Law.
These grades represent a marked improvement over earlier AI versions such as ChatGPT, which previously earned a range of B's, C's, and even a D on similar exams in 2022 and 2023. Prior research had shown ChatGPT to deliver average results on law exams and speed up legal writing without enhancing its quality. Meanwhile, AI's ability to pass bar exams has been documented, but o3's performance suggests it's closing in on top law students.
How o3 Works Differently
Unlike ChatGPT, which instantly generates text based on a query, o3 operates as a reasoning model. It develops tentative answers, evaluates multiple approaches internally, revises them, and then produces a final, polished response.
Exam Performance Breakdown
Seven law professors graded o3's exam responses using the same curve applied to their students. The AI received an A+ in Constitutional Law, Professional Responsibility, and Property. It earned an A in Income Taxation and an A- in Criminal Procedure. The model scored a B+ in Secured Transactions and Torts, and a B in Administrative Law. It performed well on both multiple choice and essay portions.
Limitations and Challenges
One notable limitation was o3’s B grade in Administrative Law, largely due to its lack of knowledge about the 2024 U.S. Supreme Court decision in Loper Bright Enterprises v. Raimondo, which overturned the Chevron doctrine—a key element in administrative law. This ruling occurred shortly after the AI's knowledge cutoff.
Interestingly, when researchers provided o3 with professors’ notes during one exam, its performance declined. The team speculated that an overload of information distracted the model, hindering its ability to deliver concise answers.
Implications for Legal Education and Integrity
The study underscores how advanced AI models could challenge traditional assessment methods. The researchers plan to test whether instructing the AI to introduce spelling and grammar errors might make AI-generated exams less distinguishable from human work, highlighting potential concerns around academic integrity.
For legal professionals interested in integrating AI tools into their practice or education, exploring targeted AI training can be beneficial. Resources are available for understanding how these models function and how to leverage them effectively.
