Andrew Blair-Stanek (Maryland) & Donald B. Tobin (Maryland) et al., Is AI’s Law School Exam Performance Plateauing?:
Last spring, we had OpenAI’s reasoning model o3 take our final exams, with the reasoning effort parameter set to “high,” and graded its answers on the same curve as our students. o3 got grades ranging from A+ to B, and we noted that there were ‘clear, fixable explanations for two of o3’s lowest grades.’ [AI Gets Its First Law School A+s (And An A In Tax)] This spring, we repeated the experiment, using OpenAI’s latest reasoning model, GPT-5.5, with the reasoning effort at the new “xhigh” setting. GPT-5.5 got two A+s, three As, two As [including in tax], and a B+, a good performance but far short of superhuman. Depending on the metric, GPT-5.5 may have actually performed worse than o3 did last year, despite the new “xhigh” setting. These results may fit the broader pattern of frontier AI models’ performance plateauing on other legal benchmarks.
Conclusion. OpenAI’s newest frontier model, GPT-5.5, running with the new “xhigh” reasoning effort, did not noticeably outperform last year’s o3 model on our exams – and arguably performed worse, when considering that o3’s second-worst grade reflected human error, and its worst grade (Administrative Law) reflected that Chevron’s overruling came after its training cutoff. This apparent plateau, despite a better model and using extra-high inference-time compute, may reflect trends observed in other legal benchmarks for AI and makes theoretical sense. Of course, this may reflect noise in our small sample set, and new technological advances may cause AI’s performance on law school exams to resume its clear upward march. We plan to continue these experiments in future semesters.
Editor’s Note: If you would like to receive a daily email with links to legal education posts on TaxProf Blog, email me here.
Association of American Law Schools
125 Years of Advancing Excellence in Legal Education
1614 20th St NW
Washington, DC 20009-1001
(202) 296-8851
The views expressed in books, articles, op-eds, letters to the editor, blogs, blog comments, and other sources linked to and excerpted on TaxProf Blog are those of the authors and are not those of AALS and its member schools, Paul Caron, or Pepperdine University. Inclusion of links to and excerpts from books, articles, op-eds, letters to the editor, blogs, blog comments, and other sources does not constitute an endorsement by AALS, its member schools, Paul Caron or Pepperdine University.
© 2025 Association of American Law Schools
Subscribe now to keep reading and get access to the full archive.
Continue reading
Leave a Reply