Can a chatbot earn a JD? This one averaged C-plus on law school exams
Image from Shutterstock.
An artificial intelligence tool called ChatGPT averaged a C-plus on exams at the University of Minnesota Law School, according to four law professors who gave it a try.
The law professors used ChatGPT to answer the questions and then blindly graded the answers, along with answers by real students, report Reuters and Insider.
The average C-plus grade was still below that of law students, who had a B-plus average. And ChatGPT’s performance, while earning passing grades, was at or near the bottom of the class.
The professors’ findings are available here.
Reuters described ChatGPT as a “wildly popular” free AI tool that “generates sophisticated, humanlike responses based on requests from users and mountains of data, including from legal texts.”
ChatGPT earned its highest grade of a B in a constitutional law class focusing on federalism and separation of powers but still placed 36th out of 40 students. It got a B-minus in employee benefits (placing 18th out of 19 students), a C-minus in taxation (placing 66th out of 67 students), and a C-minus in torts (placing 75th out of 75 students).
The chatbot’s performance on multiple choice questions was generally worse than the essays, but the number of right answers was better than chance. It did the worst on questions involving math, dragging down its score on the taxation exam.
ChatGPT did better on essay questions, but its performance was “highly uneven,” the study found. Sometimes it matched or even exceeded the performance of real students. Its essays “were typically clear and well-crafted—perhaps even suspiciously so compared to real students writing a time-limited exam,” the study said. And ChatGPT “displayed a strong grasp of basic legal rules.”
But when the essay answers were incorrect, “they were dramatically incorrect, often garnering the worst scores in the class,” the study said. ChatGPT “struggled to identify relevant issues and often only superficially applied rules to facts as compared to real law students.”
Lead study author Jonathan H. Choi spoke with Reuters about the findings.
“Alone, ChatGPT would be [a] pretty mediocre law student,” Choi said.
But he and the other authors of the study pointed out that ChatGPT was competing against students at a “highly selective” law school, and its grades were sufficient to earn a JD degree.
See also:
ABAJournal.com: “AI program earned passing bar exam scores on evidence and torts; can it work in court?”
ABAJournal.com: “Should lawyers embrace or fear ChatGPT?”
