ChatGPT is asked 50 questions about Supreme Court; it got only 21 questions right
Image from Shutterstock.
SCOTUSblog has no worries that its coverage of the Supreme Court will be displaced by the artificial intelligence program known as ChatGPT.
The program had passing scores on two sections of a practice bar exam. And it had a C-plus average on four exams at the University of Minnesota Law School.
But it missed 26 out of 50 questions about the Supreme Court posed by SCOTUSblog. Google performed better, although it can’t generate multiple paragraphs of text or have a conversation, as ChatGPT can.
“ChatGPT’s mistakes varied widely,” SCOTUSblog reports. “Sometimes, it nailed the spirit of the question but misstated a factual detail or two, forcing us to mark the answer incorrect.”
One example: ChatGPT “provided a serviceable definition” for “CVSG,” an acronym for the Supreme Court calling for the views of the solicitor general.
But ChatGPT added “unnecessarily and inaccurately” that CVSGs can happen in “cases where the government is a party.” In fact, the purpose of a CVSG is to obtain the government’s perspective when it’s not a party to the case.
“Other errors were more serious,” SCOTUSblog reports. “When we asked it how many justices [former] President Donald Trump appointed (Question #35), it confidently asserted the answer was two: Neil Gorsuch and Brett Kavanaugh.”
“What about Any Coney Barrett?” SCOTUSblog asked ChatGPT.
ChatGPT apologized and said Trump had indeed appointed Barrett to the Supreme Court.
ChatGPT got two questions right about the late Justice Ruth Bader Ginsburg.
“But then, oddly, it claimed she wrote a dissent in Obergefell v. Hodges, the landmark decision that declared a constitutional right to same-sex marriage,” SCOTUSblog reports.
Ginsburg was in the five-justice majority in Obergefell. And she did not argue that the issue of same-sex marriage should be left to the states, as ChatGPT claimed.
When questioned about the error, ChatGPT flip-flopped.
“Who were the justices in the majority?” SCOTUSblog asked.
ChatGPT correctly listed the five justices, including Ginsburg.
“But you just said Ginsburg was in dissent,” SCOTUSblog pointed out.
“I apologize for the confusion,” ChatGPT replied. “Justice Ruth Bader Ginsburg was not in the majority in the Obergefell v. Hodges case. She dissented. The five justices in the majority were Anthony Kennedy, Sonia Sotomayor, Elena Kagan, Stephen Breyer and Clarence Thomas.”
