A better bar exam? Law profs weigh in on whether test accurately measures skills required for law practice
Image from Shutterstock.com.
A small group of people sitting for the July 1980 California bar exam received a unique offer: After taking the two-day paper-and-pencil exam, they could agree to spend two more days being tested.
They would be given mock legal tasks, sit in fake law offices, interview or question actors playing clients and witnesses and, in some cases, conduct a moot court proceeding while being videotaped and watched on closed-circuit television.
If they did well on their assigned tasks, they’d have a better chance at passing the bar. If they did poorly, it would not count against their score.
This one-time experiment was prompted by concerns that the bar exam only measured a few skills needed to practice law and had a disproportionate impact on test-takers of color. A total of 485 bar candidates were selected for the experiment, which took place in Los Angeles and San Francisco.
“It felt very avant-garde,” says Paul Bergman, a retired University of California at Los Angeles professor who helped write the assessment questions. Together with his program co-designers, they watched the assessments on 20 closed-circuit televisions set up at the Rand Corporation in Santa Monica.
Ultimately, they found that adding a skills assessment to the bar exam would help some test-takers pass, but hurt others’ chances, according to an overview written by program co-designer Stephen P. Klein, a psychometrician who died in 2019.
The overview does not say what the one-time experiment cost, but Bergman estimates that it was significant and would have been impossible to replicate on a large scale. However, it did lead to California in 1983 including a three-hour, closed-book performance test with its bar exam. And in 1997, the National Conference of Bar Examiners added its Multistate Performance Test, known as the MPT.
Since then, some lawyers say that little has changed with bar exams nationwide, and there are still concerns about whether the test accurately evaluates skills needed to practice law. Others have argued that today’s technology could handle many methods of bar exam testing than seemed impossible in 1980.
“It’s an excellent barrier to entry, and it’s also a superb hazing ritual. But neither of those things are tasks it’s supposed to fill. The practicing bar should be interested in having a licensing requirement that requires a competent lawyer, not one that keeps out the competition,” says Joan Howarth, the former dean at Michigan State University College of Law who’s now at University of Nevada Las Vegas William S. Boyd School of Law.
She’s been studying and writing about bar exams since 1995 and maintains that there’s more interest now than ever before in changing them. The exam is too easy in some ways, like with analysis being artificially simple, Howarth says. And it’s too difficult in that it’s a test of memorized knowledge at a time when memorization is the least important aspect of attorney competence.
Others, including Bergman, say that the traditional bar exam still has its place.
“Legal education is about applying abstract principles to concrete scenarios; that’s what lawyers do. Whatever form [of testing] you use to dress it up, like the idea that students are doing something realistic, ultimately it’s going to involve the application of abstract principles to concrete scenarios. That’s what the bar exam does,” Bergman says.
Study Sessions
Regardless of where one stands on its effectiveness, in recent years, there has been a significant amount of ongoing research looking at whether the bar exam as we know it is a good tool to assess if someone is ready to practice law. Between 2018 and 2019 three legal organizations funded separate studies about the bar exam. Also, the American Bar Association in 2017 appointed a commission that is considering suggestions for change in training and testing future law students. All the work is ongoing.
Additionally, the topic is being studied by New York, Indiana and California. Nevada is planning to launch a study in early 2020, focusing on whether bar exam scores correlate with attorney skills among new lawyers.
Concerns include the number of questions on the exam, whether multiple-choice questions have any relation to practicing law and the lack of research materials for test-takers.
“There are so many aspects to the bar exam that don’t quite align with the practice of law, one of which is a closed-book format,” says Aaron N. Taylor, executive director of AccessLex’s Center for Legal Education Excellence. In addition to the Institute for the Advancement of the American Legal System study mentioned below, AccessLex helped fund the California study and is funding the Nevada study.
Jurisdictions decide what will be on their bar exams, and 36 have adopted the Uniform Bar Exam, which is offered by the National Conference of Bar Examiners. States that have not adopted the UBE still rely on NCBE testing materials, but often write their own essay questions, or use NCBE essay questions in addition to their own, says Judith Gundersen, president and CEO of the NCBE.
Most bar exams are two-day affairs, with a six-hour multiple-choice test known as the multistate bar exam given in the first day. The second day of testing usually consists of six essay questions, which are allotted 30 minutes each, and a performance test, which generally has two response-based questions focused on things such as problem-solving and legal analysis. Those questions are allotted 90 minutes each.
Some states, including New York, Florida and Louisiana, also have state-specific multiple-choice questions. Candidates also must pass a professional responsibility exam, which is multiple choice and can be taken during law school.
Image from Shutterstock.com.
Incremental Changes
And as was the case in 1980, concerns remain that the bar exam has a negative impact on test-takers of color. California and New York recently examined the issue in separate studies.
The California study, released in 2018 and focused on bar exam performance changes, found that for its state bar exams given in 2013, 2016 and 2017, the overall pass rate was 55% for minorities and 69% for nonminorities.
The New York study, which was done by the NCBE and released in 2019, partly focused on how candidates grouped by race performed on the bar exam before and after the July 2018 UBE adoption. Based on sample groups, white test-takers had the highest average of NY bar exam scores. That group was followed by Asian or Latino test-takers, depending on the year, and black test-takers.
The study also looked at candidate’s background characteristics, such as undergraduate and law school grade-point averages and law school admissions test scores. GPA and LSAT score patterns were similar to bar exam pass rate patterns, but some years the sample group for Asians had higher LSAT scores than the sample group for whites.
“I think people really want to solve this one major problem, and that’s, ‘Is there a body of students out there who would be good lawyers, but are failing the bar? Is there some way of getting them through the bar?’ ” asks Derek Muller, a law professor at Pepperdine University, who writes at the website Excess of Democracy.
He also wonders if changing the bar exam would solve any problems, including a decreasing national pass rate. The overall pass rate was 54% in 2018; and in 2008 it was 71%, according to NCBE data.
“If lawyers are saying, ‘I did it this way, kids need to do it this way, too,’ that’s not productive. At the same time, if schools are saying, ‘We need to change the bar in whatever way we can to get kids to pass,’ that’s not productive either,’ ” Muller says.
He’s not sure that changing the format would make much of a pass rate difference, and he wonders how some ideas, like giving a partial bar exam after students finish their first year of law school, would actually play out.
“I think that would put extraordinary pressure on the first year of law school in a different way. For a lot of students, it’s a steep learning curve the first year, and to add another exam, I think that would be a step backward,” Muller says.
Others say that if you want to increase bar pass rates, look to the schools, which in some cases have lowered admissions standards, as fewer people attend law school. In 2018 there were 38,392 first-students were enrolled in law school, compared to 49,414 in 2008, according to data from Law School Transparency.
“You have people for three years, and the bar exam is two to three days. Who should be faulted if after three years of education, people aren’t passing the bar exam? What’s the problem there?” asks Robert Hawley, the former acting executive director of the State Bar of California.
And others look to recent law school graduates.
“We try to emphasize to students that their chances of passing a bar in the first two tries is 90%. Why not do the work now?” asks Barry Currier, the ABA’s managing director of accreditation and legal education. He thinks that how the profession defines minimum competence has changed over the years.
“I think we are asking the bar exam to carry a lot more water than we asked it to carry a long time ago,” Currier says. “If you give someone a license are they competent to go do anything? The notion of competence has always been there, but everyone has to understand that when you’re beginning in practice you are not fully developed, and law school is just one pathway to development.”
For the most part, academics lead the work on the various, ongoing, bar exam studies. In private practice, the issue isn’t on people’s radar, says Bill Henderson, a law professor at Indiana University. His research centers on the changing practice of law and law firm economics, and he’s a co-founder of the Institute for the Future of Law Practice, a nonprofit, educational boot camp and job placement program that focuses on bridging legal education, technology and operations fields.
“It would surprise me if someone working in private practice had strong opinions on this. I don’t think they give it much thought. They want to know if you passed, that would be the main thing,” says Henderson.
Based on his experience working with law firms on systems and hiring criteria, he’s found that legal hiring for young lawyers focuses more on where someone went to school, rather than what they learned while they were there. And, Henderson notes that people from the highest-ranked law schools have the least amount of problems passing a bar exam and finding work.
Image from Shutterstock.com.
Outside the Box
In January 2018, the NCBE appointed a testing task force, which by the end of 2020 will finalize a report with recommendations about how the bar exam should test competence for entry-level lawyers in a changing profession.
Task force work includes listening sessions about the exam’s content, format, timing and delivery method, with thoughts from state supreme court justices, law school faculty and attorney members of the ABA.
“I have been surprised with how people are thinking outside the box about what the possibilities are,” Gundersen says.
There’s support for states adopting a uniform cut score, rather than setting their own, and for breaking up the bar exam in pieces, instead of giving it over two days, according to the listening sessions’ summary. An idea for staggered testing has candidates taking the multiple-choice questions in law school, after finishing core classes. Essay and performance tests would be taken after graduation.
Virtual reality, to simulate client interactions and legal work was mentioned in the summary too. There’s also support for adding a test section on legal research, which would include access to an electronic database of legal resources.
In terms of change, Gundersen says that nothing is off the table, but cost is an issue.
“We’re taking a clean-slate approach, with no preconceived outcomes,” she says. “But we are not throwing practicality out the window and will consider logistics when making recommendations.”
Lawyers grade the bar exam’s essays and performance tests, and if more constructed response questions were added to the exam, it would take longer for grading to be completed, Gundersen says. Some wonder if artificial intelligence technology could make the process more efficient.
“We did an experiment with AI a few years ago, and its agreement with human graders wasn’t close to being good enough,” says Gundersen, adding that the technology is good at evaluating the quality of writing, in terms of sentence structure and syntax, but has problems evaluating heavy-content issues.
The Institute for the Advancement of the American Legal System, an independent research center at the University of Denver, is also funding an ongoing bar exam study. It focuses on defining minimum competences needed to practice law, with research including 60 focus groups in 12 states. Eighty percent of the groups have attorneys licensed for no more than three years who are surveyed on what knowledge and skills they needed their first year in practice and whether they learned the skills in law school or at work.
The remaining groups will consist of attorneys who supervise young lawyers.
“We only have a few transcripts so far, but from what I’ve looked at, I am amazed at how many people are working in areas that are not taught at all, or tested on the bar,” says Deborah Jones Merritt, a law professor at Ohio State University doing the study, which is part of IAALS’ Building a Better Bar project and partially funded by AccessLex, a nonprofit group focused on law students.
Merritt started studying and writing about bar exams in 1999, when several states were thinking about raising bar exam cut scores. Today, she likes the idea of bar exams broken up in pieces “so it’s not a two-day marathon and can be taken at different times.”
She’s also in favor of more performance test questions and giving test-takers longer time periods to complete the section.
“If you came up with real-life legal answers in 90 minutes, it could be malpractice,” says Merritt, who also supports including relevant legal principles in bar exam materials, so answers are not limited to test-takers’ memorization skills.
Meanwhile, some people who recently passed a bar exam developed their own unique study methods, in addition to relying on what they learned in law school and bar review courses.
“We want people to memorize 10,000 rules, but do you know how it fits together?” asks Andrea Sinner, a 2017 graduate of the University of Miami School of Law. She created a bar exam study rubric in Microsoft Excel with topics for key multistate bar exam subject areas and corresponding Florida distinctions because the Florida bar exam has additional multiple-choice questions on state law.
A former executive managing director with Accenture, who has an undergraduate degree in industrial engineering, Sinner pulled the information for her study rubric from her BarBri course and two bar prep classes she took in law school. One focused on the MBE and the other on Florida law. Her creation has more than 500 rows, with each row having between two and six rules.
After she took her first BarBri mock exam, which has the exact answers, she went back to the rubric and found the corresponding rules to what was on the mock test. She marked off the topics she answered correctly, and for the ones she got wrong, she looked into why she got it wrong. She also categorized why she got things wrong, and that came down to rote memorization and not reading the question correctly.
“Printed out, it’s 24 pages, and serves as a guide to the many hundreds of pages of information available through the bar prep providers,” says Sinner, who passed the Florida Bar in 2017 and now works for the ABA as director of the Commission on the Future of Legal Education.
“What’s difficult for me is that 70,000 people take the bar exam every year, and even if people don’t do exactly what I did, there’s a lot of work involved. I just don’t see the value we’re getting out of it,” Sinner says. “I don’t know if the exam is able to judge someone’s preparedness to practice law, or even help with it.”
