Amid claims that online bar exam went well, some test-takers have a different view
Image from Shutterstock.com.
ExamSoft, the company that provided the software for the two-day online bar exam offered earlier this month, maintains that only a small percentage of test-takers experienced problems with their product. But for those who did, many say there should have been no issues and some suspect the hitches were expected by the company.
The online exam, which was composed of materials from the National Conference of Bar Examiners, took place Oct. 5 and Oct. 6. Approximately 30,000 people took the exam, and it’s believed to have been the first professional licensing exam administered remotely, according to Judith Gundersen, president of the NCBE. In an email to the ABA Journal, she said it appears that more than 98% of applicants had no software issues.
“We understand that most technical concerns were addressed quickly, but we are currently gathering data and feedback from the jurisdictions that administered the remote exam, and from ExamSoft, about any issues that arose,” she added.
The five test-takers who were willing to be interviewed on the record by the Journal took the California and New York bar exams. All say they finished the test despite significant software problems.
Ricardo Sabater, a 2020 University of Chicago Law School graduate.
Ricardo Sabater, a 2020 University of Chicago Law School graduate, took the New York bar exam. He had no technical issues on the first day of the test, but on the second day, when there were 100 Multistate Bar Examination questions split into two 90-minute sessions, he says the software repeatedly crashed.
The software included facial recognition identity verification, according to a New York Bar exam website created by ExamSoft. There have been concerns about the technology having issues reading faces for women and people of color accurately. Sabater, who is Puerto Rican, did not know whether his ethnicity played a part, but states the software would read his face, then he would get an “unexpectedly quit” notification.
After “three or four more tries,” he was given access to the first section of the multiple-choice questions. The section had 50 questions, and the software crashed on No. 20. He called ExamSoft support but hung up after one minute out of fear that the wait time would eat up his testing time on the 90-minute section, and he might be told to withdraw and take the February exam.
So rather than staying on the phone, he restarted his computer, a 2016 MacBook Air, opened the app and was taken back to the place on the test he’d left off. Sabater estimates it crashed three more times during the first section and at least three additional times in the second section. He was, however, able to finish each section.
After completing the exam, he called ExamSoft support to make sure his submissions were successful. That was verified by the person who helped him, whom Sabater says was friendly but also did not seem surprised by the issues.
“They were fully expecting the site to crash this way. They said, ‘Yes, sometimes the app will die, you will get back in and it should be fine.’ But we weren’t warned about this. I can’t imagine how many people went through what I did and quit on the third time,” he adds.
Nici Sandberg, an ExamSoft spokesperson, described Sabater’s assertion as “entirely false” in an email to the Journal.
“Our support team would not say something like this because it is exceedingly rare for there to be any sort of technical issue with the software functionality within an actual exam. The majority of technical issues are related to user devices not meeting published minimum system requirements, which we are able to verify through log files,” she wrote.
According to ExamSoft, nationally there was a total of 150,000 exam sessions, and 3,000 support cases that were handled successfully.
“A small percent of the support calls we received were truly technical in nature versus educational,” she wrote.
Sabater says he completed the mock exams required to take the October bar and had no technical problems with them. In September, various applicants reported having software problems with the mock exams.
According to a statement from the New York State Board of Law Examiners, 5,167 applicants downloaded the October bar exam files, and 5,154 people took the entire exam. The organization claims the October attrition rate is similar to what is usually is for in-person exams. A “small number” of New York applicants experienced technical issues during one or more of the sessions, according to the statement.
Other New York examinees had problems creating and uploading the proctoring videos, which were required to complete the exam. Erica Rebussini, a 2020 University of Richmond School of Law graduate who took the New York bar exam, says she spent the afternoon of Oct. 6 on the phone with ExamSoft support because one of her videos would not upload. According to her, it finally did on the evening of Oct. 7.
“We were told that if a video file wasn’t uploaded, we would be getting a zero. I really needed the assurance that I would not be getting a zero for something that was out of my control,” says Rebussini, who also sent the company a tweet about her problems.
“Things I did not wake up to this morning: my final confirmation email from @ExamSoft for my MPT video proctor file from yesterday. I’ll call y’all later, I’m focusing on the second half of the bar today but want to document that I still haven’t gotten it. @dp4ny @DiplomaPriv4All,” she tweeted Oct. 6.
Colin Darnell, a 2020 Indiana University Maurer School of Law graduate, took the New York bar exam on a 2019 Lenovo laptop, which he used to successfully complete a mock exam. But he noticed he wasn’t being recorded by the software on Oct. 5, a Multistate Performance Test and the Multistate Essay Examination testing day.
Darnell was interviewed on camera Oct. 8 by the Journal via Zoom.
“My webcam works just fine,” he says, adding that he called ExamSoft support about the issue. After a 30-minute wait, Darnell says he was told he should take the test without video monitoring. That didn’t sound right to him, so he called the New York Board of Law Examiners and was informed a bar exam without video recording would not be accepted.
So he called ExamSoft support again.
“I’d already lost 50 minutes, so I figured, why not,” he adds. “But once I was on hold, I realized I couldn’t complete the MPT, so I went ahead and withdrew.”
Although Darnell describes himself as “not much of a social media guy,” he decided to tweet about the experience.
“Final update: I ‘talked’ to ExamSoft again. Got transferred to tech support. After another 15 minute hold, I realized that I just wouldn’t be able to complete the MPT in time. I’ve withdrawn,” he tweeted Oct. 5.
About 20 minutes later, according to Darnell, John McAlary, executive director of the New York State Board of Law Examiners, contacted him and said the problem would be fixed.
Darnell says 15 minutes after his phone call with McAlary, ExamSoft support called him. He claims they accessed the laptop video camera and fixed the software problem through Zoom’s remote control function. Before then, he had no idea ExamSoft would use Zoom for software fixes, and he estimates the repair took less than 25 minutes.
“I got back into the test around 2:30 p.m. and finished between 7 p.m. and 8 p.m.,” he says.
On Friday, two New York lawmakers who support diploma privilege released a “snapshot survey” of 500 New York bar applicants, 41.1% of whom reported internet or software problems during the exam, according to a news release. The bar exam experience was described as negative by 74.7% of test-takers; only 8% found it positive.
When asked for a response, ExamSoft again asserted that there were few test-takers with software issues.
“While this survey represents a small number of concerned exam-takers, we are all learning to be more flexible and to rely on technology to continue our lives, work and education as effectively as possible,” Sandberg wrote in an email.
California bar exam applicants also reported software issues.
Reginald Dulaney, who took the California bar exam, says his laptop crashed Oct. 5 when he was working on the test’s first essay question.
Reginald Dulaney, who took the California bar exam, says his laptop crashed Oct. 5 when he was working on the test’s first essay question. He called ExamSoft support twice. The first call was 48 minutes long, and the fix worked “for a good 5 minutes,” he says. The second call lasted two hours.
One hour was allotted for each essay question, and Dulaney says he was 32 minutes into the first question when his computer crashed. He submitted his answer before it was finished because he claims ExamSoft support said they needed it for troubleshooting.
According to Dulaney, ExamSoft support also told him to “withdraw or get another laptop.” He took the exam at his parents’ home, and switched out his 2019 Lenovo laptop for his father’s 2017 MacBook Air. To use that machine for the test, he had to pay an additional $50 administrative fee to download the software again.
Even though Dulaney had a late start on the rest of the essay questions, he was able to finish them. He had no problems on the second day of testing, but the 2019 California Western School of Law graduate could not get back into the first essay question to finish his answer.
“That led me to call the state bar,” Dulaney says, adding that he asked if he would have any recourse for the first essay question because the system crashed before he finished it.
“The lady I spoke with said that as of right now, they didn’t have anything like that in place. That was a little troubling to hear,” Dulaney says. He adds that the person who took his call was aggressive and repeatedly told him there was no one to whom she could transfer his call.
He’s also concerned his proctoring videos may be flagged for cheating because of all his technical problems. Dulaney, who is Black, says that like Sabater, he also had problems with the software’s facial recognition.
“Because the technology is not where it needs to be and because of my facial features, I’m automatically assumed to be distrusted,” he says. And aside from that, he thinks the bar exam caused the most harm to test-takers who didn’t have access to good technology and spaces with few disturbances.
In an email to the Journal, the State Bar of California said it would not rely on the proctoring software’s artificial intelligence capabilities to determine an applicant’s “identity, integrity, eligibility or passing grade.” The agency added that the determination process relies on “several independent lawyers of human review” to assess issues flagged by the software.
Rebekah Merrill-Callaway also took the California bar, and like Dulaney, had technical problems with the essay question section. The software would ask her for a code, she says, but when she entered the attached code, she would get a system message saying it was invalid. So she developed a workaround of sorts through the software dashboard until she reached the fifth question, when the system froze and would not restart.
So she called ExamSoft support while simultaneously trying to get through to the company via its chat box. After 15 minutes of being on hold, her call was dropped, she says, so she called back. According to her, the second call also was dropped.
“In that moment, I’m thinking about the four years I’ve spent in law school, the five to six months I’ve spent studying for the bar and that crushing feeling of ‘That’s it,’” says Merrill-Callaway, a night student who graduated from Southwestern Law School in May 2020 and took the exam on a 2016 MacBook.
So she started tweeting.
“I’m still waiting. It is now 3:19. You disconnected me and I still can’t get in. HELP me,” she tweeted Oct. 5. ExamSoft then responded with a tweet, asking her to follow them so they could send a DM. She did, gave them her phone number and then got a call.
They fixed her problem through the remote control option on Zoom. Like Darnell, Merrill-Callaway says she had no idea her problem would be fixed through Zoom, adding that she used her 13-year-old son’s account since she didn’t have her own. She finished the first day of testing around 4:50 p.m.
“There are far worse things in the world, but it is relative. In this moment for me and others who did not get the outcome I got, it was absolutely horrible. In terms of the level of stress the bar exam creates, to add this on top of that—it’s beyond the pale,” she says.
In a statement, Donna Hershkowitz, interim executive director of the State Bar of California, wrote that “an overwhelming majority” of applicants completed the October online exam.
“We heard feedback from many applicants that they actually preferred the remote format. Making these points is not done to minimize the experience of anyone who had challenges, but to provide a broader context,” she wrote.
Also, there have been accusations of data security problems with the software. ExamSoft recently shared some review audit findings about its software, which were conducted before the October bar exam. The review audit was conducted by Stroz Friedberg, a risk management group owned by Aon.
According to an ExamSoft email to the Journal, the review audit found that user information it received was encrypted, the company didn’t have access to test-takers’ payment information and data the test-takers uploaded at the end of the bar exam didn’t include personal information unrelated to the bar exam.
Updated on Oct. 20 at 4:18 p.m. to clarify Sabater’s statements about the software quitting multiple times after scanning his face.
See also:
ABAJournal.com: “Florida reschedules online bar exam because of technology problems”
