Artificial intelligence software outperforms lawyers (without subject matter expertise) in matchup
During the last week of October, legal technology company CaseCrunch held an AI-versus-lawyer competition, and the machine came out on top.
The competition pitted over 100 attorneys from firms like DLA Piper and Allen & Overy against CaseCruncher Alpha to predict outcomes of just under 800 real, historic insurance misselling claims. The goal was to correctly determine if the claim would succeed or not.
According to CaseCrunch’s website, the software predicted outcomes with almost 87 percent accuracy, while the lawyers were 62 percent correct.
“The main reason for the large winning margin seems to be that the network had a better grasp of the importance of nonlegal factors than lawyers,” read a statement on the company’s website.
Digging deeper into the structure of the competition, Legal IT Insider quoted Ralph Cox, a patent litigation partner at Clyde & Co. in London and an attendee of the event, as saying: “I keep an open mind. However, it struck me that the computer was given all the database information and therefore had an unfair advantage. Lawyers who took part were from outside this area of expertise without any real experience of [Payment Protection Insurance].”
Indeed, the lawyers recruited for this competition were not subject matter experts in the topic tested in the competition.
Ludwig Bull, CaseCrunch’s scientific director, said to the same publication that the subject matter imbalance was hard to reconcile.
“We struggled to find an area because lawyers specialise in so many niche areas. So we had to find something that was relatively easily intelligible to most lawyers and where they could also understand the underlying principles relatively quickly. It was as fair as we could make it,” said Bull.
Nadia Abdul, one of the co-founders of CaseCrunch, was more succinct, adding: “The system does not have the experience of a lawyer who has worked in the field of PPI.”
Even so, this competition was covered by BBC and elsewhere as an epic man-versus-machine matchup, like the legal world’s Kasparov versus Deep Blue, a pair of six-game chess matches in the 1990s between the world’s reigning chess champion, Garry Kasparov, and IBM’s supercomputer. While Kasparov won the first match, Deep Blue won the second. At the time, it was seen has a benchmark in the development of artificial intelligence.
However, Bull seemed to temper this narrative when he told The American Lawyer simply: “These results do not mean that machines are generally better at predicting outcomes than human lawyers. These results show that if the question is defined precisely, machines are able to compete with and sometimes outperform human lawyers.”
Framed in that way, pitting humans against computers to assess the outcome of legal issues is not novel. In 2004, professors released research (PF) regarding a classification tree model that better predicted U.S. Supreme Court outcomes than human experts. In 2016, similar work showed an algorithm correctly predicted the outcome of European Court of Human Rights’ cases 79 percent of the time; however the algorithm was not pitted against humans in that study. Less abstract, the McKinsey Global Institute found that about 60 percent of all jobs, including lawyers, could have 30 percent of tasks automated today.
Regardless of novelty, the fact remains that machine based decision making is a growing part of the legal space. Of the competition, Timothy Lessen, a partner at Lewis Silkin in London, told Legal IT Insider: “CaseCrunch met my expectations. But then I think I was pre-programmed to expect this result.” He added, “I’m not completely clear what [CaseCrunch] will look like yet: an end-to-end product that means 5 percent of the workforce of private practice law firms is no longer required, or a product which allows us to deliver a much better service to clients.”
