Don't trust, verify: Data must be assessed and analyzed before it's used
From left: Jennifer Roberts and David McNeill. Photo by Victor Li.
To paraphrase the famous quotation from economics professor and Nobel laureate Ronald Coase: “Torture the data long enough, and it will confess to anything.”
That was the central theme of a Thursday-afternoon panel at ABA Techshow called “Survey Says: Can You Trust the Data?” Jennifer Roberts, senior manager of strategic research at Intapp; and David McNeill, CEO of Docket Reminder, guided attendees through the data-gathering process while reminding everyone of the major principle relating to data: It’s gathered by humans.
On the one hand, that means fears of automation and “robot lawyers” taking over the legal industry are wildly overblown.
“Humans are at the beginning stage of it all,” Roberts said. “They’re the ones who enter the data and interpret it. That’s never going to change.”
On the other hand, that means data will always be subject to human error and biases.
“Any time you ask a human to make a judgment call, bias occurs,” McNeill said. “What you think of the world is challenged by what you see, so you must challenge yourself to look for what you can’t see.”
McNeill cited factors such as optical illusions, confirmation bias and data manipulation as ways that humans can affect how data is collected and interpreted. “We bring our experience and beliefs with us, and it taints everything we see.”
Follow along with the ABA Journal’s coverage of the ABA Techshow 2020 here.
With so much data available to law firms and companies, it is incumbent on lawyers to first conduct a data inventory to determine what kind of information they want to collect. Roberts suggested looking outside the legal industry to see what kind of data they are looking at, zeroing in on client satisfaction.
“If you’re at 75% client satisfaction, then you aren’t providing the value you’re billing for,” said Roberts, who also pointed to the similar metric of net promoter score, which measures the willingness of existing clients to recommend a service or company to someone else.
McNeill, meanwhile, stressed the necessity of having a data hygiene program in place, preferably in the form of data stewards who can assess the trustworthiness of all of the facts and figures pouring in from all kinds of sources.
“The first thing I do when I get a data set is scrape it to make sure it is accurate,” he said. “For instance, I once found a court record for Yogi Bear that involved financial crimes. And a divorce case involving Mickey and Minnie Mouse.”
