Still Searching: Computers Change the Role of Lawyers in E-Discovery
Photo of Bennett Borden by James Kegley
Software providers would like attorneys to believe that technology has solved the problem of searching digital records in litigation. But the answer is not that clear.
Because computers can quickly analyze millions of documents, they can—in theory—replace many of the lawyers and paralegals that typically perform that role. Yet litigators say computer review does not completely replace human reviewers, but it does change their role in the review process.
“Absolutely it will mean fewer people, but it certainly doesn’t mean the end of human review,” says Bennett Borden, co-chair of law firm Williams Mullen’s e-discovery and information governance section in Richmond, Va. “It just means you have fewer people, but with different skills. I find you can do more with 10 expert reviewers than 100 contract attorneys doing review.”
If improperly used, the predictive coding that is built into a computer-assisted search can produce worse results than traditional review processes. “Predictive coding systems help researchers find and even organize documents containing contents for which they are looking,” says Bill Speros, a Cleveland-based e-discovery consultant. “But if misused, the systems only amplify the researchers’ selection bias.”
Today, attorneys involved in discovery select topics of relevance to a particular case, usually broadly worded discovery demands, and search for relevant information using a set of keywords. Using traditional Boolean search technology, computers typically return large volumes of documents that are then reviewed by lawyers or paralegal staff. The human reviewers make the ultimate call on what is responsive, nonresponsive or privileged under the given discovery request.
Most members of any litigation team are likely to be contract attorneys doing document review. But it’s not a job held in high regard. “You’re locked in a dungeon,” Borden says, “barely using your brain, relying on base recognition skills—is this document responsive? Yes or no, and that’s it.”
Recent advances in search technology allow reviewers to “train” their computers to find documents with relatively little human input. A legal team examines sample documents the computer returns, and then uses that material to train the computers to find similar documents.
Borden recalls a fraud case in which he didn’t know the words the subjects of his investigation had used to discuss their actions. Eventually computers found a document referencing a meeting and a code word describing the meeting topic. The computer then looked for documents that tied together the attendees, the time frame of the meeting and the code word, and was quickly able to find material he believed established a pattern of fraud.
The advantage to this system is that, unlike humans manually reviewing multitudinous documents, lawyers review only a small subset in a collection and let the computer do most of the work.
“If you find a document that you originally thought was nonresponsive, but it turns out to be responsive, it’s hard to tell your contract attorneys to go back and start all over,” says Maura Grossman, litigation counsel with Wachtell, Lipton, Rosen & Katz in New York City. “But it’s easy to tweak your search and have the computer re-review the entire collection in a matter of minutes.”
Though fewer attorneys are involved, proponents say those lawyers find their roles more satisfying.
“Instead of an army of contract reviewers, I have a small team of document reviewers that we unleash on data sets,” Borden says. “Those teams are investigating leads, tying threads together and making judgment calls —the things lawyers actually went to school for.”
STILL TREC-ING
The foundation for this growing reliance on technology is the TREC Legal Track, a research project that explores the limitations of search technology in litigation. But while the project has demonstrated that search technology can find impressive amounts of information, the research is not conclusive.
According to the most recent research from the TREC Legal Track, actual recall rates for search technology ranged widely—from 85 percent recall down to 38 percent.
“I also want to put the brakes on the hype,” says Jason R. Baron, director of litigation at the National Ar chives and a founder of the TREC Legal Track. “We are headed in the right direction, and there is certainly no going back to manual review in large cases. But it’s dangerous to say or imply the problem is solved.”
But in considering whether to use technology-assisted review, lawyers should also be careful not to make the mistake of thinking that human reviewers are more reliable than computers. TREC studies show manual reviewers—whether reviewing documents for legal or general information retrieval purposes—miss a remarkably large number of relevant documents, often 30 percent or more.
A study by Grossman and computer scientist Gordon Cormack based on data from the 2009 TREC Legal Track showed manual reviewers missed from 21 to 75 percent of the responsive documents, depending on the topic. A subsequent study showed that about 90 percent of the missed documents were clearly due to human error and should easily have been identified.
The question is no longer whether to rely on human-led review or computer review. Going forward, litigation teams will be smaller, but will demand more technical expertise in order to make effective use of computer search.
Grossman says that by using predictive coding in review, a legal team can do in days a job that used to take weeks or months. The difference is that small teams of lawyers now need to understand the subtleties of predictive coding technology and statistical sampling so as to provide a defensible computer-assisted search strategy.
But if a litigation team can build this experience in-house, they can perform review tasks no traditional review team could hope to complete. For example, Grossman claims a team can now whittle more than 300,000 case documents down to the 3,500 that matter in a single weekend.
“Technology-assisted review can certainly eliminate some of the drudgery from the review process,” she says. “But the computer cannot stand up in court for you. The role of lawyers in review is evolving, but it is certainly not diminished.”
