Editor’s Note: In the ABA Journal’s Law News Now blog, there was a summary of Casey Flaherty’s audit program. After discussion in the comments, Flaherty was invited to more thoroughly discuss his program and respond to criticism.

I am corporate counsel at Kia Motors America, Inc. I have developed a basic technology competency audit that I administer to my outside law firms. The firms have not fared well, and I have, among other things, cut their billing rates unless and until they pass a subsequent audit. In collaboration with Suffolk University Law School’s Institute on Law Practice Technology and Innovation, I’m now in the process of automating my audit and making it available free to law students and my fellow in-house counsel.

I’ve written extensively on my audit (e.g., here, here, here, and here). I’ve also given several presentations about the audit, including a LegalTech West keynote (viewable here). ABAJournal.com published a (i) summary blog post blurb (ii) of an article (iii) on my keynote. The resulting comment thread was not all positive (though much of it was). It is unwise to feed the trolls. But it is also unhealthy to reflexively dismiss criticism. Because I believe the need for technological competence is a topic worthy of debate, below is a point-by-point response to some comments.

A little background on the technology competency audit

My hypothesis is that lawyers in general are woefully deficient in using the software tools at their disposal – e.g., Word, Acrobat, Excel. To test this, I provided associates at outside firms with mock assignments. Sample tasks include (a) formatting a motion in Word, (b) preparing motion exhibits in PDF, and (c) creating an arbitration exhibit index in Excel. The specific tasks, however, are of little importance as they are designed to test general skills. The foregoing examples could just as easily be (a) formatting a contract in Word, (b) Bates stamping a document production of PDFs, or (c) isolating pertinent performance data in Excel—or, really, any of the other myriad, routine, low-value-added tasks that lawyers regularly complete on their computers (or should).

I’ve administered the audit 10 times to nine firms (one firm took it twice). As far as I am concerned, all the firms failed—some more spectacularly than others. The audit takes me 30 minutes. So, somewhat arbitrarily, I selected 1 hour as passing. The best pace of any associate was 2.5 hours. The worst pace was 8 hours. Both the median and mean (average) pace rounded to 5 hours.

I fully admit that these metrics offer a false precision beyond even the small sample size. I, for example, stop lawyers as soon as they start down a brute force path (e.g., manually updating every exhibit number) instead of utilizing a labor-saving feature (e.g., autofill). So I extrapolate their pace rather than suffer through the repetition. Moreover, many lawyers just have no clue how to complete certain tasks (e.g., index and search across a collection of PDFs). Thus, there is no pace to extrapolate. Moreover, the audit evolved each time I administered it. Mine is not a pure apples-to-apples comparison.

I do not have the data or rigor to quantify just how much waste exists in the legal system or what percentage of it is attributable to technological incompetence. My claims are much broader: a lot (of waste exists in the legal system) and enough (of that waste is attributable to technological incompetence to make this a problem worth addressing). My collaboration with Suffolk should, however, provide a more robust, numerical foundation to support these claims. Or I could find out I’m wrong.

This audit is not the answer. Alternative fee arrangements are the answer

With the audit, I am not attempting to offer “the answer” to every challenge facing the legal profession. Clients expend such substantial sums on legal work that incremental improvements at the margin save real money. Basic technological competence is low-hanging fruit, and the object of the audit is its own obsolescence in the very near term. I therefore tailored the audit to the legal market as currently structured. I would prefer to avoid having the issue subsumed into the long-standing, often vitriolic debate over AFAs.

That is, I’m hoping that the profession needs little more than a nudge. Free instruction is all over the Internet. Many law firms already have solid training programs that are under-attended due to the very real countervailing pressures on lawyers’ time. Excellent outfits, like Capensys, a company involved in automating my audit, also stand ready to fill a need that was recognized long before I came along. And lawyers, by and large, are intelligent, honest, diligent, and quick on the uptake. Technological competence comprises skills that need to be learned. But the learning curve for the basics is neither steep nor long.

By contrast, the ABA has been moving Beyond the Billable Hour (PDF) since 1989. In 2007, Scott Turow took to these very pages to join a legion of voices, before and after, proclaiming that The Billable Hour Must Die. Yet, in 2013, we are still learning Why the Billable Hour Endures and getting thought-provoking pieces In Defense of the Billable Hour. Indeed, a recent survey showed that “AFAs accounted for a decreasing portion of overall U.S. work.” (emphasis added)

This is not to dismiss AFA’s or the vital discussion surrounding them. The proper structuring of incentives is of prime importance. I’m altering the incentives only slightly. For bolder proposals, I recommend important contributions on the topic of AFAs from the ABA (PDF), Patrick Lamb, Jim Hassett, and Jeffrey Carr, just to name a few. Further, I’m a strong proponent of the continued developments in legal project management, legal process outsourcing, and budget analytics. These are all pieces of the value puzzle and attendant forces of structural change that occupy the likes of Paul Lippe, Richard Susskind, and Bruce McEwan. I do not place my audit at the forefront of these pivotal issues because it does not belong there. I recognize that basic technological competence is not of world-historical importance. But this concession is not license to dismiss the issue. The perfect need not be the enemy of the good, and not every innovation has to be disruptive.

Will lawyers waste time as long as they have the incentive to do so?

Some will. Personally, I am not so cynical about the pervasiveness of intentional overbilling or so optimistic about eradicating graft. I believe that most lawyers will avoid drudgery once they learn how. That some will still pad their bills is a separate issue.

Survival is the threshold that most people, only some of whom are lawyers, reach with most technology. Attention is a scarce commodity. We are capacity constrained in mastering the robust features on offer from smartphones, televisions, microwaves, in-car infotainment systems, etc. Lawyers, for example, do not (openly) bill for the time spent creating billing entries. Yet, barely functional is as far as most get in learning their billing software. They know just enough to get their hours logged. But they fail to take advantage of features—duplication, text expansion, hot keys—that would reduce the time necessary to produce substantially similar entries day after day after day. This becomes my problem, however, when lawyers are similarly ignorant of the software they use—e.g., Word, Acrobat—to perform my billable work.

I’m too busy.

I sympathize. The deadline pressure from clients and courts can border on suffocating. It is unfortunate that it often takes time to save time. In an isolated instance, learning a shortcut can absolutely consume more attention than just pushing through. But to understand is not to forgive. Legal bills are cumulative. Those six-minute increments aggregate at an alarming rate. When someone, including me, protests that they are “too busy,” what they are really saying is that they have other priorities. Prioritization is inescapable. It is, however, my right as the client to question the priorities of my outside counsel when they give precedence to low-value-added, billable work over non-billable training that would reduce my total spend.

Our firm would not charge for busywork. We give that to paralegals, secretaries, assistants, etc.

Excellent. What follows is an excerpt from the introductory letter I send to associates before I audit them:

It is important to understand that the distribution of work is one focus of the audit. You should not hesitate to explain that a particular task would be performed by someone else—e.g., your secretary, a paralegal, word processing. For certain tasks, that is precisely what is expected (i.e., it is the right answer). A few caveats, however:

• Paralegals bill for their time. Invoking them does not obviate my interest in how efficiently the subject task is performed. Unless your paralegals receive substantially different training than associates, I remain interested in observing how you perform the task. If the paralegals do possess alternative, pertinent skills, then I am also interested in auditing them.

• The audit measures general proficiency, not just ability on a narrow task. While a specific task may not be entirely appropriate to your position, the task should still reveal whether you have acquired the general proficiency with which the audit is concerned.

• Sufficient support staff is not always available. In particular, associates put in late nights, long weekends, and marathon sessions on the road. They do whatever is required to achieve their primary objective: ensure that all work gets done well and on time. This includes work that would ideally be handled by support staff.

• Finally, I will take you at your word, but I will hold you, and the firm, to it. If you represent to me that efficiency on a particular task is immaterial because it will never show up on a bill, then I expect it to never show up on a bill.

It is not always easy to draw the line as to which type of work should be handled at which level. But, before even reaching the proper distribution of work, it is important to recognize that, oftentimes, appreciable human labor is unnecessary. In my presentation, I discussed basic functions in Acrobat that prepare PDFs (e.g., for e-filing) without printing and scanning.

Bates stamping, in particular, engendered considerable pushback, such as:

There is no need to charge a client $250+/hr for me to put page numbers on a stack of documents. I will do the substantive document review, prepare the written responses to the document requests, etc. My secretary or paralegal can do the photocopying and numbering. If a firm has the resources (and I recognize that not all do), I maintain that this would be the proper way to handle such tasks. If you feel otherwise, you should let your clients know.

There is truly no such need. And the workflow described is preferable to the attorney sitting at the copier herself. But this largely manual process is substantially worse than someone (attorney or staff) clicking a couple of buttons and letting the machine do the work. Even though I was making the point that the software automates Bates stamping, this $250/hr professional still did not recognize the waste endemic to her “proper way.” Even if a client were not billed directly for the wasted labor, we would be charged for photocopying. And, in an environment where the lawyer-to-staff ratio continues to shrink, staff assigned to fake work is not available to assist elsewhere. There is no such thing as a free lunch; lawyers should not pretend that overhead doesn’t exist or is not ultimately passed on to their clients.

On task X, I would use Method 2 rather than Method 1 called for in the audit.

Fair enough. Your way may in fact be better. I often feel like a fraud in evangelizing the audit because I deem my own skills so inadequate. I know just enough to recognize my own ignorance and the even more profound ignorance of the profession at large.

Still, I am less focused on the specific tasks than on general skills. For example, I use a spreadsheet of deposition exhibits to test Excel acumen. Not all lawyers are litigators—i.e., no deposition exhibits. Not all litigators maintain their exhibit lists in Excel. Many firms assign exhibit-tracking duties to non-billable staff. Thus, whether an associate can skillfully manipulate deposition-exhibit data in a spreadsheet is not all that important in and of itself. But that does not change the fact that “Excel is everywhere you look in the business world.”

Like so many companies, we export a substantial percentage of our structured data to Excel spreadsheets to be sorted, filtered, rearranged, put in pivot tables, and disseminated throughout the enterprise. My team and I rely on over 50 massive spreadsheets that are updated on a monthly, and sometimes daily, basis. We frequently send these huge collections of data to outside counsel for use in litigation. If counsel does not know how to manipulate the data, they will waste substantial resources trying to locate the pertinent information. Those are the general Excel skills I’m testing. The fake exhibit list was simply a better testing option than using real proprietary data.

I’m a Mac person. I don’t use, or even like, the Microsoft Office Suite—Outlook, Word, Excel.

I don’t care. Use whatever software you prefer. I am not an authority on software. I am, however, keenly attuned to the file types—.xlsx, .docx, .pdf—we send to outside counsel. Thus, I use those file types in my audit. My counsel needs to be able to open my files, complete tasks efficiently, and send me files I can open. Beyond that, I am not particularly concerned with the tools they use.

You should use small law firms instead of BigLaw. They would pass your audit.

I do hire small law firms. Or, rather, I hire lawyers who happen to work at small law firms just as I hire lawyers who happen to work at large law firms. The size of the firm is rarely material, though it can be a proxy for other considerations—e.g., rates, geographic reach—that sometimes factor. If lawyers I trust were to move from a big firm to a small firm, or vice versa, my business would very likely move with them.

No law firm who works on my matters was selected solely because they passed my audit. No law firm has had business withheld solely because they didn’t pass my audit (though they have had their fees cut). I believe technological competence is worth considering. But I do not think technological competence is the most important, let alone only, consideration in selecting counsel. The efficiency gains from improved technology utilization do not match the efficiency and efficacy that come with hiring experienced, effective subject-matter experts.

That some of the senior lawyers on whom I rely for sage counsel may be afflicted with mild technophobia does not trouble me. I do, however, feel compelled to test their support system to determine whether the work they delegate will be handled efficiently. Thus, I only audit associates; the idea being that the associates will handle the low-value-added, labor-intensive drudgery at which the audit is aimed.

Regardless of who takes the audit, I don’t hold individual lawyers accountable for failing my audit. The firm is responsible for their lawyers’ facility with the tools the firm provides. Technological competence is nurture, not nature. But most firms tend to leave their lawyers to their own devices, literally.

I [the commenter] would pass the audit.

Me too. I have passed my audit. I know a number of other lawyers who would fly through it. The problem isn’t that no lawyers can pass my audit; the problem is that most lawyers should be able to pass my audit but can’t. Indeed, I feel as if my case would be more compelling if some of other lawyers had already passed. Unfortunately, the lawyers who accurately predict that they can pass my audit also rightfully see themselves as exceptions. Moreover, I would caution against being too cocky (a hard warning for a lawyer to heed). Bold predictions often precede abject failure. I have observed too many manifestations of the Dunning-Kruger Effect where profound ignorance fuel delusions of adequacy, which the audit quickly shatters. One audit literally ended in tears.

For example, most lawyers know how to apply basic text formatting in Word–e.g., bold, italic, center, justify. But few are familiar with styles and cross-references. In my presentation, I use styles and cross-references to insert a new provision into a properly formatted contract. This takes 12 seconds. The software updates all of the subsequent paragraph numbers and internal references (e.g., “pursuant to Paragraph XVII”). By contrast, the very real contract I inherited had to be updated manually. Inserting the same new provision takes 10 minutes instead of 12 seconds (50x as long). And that is for a relatively short, simple contract. With proper formatting, the time required is independent of the size of the document. Adding the new provision will only ever take 12 seconds. Conversely, the time necessary to manually update the old contract is dependent upon, and increases in lockstep with, the length and complexity of the document.

Properly formatted documents are a particularly potent example of technological incompetence. Such busywork is patently avoidable. Moreover, many firms already offer their lawyers a simple solution. Along with Word, many firms provide templates with the applicable styles accessible in the Quick Access toolbar. Yet, the templates are only useful if the users (i) know they exist, (ii) can load them, and (iii) make use of the styles. Most lawyers are ignorant on all three counts. Lawyers who don’t know about templates, styles, and cross-references won’t know that they should be using templates, styles, and cross-references. So they consider their existing formatting skills perfectly adequate. Only when this illusion is shattered do they realize just how much time (and client money) they have wasted.

Whether a contract is written correctly is more important than it being formatted correctly.

Agreed. But this is a false choice. Having a contract that is formatted correctly in no way undermines how well it is written. Rather, it makes getting the contract right more likely because the lawyer will not have their focus depleted by mind-numbing repetition (e.g., updating section numbers), which the machine handles better anyway (e.g., never misses a cross-reference). One of the four horseman of the new normal is the substitution of technology for people in repetitive work. While it may reduce the quantity of billable work, it should improve its quality, as well as the quality of life of the lawyers who have to do it.

D. Casey Flaherty is corporate counsel at Kia Motors America, Inc. where he oversees dealer-related legal matters and electronic discovery. Casey regularly writes and speaks about the audit, electronic discovery, budgeting, and other topics at the intersection of law and technology. His LegalTech West keynote on the audit can be viewed for free here. The opinions he expresses are his own and not those of Kia Motors America, Inc.

Editor’s note: The New Normal is an ongoing discussion between Paul Lippe, the CEO of Legal OnRamp, Patrick Lamb, founding member of Valorem Law Group and their guests. New Normal contributors spend a lot of time thinking, writing and speaking about the changes occurring in the delivery of legal services. You’re invited to join their discussion.