5 Questions About Discovery Quality Management
An interview with Deloitte Financial Advisory Services LLP's Discovery directors, Steve Shebest and Chris Knox.
Establishing and maintaining a quality process during Discovery activities is critical, particularly given the volume of data, variety of tools and differentiation of recognized practices that can apply. The eDiscovery industry has not defined a baseline for quality and, as such, multiple competing quality initiatives are underway. It is important to understand the impact of such quality initiatives on purchasers as well as the industry supply chain.
|What was the recent action by the International Organization for Standardization and how might it impact discovery?||
The International Organization for Standardization recently gave its final approval for the development of an international standard relating to the use of Electronically Stored Information (ESI) during the eDiscovery process (ISO/IEC 27050). Its passage has heightened awareness of the lack of industry standards related to ESI and eDiscovery.
Whether the standard comes to fruition or not, ISO/IEC 27050 – like other ISO initiatives – has invigorated interest and has sparked a discussion of safe, reliable processes that minimize error and reduce cost. This is the definition of quality of process.
|Are there other initiatives in this area?||Yes. The ISO 27050 initiative is one of several nods to the importance of quality of process in the eDiscovery industry. There have been efforts by various industry participants to address the lack of a quality benchmark by utilizing or encouraging not only ISO frameworks 27001 and 90012, but also other recognized quality-related systems including AICPA’s Service Organizational Control (“SOC”), Six Sigma and the Project Management Institute’s PMP structure. The Sedona Conference’s Commentary on Achieving Quality in the E-Discovery Process spoke to a need that each of these frameworks attempts to address: How do we apply the fundamentals of quality management to a set of processes such as eDiscovery?|
|How do the fundamentals of quality management apply to electronic discovery?||
Quality management requires a systematic approach to quality assurance, control and improvement. Quality assurance (“QA”) means having checks in place to ensure each of your processes are executed correctly. Quality control (“QC”) validates the quality assurance effort by measuring the end result of your processes. Quality improvement, particularly continuous quality improvement, is the discipline of taking the variance from your expected results found either at the QA or QC stages and revising the appropriate process to minimize that risk of variance moving forward.
How do you attack variance of process? Galileo is credited with admonishing us to count what is countable, measure what is measurable and to make measurable that which isn’t already measurable. Likewise, Peter Drucker suggested that if you can’t measure it, you can’t manage it. Six Sigma’s DMAIC framework proposes that we Define, Measure, Analyze, Improve and then Control. There is a theme here worth acknowledging. For example, how do you know that the document reviewers understand the coding manual if you don’t audit their coding, particularly right after the training? Is one contract attorney moving significantly slower through the review sets due to a lack of effort or being stuck with 30-tab excel spreadsheets or very large files that take minutes for each to load to the review pane? Does each document in your production set coded as “redacted” actually have a redaction applied somewhere? Do your “exception” documents really constitute a valid exception or was there a correctable issue with processing? How accurate are your search terms and how many “false positives” are they generating, resulting in additional review cost?
All of the various means to measure and identify outliers or variances from expectations will ideally roll up into a documented process for overall quality management. All of the systems and frameworks mentioned above are means to the end of framing that quality of process.
|How have providers of discovery services sought to distinguish themselves within the quality environment?||
The electronic discovery service provider market is a fractured one, with local, regional, national and global players. Further, providers differ in the scope of offered services across the spectrum from identification and collection, to processing and hosting to document review. Finally, providers utilize different technological tools with varying capabilities and limitations. All of this can contribute to an array of different risks and experiences for the same basic set of discovery requirements.
Due to the fractured marketplace and lack of an industry benchmark, service providers have sought to differentiate themselves in numerous ways, many of them related to quality. Some have achieved quality of process through audit regimen and/or certification in other areas, e.g. ISO 9001 or AICPA SOC 2. Others have endeavored to bolster their commitment to quality by ensuring that project personnel have a grounding or certification in a relevant discipline such as Six Sigma or the Project Management Institute’s PMP. Finally, others have eschewed formal frameworks but built in rigorous quality audits such as statistical sampling to verify processes.
|In the absence of an existing, independent benchmark for quality of process, what actions can a purchaser of discovery services take to proactively manage the quality of electronic discovery obligations?||
When vetting a service provider, ask the tough questions related to their Quality Management processes. For example, you may want to determine:
Finally, even after you have selected your provider, consider applying a disciplined audit of the resulting work product. Have a plan tied to either your existing internal approaches to quality or expectations and previous pain points related to the eDiscovery process.
Our take: Quality management is something that can and should, begin at home as the industry matures and promulgates objective benchmarks for measuring the practices, partners and protocols of the discovery process.
The recent developments on the international stage serve to underline not only the importance of understanding the fundamentals of quality management but also having a plan in place prior to entering the discovery process. As John Ruskin once said, “quality is never an accident; it is always the result of intelligent effort.” As the cost and intricacy of discovery of electronically stored information increases, the need for consistency and defensibility of process does as well. Working with your discovery team to understand how they are managing quality is a critical, yet often overlooked, key to success.
1ISO 27001 – sets out the requirements for an information security management system
2ISO 9001 - sets out the requirements for a quality management system
As used in this document, “Deloitte” means Deloitte LLP and its subsidiaries. Please see www.deloitte.com/us/about for a detailed description of the legal structure of Deloitte LLP and its subsidiaries. Certain services may not be available to attest clients under the rules and regulations of public accounting.