Electronic discovery system with statistical sampling

• US 9,667,514 B1
• Filed: 01/30/2013
• Issued: 05/30/2017
• Est. Priority Date: 01/30/2012
• Status: Active Grant

• Current Assignee: Consilio LLC
• Original Assignee: DiscoverReady LLC (Consilio LLC)
• Inventors: Wagner, Jr., James Kenneth, Ritter, John T., Barsony, Stephen John, Schreiber, Stephen H., Helgeson, Merideth L., Burke, Macyl A., Richards, Philip L., Shub, David Matthew, Messing, Yerachmiel Tzvi
• Primary Examiner(s): Coulter, Kenneth R
• Application Number:
  US13/754,800
• Time in Patent Office: 1,581 Days
• Field of Search: 709224, 709223, 709217-219, 715200, 705311, 707705, 707736-747
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06Q 10/0633 Workflow analysis

Abstract

A sampler tool for an electronic discovery system implementing an electronic discovery workflow is provided. The sampler tool defines a workflow, identifies, a data set, then calculates results and generates reports and archival records. In particular, the sampler tool calculates and/or obtains various metrics characterizing the workflow and generates reports based on such metrics. These reports can serve as historical snapshots which can be used for quality control purposes and/or to provide support regarding the efficacy of the data selection process or large-scale document review in the context of the electronic discovery process. Related apparatus, systems, techniques and articles are also described.

Claims

1. A computer-implemented method comprising:

obtaining, by at least one data processor and by a sampler tool, data from a plurality of applications and data sources used to implement a software-based electronic discovery workflow, the electronic discovery workflow being used to categorize each of a plurality of documents for potential production in connection with litigation discovery, the obtained data characterizing a current state of at least one aspect of the electronic discovery workflow;

generating, by the at least one data processor, at least one benchmark input field on a graphical user interface, each of the at least one benchmark input field accepting a user-defined benchmark describing a desired statistical feature of the data;

displaying, by the at least one data processor, on the graphical user interface, a recommended sample size indicating a number of documents that are likely needed to achieve at least one of the desired statistical features;

calculating, by the at least one data processor, based on the user-defined benchmarks and the data, at least one metric, each of the at least one metric corresponding to an actual statistical feature of the data with the recommended sample size;

displaying, by the at least one data processor, on the graphical user interface, the at least one metric;

generating, by the at least one data processor, a report based on the at least one metric;

displaying, on the graphical user interface, the report to a user;

providing, by the at least one data processor, data characterizing the report to a data store;

updating, by the at least one data processor with a write-back operation to the data store, an updated report based on detected changes in the at least one benchmark input field;

providing, by the at least one data processor, updated data characterizing the updated report; and

displaying, on the graphical user interface, the updated report to the user.

2. A method as in claim 1, wherein providing data comprises at least one of:

displaying, by at least one data processor, at least a portion of the report in a graphical user interface, loading, by at least one data processor, at least a portion of the report, storing at least a portion of the report, and transmitting, by at least one data processor, at least a portion of the report.

3. A method as in claim 1, wherein the data is obtained from the applications via corresponding application programming interfaces (APIs) for each of the applications.

4. A method as in claim 1, wherein the documents are selected from a group consisting of:

word processing documents, spreadsheet documents, electronic presentations, e-mails, audio files, images, videos, data objects, computer code, container files, system files, log files, redactions, CAD files, program files, metadata, case information, custodian information, and project files.

5. A method as in claim 1, further comprising:

receiving, by at least one data processor via a graphical user interface, user-generated input specifying aspects relating to the electronic discovery workflow for analysis;

wherein the obtained data is based on the specified aspects in the user-generated input.

6. A method as in claim 1, wherein the at least one metric is selected from a group of metrics relating to the electronic discovery workflow consisting of:

completion rate, error rate, performance in relation to pre-defined benchmarks, confidence level, error margin, precision, recall, F-measure accuracy for at least a portion of the electronic discovery workflow, efficacy level for screening or review of potentially privileged data, accuracy level of specific classes of decisions, differentiation among levels of materiality, proper categorization of documents, and sample size.

7. A method as in claim 1, wherein providing data comprises:

displaying, by at least one data processor, at least a portion of the report in a graphical user interface, the graphical user interface comprising at least one graphical user interface element having associated complementary information, wherein the method further comprises;

receiving, by at least one data processor via the graphical user interface, user-generated input activating the at least one graphical user interface element; and

displaying, by at least one data processor in the graphical user interface, the associated complementary information.

8. A method as in claim 7, wherein the associated complementary information comprises at least a portion of a document used to calculate the at least one metric.

9. A method as in claim 7, wherein the associated complementary information comprises at least a portion of the obtained data.

10. A method as in claim 7, wherein the associated complementary information comprises at least one categorization decision assigning at least one document to a particular category.

11. A method as in claim 10, further comprising:

receiving, by at least one data processor, via the graphical user interface, user-generated input changing the at least one categorization decision; and

transmitting, by at least one data processor, data to at least one of the plurality of applications and data sources characterizing the changed at least one categorization decision.

12. A method as in claim 11, further comprising:

modifying, by at least one data processor, the displayed report to accommodate the changed at least one categorization decision.

13. A non-transitory computer program product storing instructions, which when executed by at least one data processor of at least one computing system, result in operations comprising:

obtaining, by a sampler tool, data from a plurality of applications and data sources used to implement a software-based electronic discovery workflow, the electronic discovery workflow being used to categorize each of a plurality of documents for potential production in connection with litigation discovery, the obtained data characterizing a current state of at least one aspect of the electronic discovery workflow;

generating at least one benchmark input field on a graphical user interface, each of the at least one benchmark input field accepting a user-defined benchmark describing a desired statistical feature of the data;

displaying, on the graphical user interface, a recommended sample size indicating a number of documents that are likely needed to achieve at least one of the desired statistical features;

calculating, based on the user-defined benchmarks and the data, at least one metric, each of the at least one metric corresponding to an actual statistical feature of the data with the recommended sample size;

displaying, on the graphical user interface, the at least one metric;

generating a report based on the at least one metric;

displaying, on the graphical user interface, the report to a user;

providing data characterizing the report to a data store;

updating, by a write-back operation to the data store, an updated report based on detected changes in the at least one benchmark input field;

providing updated data characterizing the updated report; and

displaying, on the graphical user interface, the updated report to the user.

14. A computer program product as in claim 13, wherein providing data comprises at least one of:

displaying at least a portion of the report in the graphical user interface, loading at least a portion of the report, storing at least a portion of the report, and transmitting at least a portion of the report.

15. A computer program product as in claim 13, wherein the data is obtained from the applications via corresponding application programming interfaces (APIs) for each of the applications.

16. A computer program product as in claim 13, wherein the documents are selected from a group consisting of:

word processing documents, spreadsheet documents, electronic presentations, e-mails, audio files, images, videos, data objects, computer code, container files, system files, log files, redactions, CAD files, program files, metadata, case information, custodian information, and project files.

17. A computer program product as in claim 13, wherein the at least one metric is selected from a group of metrics relating to the electronic discovery workflow consisting of:

completion rate, error rate, performance in relation to pre-defined benchmarks, confidence level, error margin, precision, recall, F-measure accuracy for at least a portion of the electronic discovery workflow, efficacy level for screening or review of potentially privileged data, accuracy level of specific classes of decisions, differentiation among levels of materiality, proper categorization of documents, and sample size.

18. A computer program product as in claim 13, wherein providing data comprises:

displaying at least a portion of the report in the graphical user interface, the graphical user interface comprising at least one graphical user interface element having associated complementary information, wherein the wherein the operations further comprise;

receiving, via the graphical user interface, user-generated input activating the at least one graphical user interface element; and

displaying, in the graphical user interface, the associated complementary information.

19. A computer program product as in claim 18, wherein the associated complementary information comprises at least a portion of a document used to calculate the at least one metric.

20. A computer program product as in claim 18, wherein the associated complementary information comprises at least a portion of the obtained data.

21. A computer program product as in claim 18, wherein the associated complementary information comprises at least one categorization decision assigning at least one document to a particular category.

22. A computer program product as in claim 21, wherein the operations further comprise:

receiving, via the graphical user interface, user-generated input changing the at least one categorization decision; and

transmitting data to at least one of the plurality of applications and data sources characterizing the changed at least one categorization decision.

23. A computer program product as in claim 22, wherein the operations further comprise:

modifying the displayed report to accommodate the changed at least one categorization decision.

24. A system comprising:

at least one data processor; and

memory storing instructions, which when executed by the at least one data processor, result in operations comprising;

obtaining, by a sampler tool, data from a plurality of applications and data sources used to implement a software-based electronic discovery workflow, the electronic discovery workflow being used to categorize each of a plurality of documents for potential production in connection with litigation discovery, the obtained data characterizing a current state of at least one aspect of the electronic discovery workflow;

generating at least one benchmark input field on a graphical user interface, each of the at least one benchmark input field accepting a user-defined benchmark describing a desired statistical feature of the data;

displaying, on the graphical user interface, a recommended sample size indicating a number of documents that are likely needed to achieve at least one of the desired statistical features;

calculating, based on the user-defined benchmarks and the data, at least one metric, each of the at least one metric corresponding to an actual statistical feature of the data with the recommended sample size;

displaying, on the graphical user interface, the at least one metric;

generating a report based on the at least one metric;

displaying, on the graphical user interface, the report to a user;

providing data characterizing the report to a data store;

updating, by a write-back operation to the data store, an updated report based on detected changes in the at least one benchmark input field;

providing updated data characterizing the updated report; and

displaying, on the graphical user interface, the updated report to the user.