Systems and methods for conducting and terminating a technology-assisted review

• US 10,242,001 B2
• Filed: 06/17/2016
• Issued: 03/26/2019
• Est. Priority Date: 06/19/2015
• Status: Active Grant

• Current Assignee: Gordon V. Cormack, Maura R. Grossman
• Original Assignee: Gordon V. Cormack, Maura R. Grossman
• Inventors: Cormack, Gordon V., Grossman, Maura R.
• Primary Examiner(s): Phillips, III, Albert M
• Application Number:
  US15/186,377
• Publication Number:
  US 20160371369A1
• Time in Patent Office: 1,012 Days
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Systems and methods are provided for classifying electronic information and terminating a classification process which utilizes Technology-Assisted Review (“TAR”) techniques. In certain embodiments, the TAR process, which is an iterative process, is terminated based upon one more stopping criteria. In certain embodiments, use of the stopping criteria ensures that the TAR process will reliably achieve a level of quality (e.g., recall) with a certain probability. In certain embodiments, the TAR process is terminated when it independently identifies a target set of documents. In certain embodiments, the TAR process is terminated based upon whether the ratio of the slope of the TAR process'"'"'s gain curve before an inflection point to the slope of the TAR process'"'"' gain curve after the inflection point exceeds a threshold. In certain embodiments, the TAR process is terminated when a review budget and slope ratio of the gain curve each exceed a respective threshold.

Claims

1. A system for terminating a classification process, the system comprising:

at least one computing device having a processor and physical memory, the physical memory storing instructions that cause the processor to;

execute the classification process, wherein the classification process utilizes an iterative search strategy to classify documents in a document collection, wherein the documents in the document collection are stored on a non-transitory storage medium;

determine an upper bound for an expected review effort using an estimate of a number of relevant documents identified as part of the classification process;

select a gain curve slope ratio threshold;

compute points on a gain curve using a selected set of documents in the document collection and results from the classification process;

detect an inflection point in the gain curve;

determine a slope ratio for the detected inflection point using a slope of the gain curve before the detected inflection point, and a slope of the gain curve after the detected inflection point; and

terminate the classification process based upon a determination that the slope ratio for the detected inflection point exceeds the selected slope ratio threshold, and based upon a determination that the upper bound for the expected review effort has been exceeded.

2. The system of claim 1, wherein the upper bound is an estimate of an effort that would provide a reliable statistical estimate of the number of relevant documents in the document collection.

3. The system of claim 2, wherein the reliable statistical estimate is achieved by random sampling the document collection.

4. The system of claim 1, wherein the upper bound is refined during one or more iterations of the classification process.

5. The system of claim 1, wherein the classification process is terminated when a pre-determined portion of the document collection has been reviewed.

6. The system of claim 5, wherein the pre-determined portion is 75 percent of the document collection.

7. The system of claim 1, wherein the instructions further cause the processor to evaluate the reliability or the effort of the classification process using a quadratic loss function.

8. A computerized method for terminating a classification process, the method comprising:

executing the classification process, wherein the classification process utilizes an iterative search strategy to classify documents in a document collection, wherein the documents in the document collection are stored on a non-transitory storage medium;

determining an upper bound for an expected review effort using an estimate of a number of relevant documents identified as part of the classification process;

selecting a gain curve slope ratio threshold;

computing points on a gain curve using a selected set of documents in the document collection and results from the classification process;

detecting an inflection point in the gain curve;

determining a slope ratio for the detected inflection point using a slope of the gain curve before the detected inflection point, and a slope of the gain curve after the detected inflection point; and

terminating the classification process based upon a determination that the slope ratio for the detected inflection point exceeds the selected slope ratio threshold, and based upon a determination that the upper bound for the expected review effort has been exceeded.

9. The method of claim 8, wherein the upper bound is an estimate of an effort that would provide a reliable statistical estimate of the number of relevant documents in the document collection.

10. The method of claim 9, wherein the reliable statistical estimate is achieved by random sampling the document collection.

11. The method of claim 8, wherein the upper bound is refined during one or more iterations of the classification process.

12. The method of claim 8, wherein the classification process is terminated when a pre-determined portion of the document collection has been reviewed.

13. The method of claim 12, wherein the pre-determined portion is 75 percent of the document collection.

14. The method of claim 8, further comprising evaluating the reliability or the effort of the classification process using a quadratic loss function.