Systems and methods for conducting and terminating a technology-assisted review

• US 10,353,961 B2
• Filed: 06/17/2016
• Issued: 07/16/2019
• Est. Priority Date: 06/19/2015
• Status: Active Grant

• Current Assignee: Gordon V. Cormack, Maura R. Grossman
• Original Assignee: Gordon V. Cormack, Maura R. Grossman
• Inventors: Cormack, Gordon V., Grossman, Maura R.
• Primary Examiner(s): Obisesan, Augustine K.
• Application Number:
  US15/186,366
• Publication Number:
  US 20160371260A1
• Time in Patent Office: 1,124 Days
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Systems and methods are provided for classifying electronic information and terminating a classification process which utilizes Technology-Assisted Review (“TAR”) techniques. In certain embodiments, the TAR process, which is an iterative process, is terminated based upon one more stopping criteria. In certain embodiments, use of the stopping criteria ensures that the TAR process will reliably achieve a level of quality (e.g., recall) with a certain probability. In certain embodiments, the TAR process is terminated when it independently identifies a target set of documents. In certain embodiments, the TAR process is terminated based upon whether the ratio of the slope of the TAR process'"'"'s gain curve before an inflection point to the slope of the TAR process'"'"' gain curve after the inflection point exceeds a threshold. In certain embodiments, the TAR process is terminated when a review budget and slope ratio of the gain curve each exceed a respective threshold.

Claims

1. A system for terminating a classification process, the system comprising:

at least one computing device having a processor and physical memory, the physical memory storing instructions that cause the processor to;

execute the classification process, wherein the classification process utilizes an iterative search strategy that presents documents to a human reviewer for training a classifier to classify documents in a document collection and the documents are stored on a non-transitory storage medium;

receive a user coding decision from the human reviewer and train the classifier using the received user coding decision;

select a gain curve slope ratio threshold;

compute points on a gain curve using a selected set of documents in the document collection and results from the classification process, the points on the gain curve relating a ranking of the selected set of documents to the number of relevant documents retrieved at one or more ranks of the ranking, wherein the ranking relates to an order in which the documents were presented to the human reviewer;

detect an inflection point in the gain curve, wherein to detect the inflection point in the gain curve, the instructions further cause the processor to;

solve for parameters of a line running from an origin of the gain curve to a first point on the gain curve corresponding to a level of recall achieved at a rank of one document in the selected set of documents; and

determine the inflection point as a point on the gain curve from where a perpendicular line of suitable length extends to the line for which the parameters were solved, wherein the perpendicular line of suitable length is a longest perpendicular line;

determine a candidate rank associated with the detected inflection point, wherein the candidate rank is a projection of the intersection of the perpendicular line of suitable length from the gain curve and the gain curve onto an axis of the gain curve;

determine a slope ratio for the detected inflection point using a slope of the gain curve before the detected inflection point, and a slope of the gain curve after the detected inflection point; and

terminate the presentation of documents to the human reviewer in the classification process and classify one more documents in the document collection using the received user coding decision or scores generated by the classifier based upon a determination that the slope ratio for the detected inflection point exceeds the selected slope ratio threshold,

continue the classification process based upon a determination that the slope ratio for the detected inflection point does not exceed the selected slope ratio threshold by selecting and presenting one or more documents to the human reviewer for additional user coding decisions, the selection of the presented document being based on the trained classifier.

2. The system of claim 1, wherein the classification process is a technology-assisted review process.

3. The system of claim 2, wherein the technology-assisted review process is a continuous active learning process.

4. The system of claim 1, wherein the slope ratio threshold is 6.0.

5. The system of claim 1, wherein the slope ratio threshold is adjusted during iterations of the classification process.

6. The system of claim 5, wherein the slope ratio is adjusted based on the number of relevant documents retrieved during iterations of the classification process.

7. The system of claim 1, wherein the points on the gain curve are computed using scores generated by applying a classifier to document information profiles for the selected set of documents.

8. The system of claim 1, wherein the set of documents is selected from a batch of documents presented to a reviewer during an iteration of the classification process.

9. The system of claim 1, wherein the gain curve is smoothed by fitting a curve between two or more of the computed points on the gain curve.

10. The system of claim 9, wherein the fitted curve is linear.

11. The system of claim 9, wherein the fitted curve is quadratic.

12. The system of claim 1, wherein the gain curve is smoothed by fitting a first curve between a first set of computed points on the gain curve and fitting a second curve between a second set of computed points on the gain curve.

13. The system of claim 1, wherein the slope ratio of the gain curve is calculated at the candidate rank according to the equation:

14. The system of claim 13, where SF is non-zero.

15. The system of claim 1, wherein the instructions further cause the processor to evaluate the reliability or the effort of the classification process using a quadratic loss function.

16. A computerized method for terminating a classification process, the method comprising:

executing the classification process, wherein the classification process utilizes an iterative search strategy that presents documents to a human reviewer for training a classifier to classify documents in a document collection and the documents are stored on a non-transitory storage medium;

receive a user coding decision from the human reviewer and train the classifier using the received user coding decision;

selecting a gain curve slope ratio threshold;

computing points on a gain curve using a selected set of documents in the document collection and results from the classification process, the points on the gain curve relating a ranking of the selected set of documents to the number of relevant documents retrieved at one or more ranks of the ranking, wherein the ranking relates to an order in which the documents were presented to the human reviewer;

detecting an inflection point in the gain curve, wherein the inflection point is detected by;

solving for parameters of a line running from an origin of the gain curve to a first point on the gain curve corresponding to a level of recall achieved at a rank of one document in the selected set of documents; and

determining the inflection point as a point on the gain curve from where a perpendicular line of suitable length extends to the line for which the parameters were solved, wherein the perpendicular line of suitable length is a longest perpendicular line;

determining a candidate rank associated with the detected inflection point, wherein the candidate rank is a projection of the intersection of the perpendicular line of suitable length from the gain curve and the gain curve onto an axis of the gain curve;

determining a slope ratio for the detected inflection point using a slope of the gain curve before the detected inflection point, and a slope of the gain curve after the detected inflection point; and

terminating the presentation of documents to the human reviewer in the classification process and classifying one more documents in the document collection using the received user coding decision or scores generated by the classifier based upon a determination that the slope ratio for the detected inflection point exceeds the selected slope ratio threshold,

continuing the classification process based upon a determination that the slope ratio for the detected inflection point does not exceed the selected slope ratio threshold by selecting and presenting one or more documents to the human reviewer for additional user coding decisions, the selection of the presented document being based on the trained classifier.

17. The method of claim 16, wherein the classification process is a technology-assisted review process.

18. The method of claim 17, wherein the technology-assisted review process is a continuous active learning process.

19. The method of claim 16, wherein the slope ratio threshold is 6.0.

20. The method of claim 16, wherein the slope ratio threshold is adjusted during iterations of the classification process.

21. The method of claim 20, wherein the slope ratio is adjusted based on the number of relevant documents retrieved during iterations of the classification process.

22. The method of claim 16, wherein the points on the gain curve are computed using scores generated by applying a classifier to document information profiles for the selected set of documents.

23. The method of claim 16, wherein the set of documents is selected from a batch of documents presented to a reviewer during an iteration of the classification process.

24. The method of claim 16, wherein the gain curve is smoothed by fitting a curve between two or more of the computed points on the gain curve.

25. The method of claim 24, wherein the fitted curve is linear.

26. The method of claim 24, wherein the fitted curve is quadratic.

27. The method of claim 16, wherein the gain curve is smoothed by fitting a first curve between a first set of computed points on the gain curve and fitting a second curve between a second set of computed points on the gain curve.

28. The method of claim 16, wherein the slope ratio of the gain curve is calculated at the candidate rank according to the equation:

29. The method of claim 28, where SF is non-zero.

30. The method of claim 16, further comprising evaluating the reliability or the effort of the classification process using a quadratic loss function.