Systems and methods for conducting and terminating a technology-assisted review

• US 10,445,374 B2
• Filed: 06/17/2016
• Issued: 10/15/2019
• Est. Priority Date: 06/19/2015
• Status: Active Grant

• Current Assignee: Gordon V. Cormack, Maura R. Grossman
• Original Assignee: Gordon V. Cormack, Maura R. Grossman
• Inventors: Cormack, Gordon V., Grossman, Maura R.
• Primary Examiner(s): Robinson, Greta L
• Application Number:
  US15/186,360
• Publication Number:
  US 20160371364A1
• Time in Patent Office: 1,215 Days
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Systems and methods are provided for classifying electronic information and terminating a classification process which utilizes Technology-Assisted Review (“TAR”) techniques. In certain embodiments, the TAR process, which is an iterative process, is terminated based upon one more stopping criteria. In certain embodiments, use of the stopping criteria ensures that the TAR process will reliably achieve a level of quality (e.g., recall) with a certain probability. In certain embodiments, the TAR process is terminated when it independently identifies a target set of documents. In certain embodiments, the TAR process is terminated based upon whether the ratio of the slope of the TAR process'"'"'s gain curve before an inflection point to the slope of the TAR process'"'"' gain curve after the inflection point exceeds a threshold. In certain embodiments, the TAR process is terminated when a review budget and slope ratio of the gain curve each exceed a respective threshold.

Claims

1. A system for terminating a classification process, the system comprising:

at least one computing device having a processor and physical memory, the physical memory storing instructions that cause the processor to;

receive an identification of a target set of documents in a document collection, wherein the target set of documents consists of documents identified as relevant as part of a first search strategy and are stored on a non-transitory storage medium;

execute the classification process that enables training of a classifier using documents in the target set, wherein the classification process utilizes a second iterative search strategy which does not distinguish between documents in the target set and documents in the document collection to classify documents in the document collection; and

terminate the classification process based upon a comparison between the results of the second search strategy and a characteristic of the target set of documents, wherein the classification process achieves a target level of recall with a certain probability upon termination.

2. The system of claim 1, wherein the target set is identified through a review of documents randomly sampled from the document collection.

3. The system of claim 1, wherein the second search strategy is a technology-assisted review process.

4. The system of claim 3, wherein the technology-assisted review process is a continuous active learning process.

5. The system of claim 1, wherein the termination is in response to a determination that a sufficient number of documents in the target set have been classified as relevant by the second search strategy.

6. The system of claim 5, wherein the sufficient number corresponds to the number of documents in the target set.

7. The system of claim 1, wherein the termination is in response to a determination that a distribution of types of documents classified by the classification process has satisfied a pre-defined target distribution constraint on types of documents associated with the target set.

8. The system of claim 7, wherein the target distribution constraint is based upon the occurrence of target set documents and relevant documents that are identified by a human reviewer during the classification process.

9. The system of claim 7, wherein the target distribution constraint is based upon the sequence of classifications assigned to documents presented to a human reviewer during the classification process.

10. The system of claim 1, wherein the instructions further cause the processor to evaluate the reliability or the effort of the classification process using a quadratic loss function.

11. A computerized method for terminating a classification process, the method comprising:

receiving an identification of a target set of documents in a document collection, wherein the target set of documents consists of documents identified as relevant as part of a first search strategy and are stored on a non-transitory storage medium;

executing the classification process that enables training of a classifier using documents in the target set, wherein the classification process utilizes a second iterative search strategy which does not distinguish between documents in the target set and documents in the document collection to classify documents in the document collection; and

terminating the classification process based upon a comparison between the results of the second search strategy and a characteristic of the target set of documents, wherein the classification process achieves a target level of recall with a certain probability upon termination.

12. The method of claim 11, wherein the target set is identified through a review of documents randomly sampled from the document collection.

13. The method of claim 11, wherein the second search strategy is a technology-assisted review process.

14. The method of claim 13, wherein the technology-assisted review process is a continuous active learning process.

15. The method of claim 11, wherein the termination is in response to a determination that a sufficient number of documents in the target set have been classified as relevant by the second search strategy.

16. The method of claim 15, wherein the sufficient number corresponds to the number of documents in the target set.

17. The method of claim 11, wherein the termination is in response to a determination that a distribution of types of documents classified by the classification process has satisfied a pre-defined target distribution constraint on types of documents associated with the target set.

18. The method of claim 17, wherein the target distribution constraint is based upon the occurrence of target set documents and relevant documents that are identified by a human reviewer during the classification process.

19. The method of claim 17, wherein the target distribution constraint is based upon the sequence of classifications assigned to documents presented to a human reviewer during the classification process.

20. The method of claim 11, further comprising evaluating the reliability or the effort of the classification process using a quadratic loss function.