Systems and methods for conducting and terminating a technology-assisted review
First Claim
1. A system for terminating a classification process, the system comprising:
- at least one computing device having a processor and physical memory, the physical memory storing instructions that cause the processor to;
receive an identification of a target set of documents in a document collection, wherein the target set of documents consists of documents identified as relevant as part of a first search strategy and are stored on a non-transitory storage medium;
execute the classification process that enables training of a classifier using documents in the target set, wherein the classification process utilizes a second iterative search strategy which does not distinguish between documents in the target set and documents in the document collection to classify documents in the document collection; and
terminate the classification process based upon a comparison between the results of the second search strategy and a characteristic of the target set of documents, wherein the classification process achieves a target level of recall with a certain probability upon termination.
0 Assignments
0 Petitions
Accused Products
Abstract
Systems and methods are provided for classifying electronic information and terminating a classification process which utilizes Technology-Assisted Review (“TAR”) techniques. In certain embodiments, the TAR process, which is an iterative process, is terminated based upon one more stopping criteria. In certain embodiments, use of the stopping criteria ensures that the TAR process will reliably achieve a level of quality (e.g., recall) with a certain probability. In certain embodiments, the TAR process is terminated when it independently identifies a target set of documents. In certain embodiments, the TAR process is terminated based upon whether the ratio of the slope of the TAR process'"'"'s gain curve before an inflection point to the slope of the TAR process'"'"' gain curve after the inflection point exceeds a threshold. In certain embodiments, the TAR process is terminated when a review budget and slope ratio of the gain curve each exceed a respective threshold.
-
Citations
20 Claims
-
1. A system for terminating a classification process, the system comprising:
-
at least one computing device having a processor and physical memory, the physical memory storing instructions that cause the processor to; receive an identification of a target set of documents in a document collection, wherein the target set of documents consists of documents identified as relevant as part of a first search strategy and are stored on a non-transitory storage medium; execute the classification process that enables training of a classifier using documents in the target set, wherein the classification process utilizes a second iterative search strategy which does not distinguish between documents in the target set and documents in the document collection to classify documents in the document collection; and terminate the classification process based upon a comparison between the results of the second search strategy and a characteristic of the target set of documents, wherein the classification process achieves a target level of recall with a certain probability upon termination. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A computerized method for terminating a classification process, the method comprising:
-
receiving an identification of a target set of documents in a document collection, wherein the target set of documents consists of documents identified as relevant as part of a first search strategy and are stored on a non-transitory storage medium; executing the classification process that enables training of a classifier using documents in the target set, wherein the classification process utilizes a second iterative search strategy which does not distinguish between documents in the target set and documents in the document collection to classify documents in the document collection; and terminating the classification process based upon a comparison between the results of the second search strategy and a characteristic of the target set of documents, wherein the classification process achieves a target level of recall with a certain probability upon termination. - View Dependent Claims (12, 13, 14, 15, 16, 17, 18, 19, 20)
-
Specification