×

Systems and methods for conducting and terminating a technology-assisted review

  • US 10,353,961 B2
  • Filed: 06/17/2016
  • Issued: 07/16/2019
  • Est. Priority Date: 06/19/2015
  • Status: Active Grant
First Claim
Patent Images

1. A system for terminating a classification process, the system comprising:

  • at least one computing device having a processor and physical memory, the physical memory storing instructions that cause the processor to;

    execute the classification process, wherein the classification process utilizes an iterative search strategy that presents documents to a human reviewer for training a classifier to classify documents in a document collection and the documents are stored on a non-transitory storage medium;

    receive a user coding decision from the human reviewer and train the classifier using the received user coding decision;

    select a gain curve slope ratio threshold;

    compute points on a gain curve using a selected set of documents in the document collection and results from the classification process, the points on the gain curve relating a ranking of the selected set of documents to the number of relevant documents retrieved at one or more ranks of the ranking, wherein the ranking relates to an order in which the documents were presented to the human reviewer;

    detect an inflection point in the gain curve, wherein to detect the inflection point in the gain curve, the instructions further cause the processor to;

    solve for parameters of a line running from an origin of the gain curve to a first point on the gain curve corresponding to a level of recall achieved at a rank of one document in the selected set of documents; and

    determine the inflection point as a point on the gain curve from where a perpendicular line of suitable length extends to the line for which the parameters were solved, wherein the perpendicular line of suitable length is a longest perpendicular line;

    determine a candidate rank associated with the detected inflection point, wherein the candidate rank is a projection of the intersection of the perpendicular line of suitable length from the gain curve and the gain curve onto an axis of the gain curve;

    determine a slope ratio for the detected inflection point using a slope of the gain curve before the detected inflection point, and a slope of the gain curve after the detected inflection point; and

    terminate the presentation of documents to the human reviewer in the classification process and classify one more documents in the document collection using the received user coding decision or scores generated by the classifier based upon a determination that the slope ratio for the detected inflection point exceeds the selected slope ratio threshold,continue the classification process based upon a determination that the slope ratio for the detected inflection point does not exceed the selected slope ratio threshold by selecting and presenting one or more documents to the human reviewer for additional user coding decisions, the selection of the presented document being based on the trained classifier.

View all claims
  • 0 Assignments
Timeline View
Assignment View
    ×
    ×