Electronic review of documents

• US 9,269,053 B2
• Filed: 04/27/2012
• Issued: 02/23/2016
• Est. Priority Date: 04/28/2011
• Status: Active Grant

• Current Assignee: Kldiscovery Ontrack, LLC (KLDiscovery, Inc.)
• Original Assignee: Kroll Ontrack Incorporated (KLDiscovery, Inc.)
• Inventors: Naslund, Jeffrey David, Thompson, Kevin B., Lewis, David Dolan
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Figueroa, Kevin W
• Application Number:
  US13/458,219
• Publication Number:
  US 20120278266A1
• Time in Patent Office: 1,397 Days
• Field of Search: 706/45
• US Class Current: 1/1
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

An example method for reviewing documents includes scoring documents using an artificial intelligence model, and selecting a subset of highest scoring documents. The method further includes inserting a number of randomly-selected documents into the subset of highest scoring documents to form a set of documents for review, wherein a reviewer cannot differentiate between the randomly-selected documents and the subset of highest scoring documents included in the set of documents for review, and presenting the set of documents for review by the reviewer.

Claims

1. A method for reviewing a collection of documents, the method comprising:

forming a set of scored documents by scoring all documents in the collection of documents using an artificial intelligence model, wherein every document in the collection of documents has not been scored previously;

forming a subset of scored documents by selecting a subset of documents meeting a certain criteria from the set of scored documents;

forming a set of randomly-selected documents by randomly-selecting one or more documents from the collection of documents;

forming a set of documents for review, the set of documents for review including both the subset of scored documents and the set of randomly-selected documents;

presenting the set of documents for review by a reviewer in a manner that does not differentiate between the set of randomly-selected documents and the subset of scored documents;

outputting, by the artificial intelligence model, a class label encoding a prediction of whether a particular document in the set of documents for review is responsive or nonresponsive; and

if the particular document is responsive, outputting, by the artificial intelligence model, a prediction of a secondary class label associated with the particular document,

wherein the secondary class label encodes a prediction of the particular document'"'"'s membership in one or more classes including;

areas of law or type of document, wherein the type of document includes marketing document, sales document, and technical document.

2. The method of claim 1, further comprising receiving class labels assigned by the reviewer to one or more of the documents in the set of documents for review.

3. The method of claim 2, further comprising:

using one or more of the class labels assigned by the reviewer to further train the artificial intelligence model; and

re-scoring all documents in the collection of documents using the artificial intelligence model.

4. The method of claim 2, further comprising verifying that the reviewer provided at least one of the class labels for each document in the set of documents for review.

5. The method of claim 2, wherein at least one of the class labels indicates if a particular document is responsive or nonresponsive.

6. The method of claim 2, further comprising:

receiving a first class label assigned by the reviewer for a particular document in the set of documents for review indicating if the particular document is responsive or nonresponsive; and

receiving a second class label assigned by the reviewer for the particular document indicating the particular document'"'"'s membership in one or more classes including;

area of law;

type of document;

or importance.

7. The method of claim 1, wherein the artificial intelligence model is a binary classifier outputting a class label.

8. The method of claim 1, further comprising calculating a size of a sample to be drawn from the documents using:

(i) an estimate of a final number of documents that will be collected;

(ii) a confidence level specifying a degree of confidence that estimates produced using a random sample should have;

(iii) a confidence interval half-width specifying how wide a confidence interval produced using the random sample should be; and

(iv) an estimate of a maximum plausible proportion of documents accounted for by a most frequent class in some class distinction.

9. The method of claim 8, further comprising assigning a random number to each of the documents to be reviewed, the random number being used to select which documents are included in the random sample.

10. The method of claim 1, further comprising allowing a user to select a blend of training and testing documents.

11. A method for reviewing a collection of documents for production during litigation, the method comprising:

forming a set of scored documents by scoring the collection of documents for production using an artificial intelligence model, wherein every document in the collection of documents for production has not been scored previously;

selecting a subset of highest scoring documents from the set of scored documents;

selecting one or more randomly-selected documents from the collection of documents;

forming a set of documents for review, the set of documents for review including both the subset of highest scoring documents and the randomly-selected documents;

presenting the set of documents for review by a reviewer in a manner that does not differentiate between the randomly-selected documents and the highest scoring documents;

receiving a class label assigned by the reviewer for one or more of the documents in the set of documents for review, wherein the class label indicates if one or more of the documents are responsive or nonresponsive; and

allowing the reviewer to assign a second class label to the one or more documents, the second class label including one or more of;

area of law;

type of document;

or importance, wherein the type of document includes marketing document, sales document, and technical document.

12. The method of claim 11, further comprising:

using the class label assigned by the reviewer to further train the artificial intelligence model; and

re-scoring the documents using the artificial intelligence model.

13. The method of claim 11, further comprising calculating a size of a sample to be drawn from the documents for production using:

(i) an estimate of a final number of documents that will be collected for production;

(ii) a confidence level specifying a degree of confidence that estimates produced using a random sample should have;

(iii) a confidence interval half-width specifying how wide a confidence interval produced using the random sample should be; and

(iv) an estimate of a maximum plausible proportion of documents accounted for by a most frequent class in some class distinction.

14. The method of claim 13, further comprising assigning a random number to each of the documents to be reviewed, the random number being used to select which documents are included in the random sample.

15. A method for reviewing a collection of documents for production during litigation, the method comprising:

forming a set of scored documents by scoring the collection of documents for production using an artificial intelligence model, wherein every document in the collection of documents for production has not been scored previously;

selecting a subset of highest scoring documents from the set of scored documents;

selecting one or more randomly-selected documents from the collection of documents for production;

forming a set of documents for review, the set of documents for review including both the subset of highest scoring documents and the randomly-selected documents;

presenting the set of documents for review by a reviewer in a manner that does not differentiate between the randomly-selected documents and the highest scoring documents included in the set of documents for review;

receiving a class label assigned by the reviewer for one or more of the documents in the set of documents for review;

allowing the reviewer to assign a second class label to the one or more documents, the second class label including one or more of;

area of law;

type of document;

or importance, wherein the type of document includes marketing document, sales document, and technical document; and

calculating a sample set of the documents for production using;

(i) an estimate of a total number of the documents for production;

(ii) a confidence level specifying a degree of confidence that estimates produced using a random sample should have;

(iii) a confidence interval half-width specifying how wide a confidence interval produced using the random sample should be; and

(iv) an estimate of a maximum plausible proportion of documents accounted for by a most frequent class in some class distinction.

16. The method of claim 15, further comprising assigning a random number to each of the documents to be reviewed, the random number being used to select which documents are included in the random sample.