Electronic Review of Documents

• US 20120278266A1
• Filed: 04/27/2012
• Published: 11/01/2012
• Est. Priority Date: 04/28/2011
• Status: Active Grant

• Current Assignee: Kldiscovery Ontrack, LLC (KLDiscovery, Inc.)
• Original Assignee: Kroll Ontrack Incorporated (KLDiscovery, Inc.)
• Inventors: Lewis, David Dolan, Naslund, Jeffrey David, Thompson, Kevin B.
• Granted Patent: US 9,269,053 B2
• Time in Patent Office: Days
• US Class Current: 706/45
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

An example method for reviewing documents includes scoring documents using an artificial intelligence model, and selecting a subset of highest scoring documents. The method further includes inserting a number of randomly-selected documents into the subset of highest scoring documents to form a set of documents for review, wherein a reviewer cannot differentiate between the randomly-selected documents and the subset of highest scoring documents included in the set of documents for review, and presenting the set of documents for review by the reviewer.

Claims

1. A method for reviewing documents, the method comprising:

scoring the documents using an artificial intelligence model;

selecting a subset of documents based on scoring by the artificial intelligence model;

selecting one or more randomly-selected documents from the documents;

forming a set of documents for review, the set of documents for review including both the subset of documents and the randomly-selected documents; and

presenting the set of documents for review by a reviewer in a manner that does not differentiate between the randomly-selected documents and the subset of documents.

2. The method of claim 1, further comprising receiving class labels assigned by the reviewer to one or more of the documents in the set of documents for review.

3. The method of claim 2, further comprising:

using one or more of the class labels assigned by the reviewer to further train the artificial intelligence model; and

re-scoring the documents using the artificial intelligence model.

4. The method of claim 2, further comprising verifying that the reviewer provided at least one of the class labels for each document in the set of documents for review.

5. The method of claim 2, wherein at least one of the class labels indicates if a particular document is responsive or nonresponsive.

6. The method of claim 2, further comprising:

outputting, by the artificial intelligence model, a class label encoding a prediction of whether a particular document is responsive or nonresponsive; and

if the particular document is responsive, outputting, by the artificial intelligence model, a prediction of a secondary class label associated with the particular document.

7. The method of claim 6, wherein the secondary class label encodes a prediction of the particular document'"'"'s membership in one or more classes including:

areas of law;

types of document;

or levels of importance.

8. The method of claim 2, further comprising:

receiving a first class label assigned by the reviewer for a particular document in the set of documents for review indicating if the particular document is responsive or nonresponsive; and

receiving a second class label assigned by the reviewer for the particular document indicating the particular document'"'"'s membership in one or more classes including;

area of law;

type of document;

or importance.

9. The method of claim 1, wherein the artificial intelligence model is a binary classifier outputting a class label.

10. The method of claim 1, further comprising calculating a size of a sample to be drawn from the documents using:

(i) an estimate of a final number of documents that will be collected;

(ii) a confidence level specifying a degree of confidence that estimates produced using a random sample should have;

(iii) a confidence interval half-width specifying how wide a confidence interval produced using the random sample should be; and

(iv) an estimate of a maximum plausible proportion of documents accounted for by a most frequent class in some class distinction.

11. The method of claim 10, further comprising assigning a random number to each of the documents to be reviewed, the random number being used to select which documents are included in the random sample.

12. The method of claim 1, further comprising allowing a user to select a blend of training and testing documents.

13. A method for reviewing documents for production during litigation, the method comprising:

scoring the documents for production using an artificial intelligence model;

selecting a subset of highest scoring documents;

selecting one or more randomly-selected documents from the documents;

forming a set of documents for review, the set of documents for review including both the subset of highest scoring documents and the randomly-selected documents;

presenting the set of documents for review by a reviewer in a manner that does not differentiate between the randomly-selected documents and the highest scoring documents; and

receiving a class label assigned by the reviewer for one or more of the documents in the set of documents for review, wherein the class label indicates if one or more of the documents are responsive or nonresponsive.

14. The method of claim 13, further comprising:

using the class label assigned by the reviewer to further train the artificial intelligence model; and

re-scoring the documents using the artificial intelligence model.

15. The method of claim 13, further comprising allowing the reviewer to assign a second class label to the one or more documents, the second class label including one or more of:

area of law;

type of document;

or importance.

16. The method of claim 13, further comprising calculating a size of a sample to be drawn from the documents for production using:

(i) an estimate of a final number of documents that will be collected for production;

(ii) a confidence level specifying a degree of confidence that estimates produced using a random sample should have;

(iii) a confidence interval half-width specifying how wide a confidence interval produced using the random sample should be; and

(iv) an estimate of a maximum plausible proportion of documents accounted for by a most frequent class in some class distinction.

17. The method of claim 16, further comprising assigning a random number to each of the documents to be reviewed, the random number being used to select which documents are included in the random sample.

18. A method for reviewing documents for production during litigation, the method comprising:

scoring the documents for production using an artificial intelligence model;

selecting a subset of highest scoring documents;

selecting one or more randomly-selected documents from the documents for production;

forming a set of documents for review, the set of documents for review including both the subset of highest scoring documents and the randomly-selected documents;

presenting the set of documents for review by a reviewer in a manner that does not differentiate between the randomly-selected documents and the highest scoring documents included in the set of documents for review; and

calculating a sample set of the documents for production using;

(i) an estimate of a total number of the documents for production;

(ii) a confidence level specifying a degree of confidence that estimates produced using a random sample should have;

(iii) a confidence interval half-width specifying how wide a confidence interval produced using the random sample should be; and

(iv) an estimate of a maximum plausible proportion of documents accounted for by a most frequent class in some class distinction.

19. The method of claim 18, further comprising assigning a random number to each of the documents to be reviewed, the random number being used to select which documents are included in the random sample.

20. The method of claim 18, further comprising receiving a class label assigned by the reviewer for one or more of the documents in the set of documents for review.