Systems and methods for automated document review and quality control

• US 10,504,037 B1
• Filed: 03/31/2016
• Issued: 12/10/2019
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Inventors: Anant, Bashyam, Beer, Mat, Livian, Houshang, Iyer, Indira
• Primary Examiner(s): Waldron, Scott A.
• Assistant Examiner(s): Tamulonis, Fen Christopher
• Application Number:
  US15/087,946
• Time in Patent Office: 1,349 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services
• CPC Class Codes: G06Q 50/18 Legal services

Abstract

A computer-implemented method for automated document review and quality control may include (1) dividing a set of documents to be reviewed for relevancy into sets of control documents, training documents, quality-control documents, and review documents, (2) calculating, based on a set of reviews performed by a group of reviewers on the set of quality-control documents, an effective speed score for each reviewer in the group of reviewers, (3) assigning, based on the effective speed score, the set of control documents and the set of training documents to the group of reviewers, (4) training a predictive model using a set of training reviews performed by the group of reviewers on both the set of training documents and the set of control documents, and (5) using the predictive model to evaluate the set of review documents. Various other methods, systems, and computer-readable media are also disclosed.

Claims

1. A computer-implemented method for automated document review and quality control, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:

dividing a set of documents to be reviewed for relevancy into;

a set of control documents;

a set of training documents;

a set of quality-control documents; and

a set of review documents;

calculating, based on a set of reviews performed by a group of reviewers on the set of quality-control documents, an effective speed score for each reviewer in the group of reviewers, wherein the effective speed score indicates a reviewer'"'"'s ability to accurately complete a review in a given time, wherein accuracy is determined by agreement between the reviewer and other reviewers;

assigning, based on the effective speed score, the set of control documents and the set of training documents to the group of reviewers;

training a predictive model using a set of training reviews performed by the group of reviewers on both the set of training documents and the set of control documents; and

using the predictive model to evaluate the set of review documents.

2. The method of claim 1, wherein dividing the set of documents comprises selecting representative samples of the set of documents for the set of control documents and the set of training documents.

3. The method of claim 1, wherein the set of quality-control documents comprises documents selected from both the set of control documents and the set of training documents for additional review.

4. The method of claim 1, wherein the set of reviews performed by the group of reviewers comprises reviews received from at least three reviewers from the group of reviewers for each document in the set of quality-control documents.

5. The method of claim 4, wherein the effective speed score comprises:

an accuracy score based on an agreement between the three reviewers for each document reviewed by the reviewer; and

a speed score based on an average time taken by the reviewer to review the documents.

6. The method of claim 1, wherein assigning the set of control documents and the set of training documents to the group of reviewers comprises dividing a quantity of documents among the group of reviewers in proportion to the effective speed score for each reviewer.

7. The method of claim 1, wherein training the predictive model comprises at least one of:

using machine learning to train the predictive model with the set of training reviews;

determining that the predictive model passes a benchmark for accuracy; and

determining that the predictive model fails the benchmark for accuracy.

8. The method of claim 7, further comprising, in response to determining that the predictive model fails the benchmark:

adding at least one document from the set of review documents to the set of training documents;

adding a review of the document to the set of training reviews; and

retraining the predictive model using the set of training reviews.

9. The method of claim 1, wherein using the predictive model to evaluate the set of review documents comprises using the predictive model to calculate a prediction rank for each document in the set of review documents.

10. The method of claim 9, wherein the prediction rank comprises a likelihood score that the document is classified as relevant.

11. The method of claim 9, further comprising:

identifying a set of relevant documents based on a predetermined threshold for the prediction rank; and

assigning the set of relevant documents to the group of reviewers based on the effective speed score.

12. The method of claim 11, wherein assigning the set of relevant documents to the group of reviewers comprises assigning a document with a lower prediction rank to a reviewer with a higher effective speed score.

13. A system for automated document review and quality control, the system comprising:

a division module, stored in memory, that divides a set of documents to be reviewed for relevancy into;

a set of control documents;

a set of training documents;

a set of quality-control documents; and

a set of review documents;

a calculation module, stored in memory, that calculates;

based on a set of reviews performed by a group of reviewers on the set of quality-control documents, an effective speed score for each reviewer in the group of reviewers, wherein the effective speed score indicates a reviewer'"'"'s ability to accurately complete a review in a given time, wherein accuracy is determined by agreement between the reviewer and other reviewers;

an assignment module, stored in memory, that assigns, based on the effective speed score, the set of control documents and the set of training documents to the group of reviewers;

a training module, stored in memory, that trains a predictive model using a set of training reviews performed by the group of reviewers on both the set of training documents and the set of control documents;

an evaluation module;

stored in memory, that uses the predictive model to evaluate the set of review documents; and

at least one processor that executes the division module, the calculation module, the assignment module, the training module, and the evaluation module.

14. The system of claim 13, wherein the division module divides the set of documents by selecting representative samples of the set of documents for the set of control documents and the set of training documents.

15. The system of claim 13, wherein the set of quality-control documents comprises documents selected from both the set of control documents and the set of training documents for additional review.

16. The system of claim 13, wherein the set of reviews performed by the group of reviewers comprises reviews received from at least three reviewers from the group of reviewers for each document in the set of quality-control documents.

17. The system of claim 16, wherein the effective speed score comprises:

an accuracy score based on an agreement between the three reviewers for each document reviewed by the reviewer; and

a speed score based on an average time taken by the reviewer to review the documents.

18. The system of claim 13, wherein the assignment module assigns the set of control documents and the set of training documents to the group of reviewers by dividing a quantity of documents among the group of reviewers in proportion to the effective speed score for each reviewer.

19. The system of claim 13, wherein the training module trains the predictive model by at least one of:

using machine learning to train the predictive model with the set of training reviews;

determining that the predictive model passes a benchmark for accuracy; and

determining that the predictive model fails the benchmark for accuracy.

20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

divide a set of documents to be reviewed for relevancy into;

a set of control documents;

a set of training documents;

a set of quality-control documents; and

a set of review documents;

calculate, based on a set of reviews performed by a group of reviewers on the set of quality-control documents, an effective speed score for each reviewer in the group of reviewers, wherein the effective speed score indicates a reviewer'"'"'s ability to accurately complete a review in a given time, wherein accuracy is determined by agreement between the reviewer and other reviewers;

assign, based on the effective speed score, the set of control documents and the set of training documents to the group of reviewers;

train a predictive model using a set of training reviews performed by the group of reviewers on both the set of training documents and the set of control documents; and

use the predictive model to evaluate the set of review documents.