Integration and combination of random sampling and document batching

• US 9,785,634 B2
• Filed: 06/04/2011
• Issued: 10/10/2017
• Est. Priority Date: 06/04/2011
• Status: Active Grant

• Current Assignee: Open Text Holdings, Inc. (Open Text Corporation)
• Original Assignee: Recommind Incorporated (Open Text Corporation)
• Inventors: Puzicha, Jan
• Primary Examiner(s): Lin, Shew-Fen
• Application Number:
  US13/153,419
• Publication Number:
  US 20120310935A1
• Time in Patent Office: 2,320 Days
• Field of Search: 707737, 707804, 702181, 706 62
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06Q 10/103 Workflow collaboration or project management
• CPC Class Codes: G06Q 50/18 Legal services

Abstract

Methods and systems of integrated batching and random sampling of documents for enhanced functionality and quality control, such as validation, within a document review process are provided herein. According to various embodiments, a batching request may be received and may include a population size that corresponds to a total amount of documents available for sampling. The batching request may also include an acceptable margin of error. A random sample size may be calculated based on the batching request, and then a subset of documents corresponding to the random sample size may be selected from the total amount of documents available for sampling. The subset of documents may be grouped into one or more batches, and the one or more batches may be assigned to one or more review nodes.

Claims

1. A method of integrated batching and random sampling of documents for enhanced functionality within document review processes, comprising:

receiving a batching request, the batching request including;

a population size that corresponds to a number of a total amount of documents available for sampling; and

an acceptable margin of error;

computing a random sample size from the batching request;

randomly selecting a subset of documents from the total amount of documents available for sampling, a number of the randomly selected subset of documents corresponding to the random sample size, a set of excluded documents being documents in the total amount of documents available for sampling that are not included in the randomly selected subset of documents;

determining a range of relevant documents within the set of excluded documents by determining a population of relevant documents within the set of excluded documents, the determination performed by;

receiving a query regarding the total amount of documents,

applying a hypothesis test to the randomly selected subset of documents to calculate a first response to the query for the randomly selected subset of documents, and

utilizing the first response to calculate a second response to the query for the population of excluded but relevant documents within the total amount of documents;

randomly grouping the randomly selected subset of documents into a plurality of batches for assignment to a plurality of review nodes, at least one review node being a machine review node and at least one node being a human review node;

assigning each of the randomly grouped batches to a review node of the plurality of review nodes for review of the respective batch;

determining a range of excluded but relevant documents for both a batch of machine reviewed documents and a batch of human reviewed documents;

comparing the ranges together to determine a difference between machine reviewed documents and human reviewed documents; and

utilizing machine document review if the difference is less than a threshold amount.

2. The method according to claim 1, wherein the batching request further includes a selected confidence level.

3. The method according to claim 1, further comprising:

applying a statistical hypothesis test, the statistical hypothesis test including;

adding the margin of error to the population of excluded but relevant documents to create an upper range boundary and subtracting the margin of error to the population of excluded but relevant documents to create a lower range boundary; and

wherein the range of excluded but relevant documents within the total amount of documents extends between the lower range boundary and the upper range boundary, inclusive.

4. The method according to claim 1, wherein:

the random grouping of the randomly selected subset of documents into the plurality batches is based on at least one of type of document, names mentioned in the documents, and key words; and

the assigning of each of the randomly grouped batches to a review node of the plurality of review nodes is based on at least one of an expertise of a reviewer in a certain area and a level of experience of the reviewer.

5. The method according to claim 1, further comprising predicting a cost of the document review process as a function of at least one of a margin of error and a confidence level.

6. The method according to claim 1, further comprising:

determining a percentage of documents missed in the document review process;

and determining a level of risk based on the percentage.

7. The method according to claim 1, further comprising predicting a number of relevant documents in a further subset of documents representing the total amount of documents excluding the randomly selected subset of documents.

8. A system of integrated batching and random sampling of documents for enhanced functionality within document review processes, the system comprising:

a memory for storing executable instructions for batching and random sampling of documents for quality control within document review processes; and

a processor for executing the instructions stored in memory, the executable instructions comprising;

a query module that receives a batching request, the batching request including a population size that corresponds to a total amount of documents available for sampling and an acceptable margin of error;

an analysis module communicatively coupled to the query module that computes a random sample size from the batching request and randomly selects a subset of documents from the total amount of documents available for sampling, the subset of documents corresponding to the random sample size, wherein the analysis module determines a range of excluded but relevant documents within the total amount of documents by determining a population of excluded but relevant documents within the subset of documents, the determination performed by;

receiving a query regarding the total amount of documents,

applying a hypothesis test to the subset of documents to calculate a first response to the query for the subset of documents, and

utilizing the first response to calculate a second response to the query for the population of excluded but relevant documents within the subset of documents;

a batching module communicatively coupled to the analysis module that groups the subset of documents into a plurality of batches, and assigns one or more of the batches to a human review node and one or more batches to a machine review node; and

a communications module coupled to the batching module and review nodes that;

determines a range of excluded but relevant documents for both a subset of machine reviewed documents and a subset of human reviewed documents,

compares the ranges together to determine a difference between machine reviewed documents and human reviewed documents, the difference being expressed as a percentage, and

transmits the batches to machine document review nodes if the difference is less than a threshold amount.

9. The system according to claim 8, further comprising a communications module communicatively coupled to the batching module and communicatively coupleable to one or more review nodes, the communications module transmits the one or more batches to the one or more review nodes.

10. The system according to claim 8, further comprising a statistical evaluation module that applies a statistical hypothesis test to the randomly selected subset of documents to calculate a first statistical response to the statistical query for the randomly selected subset of documents;

and utilizes the first statistical response to calculate a second statistical response to the statistical query for the total amount of documents available for sampling, the statistical evaluation module being communicatively coupled to the query module and the analysis module.

11. A method for reviewing documents, comprising:

computing a random sample size from a batching request including a number of documents and a margin of error;

randomly selecting a subset of documents from the total amount of documents, the randomly selected subset corresponding to the random sample size;

determining a range of excluded but relevant documents within the total amount of documents by;

receiving a query regarding the total amount of documents,

applying a hypothesis test to the randomly selected subset of documents to calculate a first response to the query for the randomly selected subset of documents, and

utilizing the first response to calculate a second response to the query for the population of excluded but relevant documents within the subset of documents;

randomly grouping the randomly selected subset of documents into a plurality of batches based on at least one of type of document, names mentioned in the documents, and key words;

assigning each of the randomly grouped batches to a plurality of review nodes based on at least one of an expertise of a reviewer of a respective node in a certain area and a level of experience of the reviewer, at least one review node being a machine review node and at least one node being a human review node;

determining a range of excluded but relevant documents for both a batch of machine reviewed documents and a batch of human reviewed documents;

comparing the ranges together to determine a difference between machine reviewed documents and human reviewed documents; and

utilizing machine document review if the difference is less than a threshold amount.

12. The method according to claim 11, further comprising predicting a cost of a document review as a function of at least one of a margin of error and a confidence level.

13. The method according to claim 11, further comprising:

determining a percentage of documents missed in a document review; and

determining a level of risk based on the percentage.

14. The method according to claim 11, further comprising predicting a number of relevant documents in a further subset of documents representing the total amount of documents excluding the randomly selected subset of documents.