QUALITY CONTROL CALCULATOR FOR DOCUMENT REVIEW
First Claim
1. A computerized method for automatically managing quality of human document review in a review process, the method comprising:
- receiving, by an extraction hardware module of a computing device, tagging decisions for a plurality of documents made by a first reviewer during a first time period;
determining, by a sampling hardware module of the computing device, a subset of the plurality documents based on a first confidence level and first confidence interval;
receiving, by the sampling hardware module of the computing device, tagging decisions made by a second reviewer related to the subset of the plurality of documents;
determining, by a quality-control review hardware module of the computing device, values of a plurality of quality-control metrics based on the tagging decisions of the first and second reviewers with respect to the subset of the plurality of documents, wherein the values of the plurality of quality-control metrics reflect a level of identity between the first and second reviewers in relation to a plurality of tagging criteria;
displaying, by a graphical user interface (GUI) hardware module of the computing device, a graphical user interface on a display device coupled to the computing device, the graphical user interface comprisinga first section having a user input field configured to enable selection of one or more days of the first time period that defines a date range of tagging decisions made by the first reviewer to include in the determining values step,a second section having a plurality of user input fields configured to enable entry of data relating to the tagging decisions made by the second reviewer, anda third section having a visual comparison of the plurality of quality-control metrics between the first and second reviewers in relation to the plurality of tagging criteria;
calculating, by a quality-control calculator hardware module of the computing device, a risk-accuracy value as a weighted combination of a plurality of factors including (1) an accuracy factor determined based on the values of the plurality of quality-control metrics;
(2) a review rate factor indicating the rate of review of the first reviewer during the first time period; and
(3) one or more user-selectable factors reflecting the complexity or difficulty associated with reviewing the plurality of documents; and
recommending, by a recommendation hardware module of the computing device, a second confidence level and a second confidence interval for sampling a second plurality of documents reviewed during a second time period, wherein the second confidence level and the second confidence interval are determined based on the risk-accuracy value.
1 Assignment
0 Petitions
Accused Products
Abstract
Described are methods and apparatuses, including computer program products, for automatically managing quality of human document review in a review process. The method includes receiving tagging decisions for multiple documents made by a first reviewer during a first time period and sampling a subset of these documents based on a first confidence level and first confidence interval. The method further includes receiving tagging decisions made by a second reviewer related to the subset of the documents, from which values of multiple quality-control metrics are determined. The method further includes calculating a risk-accuracy value based in part on the values of the quality-control metrics and recommending a second confidence level and a second confidence interval for sampling a second set of documents reviewed by the first reviewer during a second time period.
52 Citations
30 Claims
-
1. A computerized method for automatically managing quality of human document review in a review process, the method comprising:
-
receiving, by an extraction hardware module of a computing device, tagging decisions for a plurality of documents made by a first reviewer during a first time period; determining, by a sampling hardware module of the computing device, a subset of the plurality documents based on a first confidence level and first confidence interval; receiving, by the sampling hardware module of the computing device, tagging decisions made by a second reviewer related to the subset of the plurality of documents; determining, by a quality-control review hardware module of the computing device, values of a plurality of quality-control metrics based on the tagging decisions of the first and second reviewers with respect to the subset of the plurality of documents, wherein the values of the plurality of quality-control metrics reflect a level of identity between the first and second reviewers in relation to a plurality of tagging criteria; displaying, by a graphical user interface (GUI) hardware module of the computing device, a graphical user interface on a display device coupled to the computing device, the graphical user interface comprising a first section having a user input field configured to enable selection of one or more days of the first time period that defines a date range of tagging decisions made by the first reviewer to include in the determining values step, a second section having a plurality of user input fields configured to enable entry of data relating to the tagging decisions made by the second reviewer, and a third section having a visual comparison of the plurality of quality-control metrics between the first and second reviewers in relation to the plurality of tagging criteria; calculating, by a quality-control calculator hardware module of the computing device, a risk-accuracy value as a weighted combination of a plurality of factors including (1) an accuracy factor determined based on the values of the plurality of quality-control metrics;
(2) a review rate factor indicating the rate of review of the first reviewer during the first time period; and
(3) one or more user-selectable factors reflecting the complexity or difficulty associated with reviewing the plurality of documents; andrecommending, by a recommendation hardware module of the computing device, a second confidence level and a second confidence interval for sampling a second plurality of documents reviewed during a second time period, wherein the second confidence level and the second confidence interval are determined based on the risk-accuracy value. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20)
-
-
21. A computer-implemented system for automatically managing quality of human document review in a review process, the computer-implemented system comprising a plurality of hardware modules each coupled to a processor and a memory of a computing device, the hardware modules including an extraction module, a sampling module, a graphical user interface (GUI) module, a quality-control review module, a quality-control calculator module, and a recommendation module:
-
the extraction module comprising registers and instructions for extracting tagging decisions for a plurality of documents made by a first reviewer during a first time period; the sampling module comprising registers and instructions for (i) determining a subset of the plurality documents based on a first confidence level and first confidence interval and (ii) receiving tagging decisions made by a second reviewer related to the subset of the plurality of documents; the quality-control review module comprising registers and instructions for determining values of a plurality of quality-control metrics based on the tagging decisions of the first and second reviewers with respect to the subset of the plurality of documents, wherein the values of the plurality of quality-control metrics reflect levels of identity between the first and second reviewers in relation to a plurality of tagging criteria; the graphical user interface (GUI) module comprising registers and instructions for displaying a graphical user interface on a display device coupled to the computing device, the graphical user interface comprising a first section having a user input field configured to enable selection of one or more days of the first time period that defines a date range of tagging decisions made by the first reviewer to include in the determining values step, a second section having a plurality of user input fields configured to enable entry of data relating to the tagging decisions made by the second reviewer, and a third section having a visual comparison of the plurality of quality-control metrics between the first and second reviewers in relation to the plurality of tagging criteria; the quality-control calculator comprising registers and instructions for calculating a risk-accuracy value as a weighted combination of a plurality of factors including (1) an accuracy factor determined based on the values of the plurality of quality-control metrics;
(2) a review rate factor indicating the rate of review of the first reviewer during the first time period; and
(3) one or more user-selectable factors reflecting the complexity associated with reviewing the plurality of documents; anda recommendation module comprising registers and instructions for recommending a second confidence level and a second confidence interval for sampling a second plurality of documents reviewed by the first reviewer during a second time period, wherein the second confidence level and the second confidence interval are determined based on the risk-accuracy value. - View Dependent Claims (22, 23, 24, 25, 26, 27, 28, 29)
-
-
30. A computer program product, tangibly embodied in a non-transitory computer readable medium, for automatically managing quality of human document review in a review process, the computer program product including instructions being configured to cause a plurality of hardware modules each coupled to a processor and a memory of a computing device, the hardware modules including an extraction module, a sampling module, a graphical user interface (GUI) module, a quality-control review module, a quality-control calculator module, and a recommendation module to:
-
receive, by the extraction module, tagging decisions for a plurality of documents made by a first reviewer during a first time period; determine, by the sampling module, a subset of the plurality documents based on a first confidence level and first confidence interval; receive, by the sampling module, tagging decisions made by a second reviewer related to the subset of the plurality of documents; determine, by the quality-control review module, values of a plurality of quality-control metrics based on the tagging decisions of the first and second reviewers with respect to the subset of the plurality of documents, wherein the values of the plurality of quality-control metrics reflect levels of identity between the first and second reviewers in relation to a plurality of tagging criteria; display, by the graphical user interface (GUI) module, a graphical user interface on a display device coupled to the computing device, the graphical user interface comprising a first section having a user input field configured to enable selection of one or more days of the first time period that defines a date range of tagging decisions made by the first reviewer to include in the determining values step, a second section having a plurality of user input fields configured to enable entry of data relating to the tagging decisions made by the second reviewer, and a third section having a visual comparison of the plurality of quality-control metrics between the first and second reviewers in relation to the plurality of tagging criteria; calculate, by the quality control calculator module, a risk-accuracy value as a weighted combination of a plurality of factors including (1) an accuracy factor determined based on the values of the plurality of quality-control metrics;
(2) a review rate factor indicating the rate of review of the first reviewer during the first time period; and
(3) one or more user-selectable factors reflecting the complexity associated with reviewing the plurality of documents; andrecommend, by the recommendation module, a second confidence level and a second confidence interval for sampling a second plurality of documents reviewed by the first reviewer during a second time period, wherein the second confidence level and the second confidence interval are determined based on the risk-accuracy value.
-
Specification