Statistical Determination of Operator Error

• US 20110060618A1
• Filed: 11/11/2010
• Published: 03/10/2011
• Est. Priority Date: 03/18/2005
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: BeyondCore, Inc. (Salesforce.com, Inc.)
• Inventors: Sengupta, Arijit
• Granted Patent: US 8,019,734 B2
• Time in Patent Office: Days
• US Class Current: 705/7.13
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 99/00 Subject matter not provided for in other groups of this subclass

Abstract

Various embodiments of the invention provide systems and methods for estimating a frequency of errors for data processing operators, but without human inspection of documents typed by the operators and without comparing documents typed by the operators against known transcriptions.

Claims

1. A computer-implemented method for estimating a frequency of errors for each of a plurality of data processing operators, the data processing operators performing a data processing operation on a plurality of documents, the method comprising a computer system executing software to effect the steps of:

assigning documents to different groups of operators, each group containing at least two operators, each operator belonging to at least two groups;

making the documents that are assigned to a group, available to the operators in the group in order for each operator to type data in the documents assigned to the group, the different operators thus generating different versions for each document;

collecting the versions of the documents typed by the different operators;

statistically analyzing discrepancies between the different versions without human intervention and without comparing to a known transcription of the document; and

based on the statistical analysis of documents assigned to groups and based on the assignment of operators to groups, estimating an average frequency of errors, the frequency of errors averaged over the plurality of documents.

2. The computer-implemented method of claim 1 wherein the step of assigning documents to different groups of operators comprises randomly assigning documents to different groups of operators.

3. The computer-implemented method of claim 1 wherein each group of operators consists of a pair of operators.

4. The computer-implemented method of claim 1 wherein the step of estimating an average frequency of errors comprises estimating an average frequency of errors for each operator.

5. The computer-implemented method of claim 4 wherein the step of estimating an average frequency of errors for each operator comprises estimating an average frequency of errors for each operator without knowing a frequency of errors for any specific document typed by that operator.

6. The computer-implemented method of claim 4 wherein the step of estimating an average frequency of errors for each operator comprises:

estimating an average frequency of errors for each group of operators, based on the statistical analysis of documents assigned to that group of operators; and

based on the assignment of operators to groups and the estimated average frequency of errors for each group, estimating the average frequency of errors for each operator.

7. The computer-implemented method of claim 4 wherein:

the documents comprise fields; and

the step of estimating an average frequency of errors for each operator comprises estimating an average frequency of errors for at least one field for each operator.

8. The computer-implemented method of claim 1 wherein the step of estimating an average frequency of errors comprises estimating an average frequency of errors for each field.

9. A computer program product for a quality management system, the computer program product stored on a tangible computer-readable medium and including instructions that, when loaded into memory, cause a processor to carry out the steps of:

assigning documents to different groups of operators, each group containing at least two operators, each operator belonging to at least two groups;

making the documents that are assigned to a group, available to the operators in the group in order for each operator to type data in the documents assigned to the group, the different operators thus generating different versions for each document;

collecting the versions of the documents typed by the different operators;

statistically analyzing discrepancies between the different versions without human intervention and without comparing to a known transcription of the document; and

based on the statistical analysis of documents assigned to groups and based on the assignment of operators to groups, estimating an average frequency of errors, the frequency of errors averaged over the plurality of documents.

10. The computer program product of claim 9 wherein the step of assigning documents to different groups of operators comprises randomly assigning documents to different groups of operators.

11. The computer program product of claim 9 wherein each group of operators consists of a pair of operators.

12. The computer program product of claim 9 wherein the step of estimating an average frequency of errors comprises estimating an average frequency of errors for each operator.

13. The computer program product of claim 12 wherein the step of estimating an average frequency of errors for each operator comprises estimating an average frequency of errors for each operator without knowing a frequency of errors for any specific document typed by that operator.

14. The computer program product of claim 12 wherein the step of estimating an average frequency of errors for each operator comprises:

estimating an average frequency of errors for each group of operators, based on the statistical analysis of documents assigned to that group of operators; and

based on the assignment of operators to groups and the estimated average frequency of errors for each group, estimating the average frequency of errors for each operator.

15. The computer program product of claim 12 wherein:

the documents comprise fields; and

the step of estimating an average frequency of errors for each operator comprises estimating an average frequency of errors for at least one field for each operator.

16. The computer program product of claim 9 wherein the step of estimating an average frequency of errors comprises estimating an average frequency of errors for each field.

17. A quality management system comprising:

means for assigning documents to different groups of operators, each group containing at least two operators, each operator belonging to at least two groups;

means for making the documents that are assigned to a group, available to the operators in the group in order for each operator to type data in the documents assigned to the group, the different operators thus generating different versions for each document;

means for collecting the versions of the documents typed by the different operators;

means for statistically analyzing discrepancies between the different versions without human intervention and without comparing to a known transcription of the document; and

means for, based on the statistical analysis of documents assigned to groups and based on the assignment of operators to groups, estimating an average frequency of errors, the frequency of errors averaged over the plurality of documents.