Collaborative and quality control scoring system and method

• US 6,749,435 B2
• Filed: 04/29/2003
• Issued: 06/15/2004
• Est. Priority Date: 02/05/1993
• Status: Expired due to Fees

• Current Assignee: NCS Pearson Incorporated (Pearson plc)
• Original Assignee: NCS Pearson Incorporated (Pearson plc)
• Inventors: Clark, Jay V., Moyer, Judith, Bramlett, William L. Jr.
• Primary Examiner(s): Cheng, Joe H.
• Application Number:
  US10/425,775
• Publication Number:
  US 20030198933A1
• Time in Patent Office: 413 Days
• Field of Search: 434/118, 434/156, 434/157, 434/169, 434/327, 434/185, 434/307.R, 434/308, 434/322, 434/323, 434/336, 434/350, 434/362, 434/365, 345/173, 345/180, 345/326, 345/961, 345/978, 382/10, 382/48, 382/59, 382/164, 382/191, 382/318, 700/29, 700/86, 700/96, 700/99, 700/108, 706/16, 706/19, 706/20, 706/45, 706/927, 705/2, 705/10, 273/430, 463/9, 283/94, 283/115
• US Class Current: 434/322
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G09B 7/066 with answer indicating cards, blocks
• CPC Class Codes: Y10S 706/927 Education or instruction

Abstract

A method and system for managing evaluators of data items to maintain consistent and efficient scoring of data items. Collaborative scoring allows two evaluators to score the same data item and resolves potential discrepancies in the scores provided by the evaluators. Quality items monitor a evaluator'"'"'s scoring performance against a known standard. A quality-control data item, having a known score, is transmitted to a evaluator. After receiving a score from the evaluator, the known score is compared with the evaluator'"'"'s score and a quality profile is generated for the evaluator.

Claims

1. A method for verifying evaluations of data items between two or more evaluators, the method comprising the steps of:

a) electronically receiving over a distributed computer network a data item created in response to a query;

b) electronically transmitting over the distributed computer network the data item to first and second evaluator terminals to be evaluated by first and second evaluators;

c) electronically receiving over a distributed computer network first and second evaluation results for the data items, as determined by the first and second evaluators, from the first and second evaluator terminals; and

d) electronically comparing the first and second evaluation results and selectively recording a value related to the first and second evaluation results if the first and second evaluation results did not contain a discrepancy based on predefined criteria.

2. The method of claim 1 further comprising the step of electronically displaying the query to the first and second evaluators at the first and second evaluator terminals.

3. The method of claim 1 wherein the step of electronically comparing the first and second evaluation results further comprises the steps of:

a) providing an electronic communication link over the distributed computer network between the first and second evaluator terminals and, if the first and second evaluation results contain a discrepancy based on the predefined criteria, signaling the first and second evaluators to interactively determine and enter a joint evaluation result using the electronic communication link; and

b) recording the joint evaluation result.

4. The method of claim 3 further comprising the step of tracking evaluation results of the first and second evaluators and preventing the first and second evaluators from evaluating a same data item if the first and second evaluators each have average evaluation results within predefined evaluation criteria.

5. The method of claim 4 wherein the step of electronically comparing the first and second evaluation results further comprises the step of transmitting over the distributed computer network the data item to a third evaluator terminal to be evaluated by a third evaluator, if the first and second evaluators do not enter the joint evaluation result within a predetermined time period.

6. The method of claim 3 wherein the step of electronically comparing the first and second evaluation results further comprises the step of preventing the first and second evaluator terminals from receiving another data item to be evaluated until the first and second evaluators have entered the joint evaluation result.

7. The method of claim 1 wherein the step of electronically comparing the first and second evaluation results further comprises the steps of:

a) transmitting over the distributed computer network the data item to a third evaluator terminal to be evaluated by a third evaluator, if the first and second evaluation results contain the discrepancy based on the predefined criteria; and

b) recording a third evaluation result, as determined by the third evaluator, received from the third evaluator terminal over the distributed computer network.

8. The method of claim 7 wherein the step of transmitting the data item to a third evaluator terminal over the distributed computer network further comprises the step of providing the third evaluator terminal with the first and second evaluation results.

9. The method of claim 7 wherein the step of transmitting the data item to a third evaluator terminal over the distributed computer network further comprises the step of preventing the third evaluator from viewing the first and second evaluation results on the third evaluator terminal before the third evaluator records the third evaluation result.

10. The method of claim 1, wherein the step of electronically comparing the first and second evaluation results further comprises the step of electronically linking the first and second evaluator terminals over the distributed computer network when the first and second evaluators are located in different geographical locations.

11. The method of claim 1 wherein the step of electronically comparing the first and second evaluation results further comprises the step of selectively recording, as the value, a greater of the first and second evaluation results, a lower of the first and second evaluation results, and an average of the first and second evaluation results.

12. A system for verifying evaluation of data items between two or more evaluators, the system comprising:

a) a receiving subsystem configured to electronically receive a data item created in response to a query;

b) a transmit subsystem configured to electronically transmit the data item to first and second evaluator terminals to be evaluation by first and second evaluators;

c) an evaluation result subsystem configured to electronically receive first and second evaluation results for the data items, as determined by the first and second evaluators, from the first and second evaluator terminals; and

d) a comparison subsystem configured to electronically compare the first and second evaluation results and to selectively record a value related to the first and second evaluation results if the first and second evaluation results did not contain a discrepancy based on predefined criteria.

13. The system of claim 12 wherein the receiving subsystem is further configured to electronically display the query to the first and second evaluators at the first and second evaluator terminals.

14. The system of claim 12 wherein the comparison subsystem is further configured to:

a) provide an electronic communication link between the first and second evaluator terminals and, if the first and second evaluation results contain the discrepancy based on the predefined criteria;

b) signal the first and second evaluators to interactively determine and enter a joint evaluation result using the electronic communication link; and

c) record the joint evaluation result.

15. The system of claim 14 wherein the comparison subsystem is further configured to track average evaluation results of the first and second evaluators and to prevent the first and second evaluators from evaluating a same data item if the first and second evaluators each have average evaluation results within predefined evaluation criteria.

16. The system of claim 14 wherein the comparison subsystem is further configured to prevent the first and second evaluator terminals from receiving another data item to be evaluated until the first and second evaluators have entered the joint evaluation result.

17. The system of claim 14 wherein the comparison subsystem is further configured to transmit the data item to a third evaluator terminal to be evaluated by a third evaluator, if the first and second evaluators do not enter the joint evaluation result within a predetermined time period.

18. The system of claim 12 wherein the comparison subsystem is further configured to:

a) transmit the data item to a third evaluator terminal to be evaluated by a third evaluator, if the first and second evaluation results contain the discrepancy based on the predefined criteria; and

b) record a third evaluation result, as determined by the third evaluator, received from the third evaluator terminal.

19. The system of claim 18 wherein the system is further configured to provide the third evaluator terminal with the first and second evaluation results.

20. The system of claim 18 wherein the system is configured to prevent the third evaluator from viewing the first and second evaluation results on the third evaluator terminal before the third evaluator records the third evaluation result.

21. The system of claim 12, wherein the comparison subsystem is further configured to electronically link the first and second evaluator terminals when the first and second evaluators are located in different geographical locations.

22. The system of claim 12 wherein the comparison subsystem is further configured to selectively record, as the joint evaluation result, a greater of the first and second evaluation results, a lower of the first and second evaluation results, and an average of the first and second evaluation results.

23. A computer-readable program storage medium tangibly embodying a data package and associated verification instructions executable by a computing system for verifying evaluation of data items between two or more evaluators by performing the steps of:

a) electronically receiving a data item created in response to a query;

b) electronically transmitting the data item to first and second evaluator terminals to be evaluated by first and second evaluators;

c) electronically receiving first and second evaluation results for the data item, as determined by the first and second evaluators, from the first and second evaluator terminals; and

d) electronically comparing the first and second evaluation results and for selectively recording a value related to the first and second evaluation results if the first and second evaluation results did not contain a discrepancy based on predefined criteria.

24. The computer-readable program storage medium of claim 23 further comprising the step of electronically displaying the query to the first and second evaluators at the first and second evaluator terminals.

25. The computer-readable program storage medium of claim 23 wherein the step of electronically comparing the first and second evaluation results further comprises the steps of:

a) providing an electronic communication link between the first and second evaluator terminals and, if the first and second evaluation results contain the discrepancy based on the predefined criteria, signaling the first and second evaluators to interactively determine and enter a joint evaluation result using the electronic communication link; and

b) recording the joint evaluation result.

26. The computer-readable program storage medium of claim 25 further comprising the step of tracking average evaluation results of the first and second evaluators and preventing the first and second evaluators from evaluating a same data item if the first and second evaluators each have average evaluation results within predefined evaluation criteria.

27. The computer-readable program storage medium of claim 25 wherein the step of electronically comparing the first and second evaluation results further comprises the step of preventing the first and second evaluator terminals from receiving another data item to be evaluated until the first and second evaluators have entered the joint evaluation result.

28. The computer-readable program storage medium of claim 25 wherein the step of electronically comparing the first and second evaluation results further comprises the step of transmitting the data item to a third evaluator terminal to be evaluated by a third evaluator, if the first and second evaluators do not enter the joint evaluation result within a predetermined time period.

29. The computer-readable program storage medium of claim 23 wherein the step of electronically comparing the first and second evaluation results further comprises the steps of:

a) transmitting the data item to a third evaluator terminal to be evaluated by a third evaluator, if the first and second evaluation results contain the discrepancy based on the predefined criteria; and

b) recording a third evaluation result, as determined by the third evaluator, received from the third evaluator terminal.

30. The computer-readable program storage medium of claim 29 wherein the step of transmitting the data item to a third evaluator terminal further comprises the step of providing the third evaluator terminal with the first and second evaluation results.

31. The computer-readable program storage medium of claim 29 wherein the step of transmitting the data item to a third evaluator terminal further comprises the step of preventing the third evaluator from viewing the first and second evaluation results on the third evaluator terminal before the third evaluator records the third evaluation result.

32. The computer-readable program storage medium of claim 23, wherein the step of electronically comparing the first and second evaluation results further comprises the step of electronically linking the first and second evaluator terminals when the first and second evaluators are located in different geographical locations.

33. The computer-readable program storage medium of claim 23 wherein the step of electronically comparing the first and second evaluation results comprises the step of selectively recording, as the value, a greater of the first and second evaluation results, a lower of the first and second evaluation results, and an average of the first and second evaluation results.