System for providing feedback to evaluators

• US 6,976,847 B2
• Filed: 10/20/2003
• Issued: 12/20/2005
• Est. Priority Date: 02/05/1993
• Status: Expired due to Fees

• Current Assignee: NCS Pearson Incorporated (Pearson plc)
• Original Assignee: NCS Pearson Incorporated (Pearson plc)
• Inventors: Clark, Jay V., Moyer, Judith, Bramlett, William L. Jr.
• Primary Examiner(s): Cheng, Joe H.
• Application Number:
  US10/690,488
• Publication Number:
  US 20040063087A1
• Time in Patent Office: 792 Days
• Field of Search: 434/118, 434/156, 434/157, 434/169, 434/185, 434/307.R, 434/308, 434/322, 434/323, 434/350, 434/362, 434/365, 463/9, 382/164, 382/191, 382/318, 273/273, 273429-432, 700/29, 700/86, 700/96, 700/99, 700/108, 705/2, 705/10, 706/16, 706/19, 706/20, 706/45, 706/927
• US Class Current: 434/322
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G09B 7/066 with answer indicating cards, blocks
• CPC Class Codes: Y10S 706/927 Education or instruction

Abstract

A method for increasing the speed at which data items are processed. The method receives electronic representation of data items and organizes the data items into separate groupings. The method next displays a particular grouping of data items to a resolver so that the resolver can evaluate the data items. In displaying the data items, the method uses a matrix of cells and displays one data item in each of the cells. The resolver can thus simultaneously view multiple data items, which facilitates efficient evaluations of the data items.

Claims

1. A system for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in evaluating data items to predetermined performance criteria, the system comprising:

a) a receiving subsystem configured to electronically receive information over a distributed computer network related to a first performance level of a first evaluator'"'"'s evaluation of data items, and a predetermined performance level;

b) a comparison subsystem configured to electronically compare the first performance level and the predetermined performance level; and

c) a discrepancy subsystem configured to electronically report a discrepancy based on predefined criteria between the first performance level and the predetermined performance level.

2. The system of claim 1 wherein the discrepancy subsystem is further configured to electronically report the discrepancy to the first evaluator over a distributed computer network.

3. The system of claim 1 wherein the discrepancy subsystem is further configured to electronically report the discrepancy to a system manager over a distributed computer network.

4. A system for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in evaluating data items to predetermined performance criteria, the system comprising:

a) a receiving subsystem configured to electronically receive information over a distributed computer network related to a first validity level of a first evaluator'"'"'s evaluation of data items, and a predetermined validity level;

b) a comparison subsystem configured to electronically compare the first validity level and the predetermined validity level; and

c) a discrepancy subsystem configured to electronically report a discrepancy based on predefined criteria between the first validity level and the predetermined validity level.

5. The system of claim 4 wherein the receiving subsystem is further configured to receive an evaluation result for a quality item from the first evaluator over a distributed computer network and a known evaluation result for the quality item.

6. The system of claim 4 wherein the discrepancy subsystem is further configured to electronically report the discrepancy to the first evaluator over a distributed computer network.

7. The system of claim 4 wherein the discrepancy subsystem is further configured to electronically report the discrepancy to a system manager over a distributed computer network.

8. A system for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in evaluating data items to predetermined performance criteria, the system comprising:

a) a receiving subsystem configured to electronically receive information over a distributed computer network related to a first reliability level of a first evaluator'"'"'s evaluation of data items, and a predetermined reliability level;

b) a comparison subsystem configured to electronically compare the first reliability level and the predetermined reliability level; and

c) a discrepancy subsystem configured to electronically report over a distributed computer network a discrepancy based on predefined criteria between the first reliability level and the predetermined reliability level.

9. The system of claim 8 wherein the receiving subsystem is further configured to receive a first evaluation result corresponding to the first evaluator'"'"'s evaluating of a data item during a first time period and to receive a second evaluation result corresponding to the first evaluator'"'"'s evaluation of the data item during a second time period.

10. The system of claim 8 wherein the discrepancy subsystem is further configured to electronically report the discrepancy to the first evaluator over a distributed computer network.

11. The system of claim 8 wherein the discrepancy subsystem is further configured to electronically report the discrepancy to a system manager over a distributed computer network.

12. A system for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in evaluating data items to predetermined performance criteria, the system comprising:

a) a receiving subsystem configured to electronically receive information over a distributed computer network related to a first speed level of a first evaluator'"'"'s evaluating of data items, and a predetermined speed level;

b) a comparison subsystem configured to electronically compare the first speed level and the predetermined speed level; and

c) a discrepancy subsystem configured to electronically report a discrepancy over a distributed computer network based on predefined criteria between the first speed level and the predetermined speed level.

13. The system of claim 12 wherein the receiving subsystem is further configured to receive at least one of the following:

a first evaluating rate corresponding to the first evaluator'"'"'s evaluating of the data items, an average evaluating rate corresponding to a selected group of evaluators'"'"' evaluating of the data items, and a second evaluating rate corresponding to a second evaluator'"'"'s evaluating of the data items.

14. The system of claim 13 wherein the discrepancy subsystem is further configured to electronically report the discrepancy to the first evaluator over a distributed computer network.

15. The system of claim 13 wherein the discrepancy subsystem is further configured to electronically report the discrepancy to a system manager over a distributed computer network.

16. A system for increasing evaluator efficiency by monitoring evaluator performance in evaluating data items and recommending break times to the evaluators, the system comprising:

a) receiving subsystem configured to electronically receive first performance levels over a distributed computer network corresponding to an evaluator'"'"'s evaluating of data items at predetermined past intervals in time, and electronically receiving over a distributed computer network a second predetermined performance level;

b) a comparison subsystem configured to electronically compare the second predetermined performance level and the first performance level; and

c) a discrepancy subsystem configured to electronically detect a discrepancy based on predefined criteria between the second predetermined performance level and the first performance level and signaling the evaluator to halt data item evaluating for a break time.

17. The system of claim 16 wherein the receiving subsystem is further configured to electronically receive first validity levels from the evaluator over a distributed computer network and to electronically receive a second predetermined validity level over a distributed computer network.

18. The system of claim 16 wherein the receiving subsystem is further configured to electronically receive first reliability levels from the evaluator over a distributed computer network and to electronically receive a second predetermined reliability level over a distributed computer network.

19. The system of claim 16 wherein the subsystem is further configured to electronically receive first speed levels from the evaluator over a distributed computer network and to electronically receive a second predetermined speed level over a distributed computer network.

20. The system of claim 16 wherein the discrepancy subsystem is further configured to electronically detect a discrepancy based on predefined criteria between the second predetermined performance level and the first performance levels and providing the evaluator with diversionary activities over a distributed computer network.

21. The system of claim 16 wherein the discrepancy subsystem is further configured to enable the evaluator to select diversionary activities over a distributed computer network when the evaluator is signaled for the break time.

22. A system for increasing evaluator efficiency by monitoring evaluator performance in evaluating data items and recommending break times to the evaluator, the system comprising:

a) a receiving subsystem configured to electronically receive first evaluation rates corresponding to an evaluator'"'"'s evaluating of data items at predetermined past intervals in time, and electronically receive a second present evaluation rate corresponding to the evaluator'"'"'s evaluating of the data items at a present time;

b) a comparison subsystem configured to electronically compare the second evaluation rate and the first evaluation rate; and

c) a discrepancy subsystem configured to electronically detect a discrepancy based on predefined criteria between the second evaluation rate and the first evaluation rate as indicated by the comparison subsystem and signal over the distributed computer network the evaluator to halt data item evaluation for a break time.

23. The system of claim 22 wherein the discrepancy subsystem is further configured to electronically detect a discrepancy based on predefined criteria between the second present evaluation rate and the first evaluation rate as indicated by the comparison subsystem and provide the evaluator with diversionary activities over the distributed computer network.

24. The system of claim 22 wherein the discrepancy subsystem is further configured to enable the evaluator to select diversionary activities over the distributed network when the evaluator has been signaled for the break time.

25. A computer-readable program storage medium tangibly embodying a data package and associated verification instructions executable by a computing system for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in evaluating data items to predetermined performance criteria, the method comprising the steps of:

a) electronically receiving information related to a first performance level of a first evaluator'"'"'s evaluation of data items, and a predetermined performance level;

b) electronically comparing the first performance level and the predetermined performance level; and

c) electronically reporting a discrepancy based on predefined criteria between the first performance level and the predetermined performance level.

26. The computer-readable program storage medium of claim 25 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator.

27. The computer-readable program storage medium of claim 25 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to a system manager.

28. A computer-readable program storage medium tangibly embodying a data package and associated verification instructions executable by a computing system for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in scoring data items to predetermined performance criteria, the method comprising the steps of:

a) electronically receiving information related to a first validity level of a first evaluator'"'"'s evaluation of data items, and a predetermined validity level;

b) electronically comparing the first validity level and the predetermined validity level; and

c) electronically reporting a discrepancy based on predefined criteria between the first validity level and the predetermined validity level.

29. The computer-readable program storage medium of claim 28 wherein the step of electronically receiving information comprises receiving an evaluation result for a quality item from the first evaluator and a known score for the quality item.

30. The computer-readable program storage medium of claim 28 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator.

31. The computer-readable program storage medium of claim 28 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to a system manager.

32. A computer-readable program storage medium tangibly embodying a data package and associated verification instructions executable by a computing system for electronically providing performance feedback based upon a comparison of a an evaluator'"'"'s performance in evaluating data items to predetermined performance criteria, the method comprising the steps of:

a) electronically receiving information related to a first reliability level of a first evaluator'"'"'s evaluation of data items, and a predetermined reliability level;

b) electronically comparing the first reliability level and the predetermined reliability level; and

c) electronically reporting a discrepancy based on predefined criteria between the first reliability level and the predetermined reliability level.

33. The computer-readable program storage medium of claim 32 wherein the step of electronically receiving information comprises receiving a first evaluation result corresponding to the first evaluator'"'"'s evaluation of a data item during a first time period and for receiving a second evaluation result corresponding to the first evaluator'"'"'s evaluation of the data item during a second time period.

34. The computer-readable program storage medium of claim 32 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator.

35. The computer-readable program storage medium of claim 32 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to a system manager.

36. A computer-readable program storage medium tangibly embodying a data package and associated verification instructions executable by a computing system for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in evaluation of data items to predetermined performance criteria, the method comprising the steps of:

a) electronically receiving information related to a first speed level of a first evaluator'"'"'s evaluation of data items, and a predetermined speed level;

b) electronically comparing the first speed level and the predetermined speed level; and

c) electronically reporting a discrepancy based on predefined criteria between the first speed level and the predetermined speed level.

37. The computer-readable program storage medium of claim 36 wherein the step of electronically receiving information comprises receiving at least one of the following:

a first evaluation rate corresponding to the first evaluator'"'"'s evaluation of the data items, an average evaluation rate corresponding to a selected group of evaluators'"'"' evaluation of the data items, and a second evaluation rate corresponding to a second evaluator'"'"'s evaluation of the data items.

38. The computer-readable program storage medium of claim 36 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator.

39. The computer-readable program storage medium of claim 36 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to a system manager.

40. A computer-readable program storage medium tangibly embodying a data package and associated verification instructions executable by a computing system for increasing evaluator efficiency by monitoring evaluator performance in evaluating data items and recommending break times to the evaluator, the method comprising the steps of:

a) electronically receiving first performance levels corresponding to an evaluator'"'"'s evaluation of data items at predetermined past intervals in time, and electronically receiving a second predetermined performance level;

b) electronically comparing the second predetermined performance level and the first performance level; and

c) electronically detecting a discrepancy based on predefined criteria between the second predetermined performance level and the first performance level and signaling the evaluator to halt data item evaluation for a break time.

41. The computer-readable program storage medium of claim 40 wherein the step of electronically receiving first performance levels comprises electronically receiving first validity levels from the evaluator and electronically receiving a second predetermined validity level.

42. The computer-readable program storage medium of claim 40 wherein the step of electronically receiving first performance levels comprises electronically receiving first reliability levels from the evaluator and electronically receiving a second predetermined reliability level.

43. The computer-readable program storage medium of claim 40 wherein the step of electronically receiving first performance levels comprises electronically receiving first speed levels from the evaluator and electronically receiving a second predetermined speed level.

44. The computer-readable program storage medium of claim 40 wherein the step of electronically detecting a discrepancy comprises electronically detecting a discrepancy based on predefined criteria between the second predetermined performance level and the first performance levels and providing the evaluator with diversionary activities.

45. The computer-readable program storage medium of claim 40 wherein the step of electronically detecting a discrepancy comprises enabling the evaluator to select diversionary activities when the evaluator is signaled for the break time.

46. A computer-readable program storage medium tangibly embodying a data package and associated verification instructions executable by a computing system for increasing evaluator efficiency by monitoring evaluator performance in evaluating data items and recommending break times to the evaluators, the method comprising the steps of:

a) electronically receiving a first evaluation rate corresponding to an evaluator'"'"'s evaluating of data items at predetermined past intervals in time, and electronically receiving a second present evaluation rate corresponding to the evaluator'"'"'s scoring of the data items at a present time;

b) electronically comparing the second evaluation rate and the first evaluation rate; and

c) electronically detecting a discrepancy based on predefined criteria between the second evaluation rate and the first evaluation rate as indicated by the comparison step and signaling the evaluator to halt data item evaluation for a break time.

47. The computer-readable program storage medium of claim 46 wherein the step of electronically detecting a discrepancy further comprises electronically detecting a discrepancy based on predefined criteria between the second present evaluation rate and the first evaluation rate as indicated by the comparison step and providing the evaluator with diversionary activities.

48. The computer-readable program storage medium of claim 46 wherein the step of electronically detecting a discrepancy further comprises enabling the evaluator to select diversionary activities when the evaluator has been signaled for the break time.

49. A method for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in evaluating data items to predetermined performance criteria, the method comprising the steps of:

a) electronically receiving information related to a first performance level of a first evaluator'"'"'s evaluation of data items, and a predetermined performance level;

b) electronically comparing the first performance level and the predetermined performance level; and

c) electronically reporting a discrepancy based on predefined criteria between the first performance level and the predetermined performance level.

50. The method of claim 49 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator.

51. The method of claim 49 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to a system manager.

52. A method for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in scoring data items to predetermined performance criteria, the method comprising the steps of:

a) electronically receiving information related to a first validity level of a first evaluator'"'"'s evaluation of data items, and a predetermined validity level;

b) electronically comparing the first validity level and the predetermined validity level; and

c) electronically reporting a discrepancy based on predefined criteria between the first validity level and the predetermined validity level.

53. The method of claim 52 wherein the step of electronically receiving information comprises receiving an evaluation result for a quality item from the first evaluator and a known score for the quality item.

54. The method of claim 52 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator.

55. The method of claim 52 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to a system manager.

56. A method for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in evaluating data items to predetermined performance criteria, the method comprising the steps of:

a) electronically receiving information related to a first reliability level of a first evaluator'"'"'s evaluation of data items, and a predetermined reliability level;

b) electronically comparing the first reliability level and the predetermined reliability level; and

c) electronically reporting a discrepancy based on predefined criteria between the first reliability level and the predetermined reliability level.

57. The method of claim 56 wherein the step of electronically receiving information comprises receiving a first evaluation result corresponding to the first evaluator'"'"'s evaluation of a data item during a first time period and for receiving a second evaluation result corresponding to the first evaluator'"'"'s evaluation of the data item during a second time period.

58. The method of claim 56 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator.

59. The method of claim 56 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to a system manager.

60. A method for electronically providing performance feedback based upon a comparison of an evaluator'"'"'s performance in evaluation of data items to predetermined performance criteria, the method comprising the steps of:

a) electronically receiving information related to a first speed level of a first evaluator'"'"'s evaluation of data items, and a predetermined speed level;

b) electronically comparing the first speed level and the predetermined speed level; and

c) electronically reporting a discrepancy based on predefined criteria between the first speed level and the predetermined speed level.

61. The method of claim 60 wherein the step of electronically receiving information comprises receiving at least one of the following:

a first evaluation rate corresponding to the first evaluator'"'"'s evaluation of the data items, an average evaluation rate corresponding to a selected group of evaluators'"'"' evaluation of the data items, and a second evaluation rate corresponding to a second evaluator'"'"'s evaluation of the data items.

62. The method of claim 60 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to the first evaluator.

63. The method of claim 60 wherein the step of electronically reporting a discrepancy comprises electronically reporting the discrepancy to a system manager.

64. A method for increasing evaluator efficiency by monitoring evaluator performance in evaluating data items and recommending break times to the evaluators, the method comprising the steps of:

a) electronically receiving first performance levels corresponding to an evaluator'"'"'s evaluation of data items at predetermined past intervals in time, and electronically receiving a second predetermined performance level;

b) electronically comparing the second predetermined performance level and the first performance level; and

c) electronically detecting a discrepancy based on predefined criteria between the second predetermined performance level and the first performance level and signaling the resolver evaluator to halt data item evaluation for a break time.

65. The method of claim 64 wherein the step of electronically receiving first performance levels comprises electronically receiving first validity levels from the evaluator and electronically receiving a second predetermined validity level.

66. The method of claim 64 wherein the step of electronically receiving first performance levels comprises electronically receiving first reliability levels from the evaluator and electronically receiving a second predetermined reliability level.

67. The method of claim 64 wherein the step of electronically receiving first performance levels comprises electronically receiving first speed levels from the evaluator and electronically receiving a second predetermined speed level.

68. The method of claim 64 wherein the step of electronically detecting a discrepancy comprises electronically detecting a discrepancy based on predefined criteria between the second predetermined performance level and the first performance levels and providing the evaluator with diversionary activities.

69. The method of claim 64 wherein the step of electronically detecting a discrepancy comprises enabling the evaluator to select diversionary activities when the evaluator is signaled for the break time.

70. A method for increasing evaluator efficiency by monitoring evaluator performance in evaluating data items and recommending break times to the evaluators, the method comprising the steps of:

a) electronically receiving a first evaluation rate corresponding to an evaluator'"'"'s evaluating of data items at predetermined past intervals in time, and electronically receiving a second present evaluation rate corresponding to the evaluator'"'"'s scoring of the data items at a present time;

b) electronically comparing the second evaluation rate and the first evaluation rate; and

c) electronically detecting a discrepancy based on predefined criteria between the second evaluation rate and the first evaluation rate as indicated by the comparison step and signaling the evaluator to halt data item evaluation for a break time.

71. The method of claim 70 wherein the step of electronically detecting a discrepancy further comprises electronically detecting a discrepancy based on predefined criteria between the second present evaluation rate and the first evaluation rate as indicated by the comparison step and providing the evaluator with diversionary activities.

72. The method of claim 70 wherein the step of electronically detecting a discrepancy further comprises enabling the evaluator to select diversionary activities when the evaluator has been signaled for the break time.