Categorized data item reporting system and method

• US 20040086841A1
• Filed: 10/20/2003
• Published: 05/06/2004
• Est. Priority Date: 02/05/1993
• Status: Active Grant

• Current Assignee: NCS Pearson Incorporated (Pearson plc)
• Original Assignee: NCS Pearson Incorporated (Pearson plc)
• Inventors: Clark, Jay V., Moyer, Judith, Bramlett, William L. Jr.
• Granted Patent: US 6,918,772 B2
• Time in Patent Office: Days
• US Class Current: 434/350
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G09B 7/066 with answer indicating cards, blocks
• CPC Class Codes: Y10S 706/927 Education or instruction

Abstract

A system for managing resolvers of test answers to maintain consistent and efficient scoring of test answers. Collaborative scoring monitors and resolves discrepancies in two or more scores for the same test answers. Quality items monitor a test resolver'"'"'s scoring performance against a known standard. Performance feedback data tracks a resolver'"'"'s scoring rate to monitor efficiency and optionally recommend break times or provide the resolver with diversionary activities during breaks. Consistent scoring is further promoted by selectively matching resolvers with test answers based on classifications of the resolvers and persons who generated the test answers. An on-line scoring guide dynamically provides test resolvers with rules for scoring answers to particular test questions.

Claims

1. A system for verifying test answers and providing collaborative scoring of the test answers between two or more test resolvers if the test answers contain a discrepancy, the system comprising:

a) first receive means for electronically receiving an answer to a test question, the answer comprising an electronic representation of at least a portion of a test answer sheet;

b) transmit means for electronically and independently transmitting the answer to first and second test resolvers;

c) second receive means for electronically receiving first and second test scores from the first and second test scorers; and

d) verification means for electronically comparing the first and second test scores and for selectively recording a value related to the first and second test scores if the first and second test scores did not contain a discrepancy based on predefined criteria.

2. The system of claim 1 further comprising display means for electronically displaying the test question.

3. The system of claim 1 wherein the verification means further comprises collaboration means for providing an electronic communication link between the first and second test resolvers and, if the first and second scores contain the discrepancy based on the predefined criteria, for signalling the first and second test resolvers to interactively determine and enter a joint score, the verification means further comprising recording means for recording the joint score.

4. The system of claim 3 wherein the collaboration means further comprises means for tracking average scores entered by the first and second resolvers and for preventing the first and second resolvers from scoring the same test item if the first and second resolvers each have average scores within predefined scoring criteria.

5. The system of claim 3 wherein the verification means further comprises means for preventing the first and second test resolvers from receiving another answer to score until the first and second test resolvers have entered the joint score.

6. The system of claim 3 wherein the verification means comprises third resolver means for transmitting the test item to a third test resolver if the first and second test scorers do not enter the joint score within a predetermined time period.

7. The system of claim 1 wherein the verification means further comprises third resolver means for transmitting the answer to a third test resolver, if the first and second scores contain the discrepancy based on the predefined criteria, and for recording a third test score received from the third test resolver.

8. The system of claim 7 wherein the third resolver means further comprises means for providing the third test resolver with the first and second scores.

9. The system of claim 7 wherein the third resolver means further comprises means for preventing the third test resolver from viewing the first and second scores before the third test resolver records the third test score.

10. The system of claim 1, wherein the verification means further comprises means for electronically linking the first and second test resolvers when the first and second test resolvers are located in different geographical locations.

11. The system of claim 1 wherein the verification means comprises means for recording at least one of a greater value of the first and second test scores, a lower value of the first and second test scores, and an average value of the first and second test scores.

12. A system for evaluating the performance of test resolvers by routing quality-control questions for scoring to test resolvers, the quality-control questions comprising a test answer with a score known to quality-control personnel, the system comprising:

a) transmit means for electronically transmitting to a test resolver an unidentified quality-control question comprising a test answer with a score known to quality-control personnel;

b) receive means for electronically receiving from the test resolver a score for the quality-control question; and

c) processing means for electronically comparing the score known to the quality-control personnel and the resolver score and for reporting a discrepancy between the known score and the resolver score.

13. A system for electronically providing performance feedback based upon a comparison of a test resolver'"'"'s performance to predetermined performance criteria, the system comprising:

a) receive means for electronically receiving information related to a first performance level from a first resolver and a predetermined performance level;

b) comparison means for electronically comparing the first performance level and the predetermined performance level; and

c) feedback means for electronically reporting a discrepancy based on predefined criteria between the first performance level and the predetermined performance level.

14. The system of claim 13 wherein the feedback means comprises means for electronically reporting the discrepancy to the first test resolver.

15. The system of claim 13 wherein the feedback means comprises means for electronically reporting the discrepancy to a system manager.

16. A system for electronically providing performance feedback based upon a comparison of a test resolver'"'"'s performance to predetermined performance criteria, the system comprising:

a) receive means for electronically receiving information related to a first test-score validity level from a first resolver and a predetermined validity level;

b) comparison means for electronically comparing the first validity level and the predetermined validity level; and

c) feedback means for electronically reporting a discrepancy based on predefined criteria between the first validity level and the predetermined validity level.

17. The system of claim 16 wherein the receive means comprises means for receiving a test score for a quality item from the first resolver and a known score for the quality item.

18. The system of claim 16 wherein the feedback means comprises means for electronically reporting the discrepancy to the first test resolver.

19. The system of claim 16 wherein the feedback means comprises means for electronically reporting the discrepancy to a system manager.

20. A system for electronically providing performance feedback based upon a comparison of a test resolver'"'"'s performance to predetermined performance criteria, the system comprising:

a) receive means for electronically receiving information related to a first reliability level from a first resolver and a predetermined reliability level;

b) comparison means for electronically comparing the first reliability level and the predetermined reliability level; and

c) feedback means for electronically reporting a discrepancy based on predefined criteria between the first reliability level and the predetermined reliability level.

21. The system of claim 20 wherein the receive means comprises means for receiving during a first time period a first score from the first resolver for a test item and for receiving during a second time period a second score from the first resolver for the test item.

22. The system of claim 20 wherein the feedback means comprises means for electronically reporting the discrepancy to the first test resolver.

23. The system of claim 20 wherein the feedback means comprises means for electronically reporting the discrepancy to a system manager.

24. A system for electronically providing performance feedback based upon a comparison of a test resolver'"'"'s performance to predetermined performance criteria, the system comprising:

a) receive means for electronically receiving information related to a first speed level from a first resolver and a predetermined speed level;

b) comparison means for electronically comparing the first speed level and the predetermined speed level; and

c) feedback means for electronically reporting a discrepancy based on predefined criteria between the first speed level and the predetermined speed level.

25. The system of claim 24 wherein the receive means comprises means for receiving at least one of the following:

a first scoring rate for the first resolver, an average scoring rate from a selected group of resolvers, and a second scoring rate from a second resolver.

26. The system of claim 24 wherein the feedback means comprises means for electronically reporting the discrepancy to the first test resolver.

27. The system of claim 24 wherein the feedback means comprises means for electronically reporting the discrepancy to a system manager.

28. A system for increasing test resolver efficiency by monitoring test resolver performance and recommending break times to the test resolvers, the system comprising:

a) receive means for electronically receiving at predetermined past intervals in time first performance levels from a test resolver and for electronically receiving a second predetermined performance level;

b) comparison means for electronically comparing the second predetermined performance level and the first performance level; and

c) recommendation means for electronically detecting a discrepancy based on predefined criteria between the second predetermined performance level and the first performance level as indicated by the comparison means and for signalling the test resolver to halt test scoring for a break time.

29. The system of claim 28 wherein the receive means comprises means for electronically receiving first validity levels from the test resolver and for electronically receiving a second predetermined validity level.

30. The system of claim 28 wherein the receive means comprises means for electronically receiving first reliability levels from the test resolver and for electronically receiving a second predetermined reliability level.

31. The system of claim 28 wherein the receive means comprises means for electronically receiving first speed levels from the test resolver and for electronically receiving a second predetermined speed level.

32. The system of claim 28 wherein the recommendation means further comprises diversion means for electronically detecting a discrepancy based on predefined criteria between the second scoring rate and the first scoring rates as indicated by the comparison means and for providing the test resolver with diversionary activities.

33. The system of claim 28 wherein the recommendation means further comprises resolver-selected diversion means for enabling the test resolver to select diversionary activities when the recommendation means has signalled the test resolver for the break.

34. A system for increasing test resolver efficiency by monitoring test resolver performance and recommending break times to the test resolvers, the system comprising:

a) receive means for electronically receiving at predetermined past intervals in time first test scoring rates from a test resolver and for electronically receiving a second present scoring rate from the test resolver;

b) comparison means for electronically comparing the second scoring rate and the first scoring rates; and

c) recommendation means for electronically detecting a discrepancy based on predefined criteria between the second scoring rate and the first scoring rates as indicated by the comparison means and for signalling the test resolver to halt test scoring for a break time.

35. The system of claim 34 wherein the recommendation means further comprises diversion means for electronically detecting a discrepancy based on predefined criteria between the second scoring rate and the first scoring rates as indicated by the comparison means and for providing the test resolver with diversionary activities.

36. The system of claim 34 wherein the recommendation means further comprises resolver-selected diversion means for enabling the test resolver to select diversionary activities when the recommendation means has signalled the test resolver for the break.

37. A system for preventing bias in the scoring of answers to tests taken by persons in different particular classifications, the system comprising:

a) receive means for electronically receiving a plurality of answers to test questions, the test answers each corresponding to a test taken by a person from a particular definable classification and comprising an electronic representation of at least a portion of one or more test answer sheets;

b) classification means for receiving a particular classification of each of the respondents or a group of the respondents and for electronically associating each of the particular classifications with the test answers for the corresponding respondent or group; and

c) routing means for electronically selecting a test resolver for each of the answers based upon the particular classification of the corresponding respondent or group and for transmitting the answers to the selected test resolver.

38. The system of claim 37 wherein:

a) the classification means comprises means for receiving a racial classification identifying a race of each of the test resolvers; and

b) the routing means comprises means for electronically transmitting each of the test answers to a test resolver having a same, different, or preferred blend racial classification as the racial classification of the corresponding respondent or group.

39. The system of claim 37 wherein:

a) the classification means comprises means for receiving a geographical classification identifying a location of each of the test resolvers; and

b) the routing means comprises means for electronically transmitting each of the answers to a test resolver having a same, different, or preferred blend geographical classification as the geographical classification of the corresponding respondent or group.

40. A system for electronically and dynamically providing scoring rules to test resolvers to assist the test resolvers in scoring test items to which the scoring rules apply, the system comprising:

a) receive means for electronically receiving a plurality of answers for test questions, the answers each comprising an electronic representation of at least a portion of a test answer sheet;

b) storage means for electronically storing a plurality of scoring rules, each of the scoring rules relating to a procedure for scoring at least a particular one of the test questions;

c) display means for selectively presenting the test answers to a test resolver; and

d) rules means for selectively presenting to the test resolver the scoring rules related to the test question corresponding to the answer presented to the test resolver.