Test question response verification system

• US 6,751,351 B2
• Filed: 03/05/2001
• Issued: 06/15/2004
• Est. Priority Date: 03/05/2001
• Status: Active Grant

• Current Assignee: NCS Pearson Incorporated (Pearson plc)
• Original Assignee: NSC Pearson Incorporated
• Inventors: Knowles, Kristian, Pascale, Robert Nicola
• Primary Examiner(s): Mariam, Daniel
• Assistant Examiner(s): Patel, Shefali
• Application Number:
  US09/799,299
• Publication Number:
  US 20020123028A1
• Time in Patent Office: 1,198 Days
• Field of Search: 382/181, 382/173, 382/177, 434/324, 434/353, 434/354, 709/1
• US Class Current: 382/181
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

The present invention relates to a system and method for verifying and editing optically imaged responses received from a response provider.

Claims

1. In a computerized test processing workflow system configured for analyzing electronic images of test question responses provided by a test taker, the system comprising a server subsystem having at least one server computer and a client subsystem having one or more client computers used by human editors located at the client computers, each human editor having particular skills for examining particular test portion subject matter, a method for enhancing the speed and accuracy of data within the workflow system in order to effect productive use of geographically-distributed human editors having particular appropriate skills, the method comprising:

a. selecting from each electronic image of test question responses at least one image clip;

b. applying image recognition to generate data electronically from the at least one image clip;

c. determining whether verification is needed of the data derived from the at least one image clip;

d. matching image clip information to the particular skills of the human editors in order to select human editors having the particular appropriate skills;

e. recompiling selected image clips into coherent image portions of the test question responses; and

f. selectively presenting the coherent image portions to appropriate human editors located at the remote locations;

whereby, through the transfer of the recompiled coherent image portions to appropriate human editors, widely-distributed human editors of greatly varying skills can be effectively used at the remote locations without the inefficiency of transferring relatively-large, unclipped electronic images of test question responses.

2. The method of claim 1 further comprising the step of analyzing through observation by an appropriate human editor the portion of the response disclosed by the coherent image portions.

3. The method of claim 2 further comprising the step of electronically recording a decision as to the human editors analysis, the decision being generated by the selected human editor for each distributed coherent image portion.

4. The method of claim 2, wherein the step of analyzing the coherent image portions through observation further comprises determining whether the coherent image portion presented matches particular data simultaneously presented to the human editor with the coherent image portion.

5. The method of claim 2, wherein the step of analyzing the coherent image portions through observation further comprises reaching a decision as to what the portion of the response disclosed by the coherent image portion represents.

6. The method of claim 1, wherein the step of applying image recognition to the at least one image clip comprises applying image character recognition or optical character recognition.

7. The method of claim 1, wherein the step of matching image clip information to the particular skills of the human editors further comprises matching the particular test subject matter of the test question with the particular subject matter skills of the human editor.

8. The method of claim 1, wherein the step of matching image clip information to the particular skills of the human editors further comprises using a bit mask of up to 32 bits in which to store information related to each image clip.

9. The method of claim 1 wherein, following the step of selecting at least one image clip from the electronic images of test question responses, the method further comprises:

a. electronically assigning an individual digital identifier to each of the image clips, wherein each individual digital identifier represents at least a portion of information related to the test taker or the test;

b. the step of matching image clip information to the particular skills of the human editors comprises processing the individual digital identifiers related to each coherent image portion against the particular skills of the human editors;

whereby, through the transfer of the recompiled image portions tagged with the digital codes to appropriate human editors, widely-distributed human editors of greatly varying skills can be effectively used at the remote locations without the inefficiency of transferring relatively-large, unclipped electronic images of test question responses.

10. The method of claim 1, further comprising the following steps:

a. recompiling again selected image clips already viewed by appropriate human editors into coherent image portions of the test question response;

b. selectively presenting the coherent image portions to appropriate human editors located at the remote locations but not to the same human editors who previously viewed the coherent image portions; and

c. electronically recording at least one entry generated by the appropriate human editor for each presented coherent image portion.

11. In a computerized test processing workflow system configured for analyzing electronic images of test question responses provided by a test taker, the system comprising a server subsystem having at least one server computer and a client subsystem having one or more client computers used by human editors located at the client computers, each human editor having particular skills for examining particular test portion subject matter, a method for enhancing the speed and accuracy of data within the workflow system in order to effect productive use of geographically-distributed human editors having particular appropriate skills, the method comprising:

a. selecting from each electronic image of test question responses at least one image clip;

b. applying image recognition to generate data electronically from the at least one image clip;

c. determining whether verification is needed of the data derived from the at least one image clip;

d. matching image clip information to the particular skills of the human editors in order to select human editors having the particular appropriate skills;

e. recompiling selected image clips into coherent image portions of the test question responses;

f. selectively presenting the coherent image portions to appropriate human editors located at the remote locations;

g. analyzing through observation by an appropriate human editor the portion of the response disclosed by the coherent image portions;

h. electronically recording a decision as to the human editors analysis, the decision being generated by the selected human editor for each distributed coherent image portion;

whereby, through the transfer of the recompiled coherent image portions to appropriate human editors, widely-distributed human editors of greatly varying skills can be effectively used at the remote locations without the inefficiency of transferring relatively-large, unclipped electronic images of test question responses.

12. In a computerized test processing workflow system configured for analyzing electronic images of test question responses provided by a test taker, the system comprising a server subsystem having at least one server computer and a client subsystem having one or more client computers used by human editors located at the client computers, each human editor having particular skills for examining particular test portion subject matter, a method for enhancing the speed and accuracy of data within the workflow system in order to effect productive use of geographically-distributed human editors having particular appropriate skills, the method comprising:

a. selecting from each electronic image of test question responses at least one image clip;

b. electronically assigning an individual digital identifier to each of the image clips, wherein each individual digital identifier represents at least a portion of information related to the test taker or the test;

c. applying image recognition to generate data electronically from the at least one image clip;

d. determining whether verification is needed of the data derived from the at least one image clip;

e. processing the individual digital identifier related to each image clip against the particular skills of the human editors in order to select human editors having the particular appropriate skills;

f. recompiling selected image clips into coherent image portions of the test question response;

g. selectively presenting the coherent image portions to appropriate human editors located at the remote locations;

h. analyzing through observation by an appropriate human editor the portion of the response disclosed by the coherent image portions;

i. electronically recording a decision verifying the particular data, the decision being generated by the selected human editor for each distributed coherent image portion;

whereby, through the transfer of the recompiled coherent image portions tagged with the digital codes to appropriate human editors, widely-distributed human editors of greatly varying skills can be effectively used at the remote locations without the inefficiency of transferring for viewing relatively-large, unclipped electronic images of test question responses.

13. The method of claim 12, wherein the step of analyzing through observation includes determining whether the coherent image portion presented matches the particular data, also presented, electronically created from the image clips that form the coherent image portion.

14. The method of claim 12, wherein the step of analyzing through observation includes reaching a decision as to what the portion of the response disclosed by the coherent image portion represents.

15. A system for improving the productive use of geographically-distributed human editors having particular appropriate skills by enhancing the speed and accuracy of data within a computerized test processing workflow system configured for analyzing electronic images of test question responses provided by a test taker, the system comprising

a. a server subsystem having at least one server computer;

b. a client subsystem having at least one client computer used by human editors located at the client computers and coupled to the server subsystem;

c. the at least one client computer configured for selecting from each electronic image of test question responses at least one image clip;

d. the at least one client computer configured to apply image recognition to generate data electronically from the at least one image clip;

e. the at least one server computer configured to match image clip information to the particular skills of the human editors in order to select human editors having the particular appropriate skills;

f. the at least one server computer configured to recompile selected image clips into coherent image portions of the test question responses; and

g. the at least one client computer configured to selectively present the coherent image portions to appropriate human editors located at the remote locations.

16. The system of claim 15, further comprising the client subsystem configured to facilitate analyzing through observation by an appropriate human editor the portion of the response disclosed by the coherent image portions.

17. The system of claim 15, further comprising the client subsystem configured to facilitate electronically recording a decision verifying the particular data, the decision being generated by the selected human editor for each distributed coherent image portion.

18. The system of claim 15, further comprising the server subsystem configured to electronically assign an individual digital identifier to each of the image clips, wherein each individual digital identifier represents at least a portion of information related to the test taker or the test.

19. The system of claim 18, further comprising the server subsystem configured to process the individual digital identifier related to each image clip against the particular skills of the human editors in order to select human editors having the particular appropriate skills.

20. The system of claim 15, further comprising the server subsystem configured to:

a. recompile again selected image clips already viewed by appropriate human editors into coherent image portions of the test question response;

b. selectively present the coherent image portions to appropriate human editors located at the remote locations but not to the same human editors who previously viewed the coherent image portions; and

c. electronically record at least one entry generated by the appropriate human editor for each presented coherent image portion.

21. A computer program product readable by a computing system for a computer process for a computerized test processing workflow system configured for analyzing electronic images of test question responses provided by a test taker, the system comprising a server subsystem having at least one server computer and a client subsystem having one or more client computers used by human editors located at the client computers, each human editor having particular skills for examining particular test portion subject matter, a method for enhancing the speed and accuracy of data within the workflow system in order to effect productive use of geographically-distributed human editors having particular appropriate skills, the computer program product comprising encoded instructions either embedded on a computer-readable media or transmitted in a propagating wave, the encoded instructions being arranged and configured to be read by the computing system for causing the computing system to perform the steps of:

a. selecting from each electronic image of test question responses at least one image clip;

b. applying image recognition to generate data electronically from the at least one image clip;

c. determining whether verification is needed of the data derived from the at least one image clip;

d. matching image clip information to the particular skills of the human editors in order to select human editors having the particular appropriate skills;

e. recompiling selected image clips into coherent image portions of the test question responses; and

f. selectively presenting the coherent image portions to appropriate human editors located at the remote locations;

whereby, through the transfer of the recompiled coherent image portions to appropriate human editors, widely-distributed human editors of greatly varying skills can be effectively used at the remote locations without the inefficiency of transferring relatively-large, unclipped electronic images of test question responses.

22. A computer program product readable by a computing system for a computer process for a computerized test processing workflow system configured for analyzing electronic images of test question responses provided by a test taker, the system comprising a server subsystem having at least one server computer and a client subsystem having one or more client computers used by human editors located at the client computers, each human editor having particular skills for examining particular test portion subject matter, a method for enhancing the speed and accuracy of data within the workflow system in order to effect productive use of geographically-distributed human editors having particular appropriate skills, the computer program product comprising encoded instructions either embedded on a computer-readable media or transmitted in a propagating wave, the encoded instructions being arranged and configured to be read by the computing system for causing the computing system to perform the steps of:

a. selecting from each electronic image of test question responses at least one image clip;

b. applying image recognition to generate data electronically from the at least one image clip;

c. determining whether verification is needed of the data derived from the at least one image clip;

d. matching image clip information to the particular skills of the human editors in order to select human editors having the particular appropriate skills;

e. recompiling selected image clips into coherent image portions of the test question responses;

f. selectively presenting the coherent image portions to appropriate human editors located at the remote locations;

g. analyzing through observation by an appropriate human editor the portion of the response disclosed by the coherent image portions;

h. electronically recording a decision as to the human editors analysis, the decision being generated by the selected human editor for each distributed coherent image portion;

whereby, through the transfer of the recompiled coherent image portions to appropriate human editors, widely-distributed human editors of greatly varying skills can be effectively used at the remote locations without the inefficiency of transferring relatively-large, unclipped electronic images of test question responses.

23. A system for improving the productive use of geographically-distributed human editors having particular appropriate skills by enhancing the speed and accuracy of data within a computerized test processing workflow system configured for analyzing electronic images of test question responses provided by a test taker, the system comprising:

a. means for selecting from each electronic image of test question responses at least one image clip;

b. means for applying image recognition to generate data electronically from the at least one image clip;

c. means for matching image clip information to the particular skills of the human editors in order to select human editors having the particular appropriate skills;

d. means for recompiling selected image clips into coherent image portions of the test question responses; and

e. means for present the coherent image portions to appropriate human editors located at the remote locations.