Apparatus and method for scoring nonobjective assessment materials through the application and use of captured images

• US 5,672,060 A
• Filed: 11/28/1994
• Issued: 09/30/1997
• Est. Priority Date: 07/08/1992
• Status: Expired due to Term

• Current Assignee: NCS Pearson Incorporated (Pearson plc)
• Original Assignee: Meadowbrook Industries Limited
• Inventors: Poor, David D. S.
• Primary Examiner(s): Chin, Wellington
• Assistant Examiner(s): Corrielus, Jean B.
• Application Number:
  US08/345,214
• Time in Patent Office: 1,037 Days
• Field of Search: 382/321
• US Class Current: 434/322
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G09B 7/066 with answer indicating cards, blocks

Abstract

An apparatus and method are provided for scoring nonobjective assessment materials recorded on paper, such as essays, by displaying digitized images of the assessment materials on computer screens for scoring by human scorers. A digitized image of the assessment material is captured by scanning an assessment form and storing the image in an electronic file. The stored digitized image is displayed to the scorer and a score entered by the scorer is returned to a scoring center where reports are generated. Embodiments of the invention include capturing the image in association with Optical Mark Reading (OMR), capturing identifying information through OMR, training and testing the scorer by displaying images of anchor items and validity items for scoring. A series of computer data files ensure control of the scoring process.

Claims

1. A method of electronically displaying to a human scorer an image of handwritten nonobjective assessment materials recorded by a test taker in an assessment area of an assessment form, and recording a score entered by the scorer, comprising the steps of:

providing an assessment form having thereon at least one assessment area;

recording in an electronic file information identifying the test taker;

recording in an electronic file information identifying the assessment form;

capturing from the assessment form a digitized image of said assessment materials recorded in the at least one assessment area;

storing said digitized image in an

electronic image file;

recording in an electronic file information to locate said stored image of said assessment materials;

selectively displaying electronically said stored image of said assessment materials but not said information identifying the test taker on a computer screen for scoring by the scorer; and

recording in an electronic file a score entered by the scorer for the stored digitized image of the assessment materials.

2. A method in accordance with claim 1, wherein the digitized image is stored as a bit-mapped image.

3. A method in accordance with claim 1, wherein the stored digitized image is converted to a bit-mapped image and stored as a compressed bit-mapped image.

4. A method in accordance with claim 1, wherein said step of capturing a digitized image comprises the further steps of:

identifying the form by OMR;

determining the location of at least one of the assessment areas on the form;

determining by OMR whether marks readable by OMR are present within the at least one assessment area;

determining by OMR the portion of the at least one assessment area in which marks are present; and

storing in digital form the image of the portion of the at least one assessment area that was determined to contain marks.

5. A method in accordance with claim 1, wherein said step of storing said digitized image further comprises a step of recording the locations in a data file of the digitized image comprising the steps of:

creating a companion data file; and

creating a separate record in the companion data file for each assessment form containing one or more digitized images, the separate record having the location of the digitized image specified as an offset from the start of the file containing the digitized image.

6. A method in accordance with claim 5, wherein the step of creating a separate record further comprises the step of providing a second companion data file and storing information identifying the test taker in the second companion data file.

7. A method in accordance with claim 5, wherein the step of storing information identifying the test taker further comprises the step of capturing information from the assessment form using OMR.

8. A method in accordance with claim 1, further comprising the steps of:

creating and storing in computer memory training items having the same format as said digitized image; and

displaying said items to the scorer in the same format as items to be scored.

9. A method in accordance with claim 1, further comprising the steps of:

creating and storing in computer memory assessment responses having the same format as said digitized image; and

supplying and displaying said assessment responses to the scorer in the same format as items to be scored, whereby said assessment responses serve as validating items for testing the accuracy of the scorer'"'"'s selection of scores.

10. A method in accordance with claim 1, wherein the process of displaying the digitized image further comprises the steps of:

transferring to a scorer an image file containing the image to be displayed and a companion data file including an offset so that the image can be located in the image file;

locating the image to be displayed;

reading the digitized image from the file; and

converting the image to a form appropriate for display on a computer screen.

11. A method in accordance with claim 1, wherein the process of displaying the digitized image comprises the steps of:

transferring to a scorer at a remote site the image file containing the image to be displayed and a companion data file for locating the image in the image file;

locating the image to be displayed;

reading the digitized image from the image file; and

converting the image to a form appropriate for display on a computer screen at the remote site.

12. A method in accordance with claim 1, wherein the step of recording a score further comprises the step of entering the score in a field displayed on the same computer screen used to display the image.

13. A method in accordance with claim 9 further comprising the steps of:

providing a main scoring center;

transferring the recorded score for a validity item to the main scoring center; and

using the transferred score to determine scorer validity.

14. A method in accordance with claim 1, further comprising the steps of:

providing a scoring site and a main scoring center, said recording step being performed at the scoring cite;

transferring the recorded score to the main scoring center; and

associating in a computer file, the transferred score with the identity of the person who completed the assessment material.

15. A method in accordance with claim 14, further comprising the step of merging the recorded score transferred to the main scoring center with the respondent identification to produce scoring reports.

16. A method of displaying images of handwritten nonobjective assessment materials to a human scorer and recording the scores recorded by the scorer, comprising the steps of:

capturing digitized images for a plurality of assessment materials;

storing the digitized images in a computer file;

displaying a first subset of the digitized images on a computer screen for scoring by the scorer to train the scorer;

displaying a second subset of the digitized images on a computer screen for scoring by the scorer to test the scorer'"'"'s selection of scores; and

displaying additional digitized images on a computer screen for scoring by the scorer to provide scores for the assessment materials represented by the additional images; and

recording in an electronic file scores entered by the scorer for the digitized images.

17. A method in accordance with claim 16, further comprising the steps of:

designating the first subset of the digitized images as anchor items to be used for training scorers;

assigning one or more valid scores for each of the anchor items;

said step of displaying a first subset comprising displaying the digitized images of the anchor items on a computer screen for scoring by the scorer as a training technique;

recording the score selected by the scorer for each anchor item;

comparing the score selected by the scorer to the values assigned as valid for each anchor item; and

informing the scorer as to the accuracy of the selected score for each anchor item.

18. A method in accordance with claim 16, further comprising the steps of:

designating the second subset of the digitized images as validity items to be used for testing the accuracy of the scorer'"'"'s selection of scores;

recording the score selected by the scorer for a validity item;

providing a control scoring center;

transferring the score selected by the scorer to the central scoring center;

comparing the score selected by the scorer to the values assigned as valid; and

determining the accuracy and reliability of the scorer'"'"'s performance on the set of validity items, to determine eligibility to score the assessment materials.

19. A method in accordance with claim 16, wherein the step of displaying the digitized image further comprises the steps of:

transferring to a scorer at a remote site, the file containing a set of images to be displayed and a companion data file so that the image can be located;

locating the image to be displayed;

reading the digitized image from the file; and

converting the image to a form appropriate for display on a computer screen.

20. An apparatus for displaying to a human scorer an image of handwritten assessment materials recorded by a test taker in an assessment area of an assessment form and for recording a score selected by the scorer, comprising:

input means for capturing from the assessment form a digitized image of the assessment materials to be scored, said means for capturing a digitized image comprising;

means for storing the digitized image in an electronic file;

means for selectively displaying the digitized image on a computer screen for the scoring by the scorer;

means for electronically recording a score entered by the scorer for the digitized image;

means for identifying the form by OMR;

means responsive to said means for identifying the form for determining the location of the assessment area;

means for determining by OMR the presence of marks within the assessment area of assessment materials recorded by the test taker and the portion of the assessment area in which marks are present; and

means for storing in digital electronic form as part of an image file an image of at least the portion of the assessment area in which assessment materials are present.

21. A method of displaying to a human scorer an image of handwritten nonobjective assessment materials recorded by a test taker in an assessment area of an assessment form, and recording a score entered by the scorer, comprising the steps of:

providing an assessment form having thereon at least one assessment area;

capturing a digitized image of said assessment materials recorded in at least one assessment area in digital form from the assessment form;

storing said digitized image in a computer file;

displaying said digitized image of the assessment materials on a computer screen for scoring by the scorer; and

recording a score entered by the scorer for the digitized image of the assessment materials;

said step of capturing a digitized image comprising the further steps of;

identifying the form by OMR;

determining the location of at least one assessment area on the form based on said step of identifying the form by OMR;

determining by OMR whether marks are present within said at least one assessment area;

determining by OMR the portion of said assessment area in which marks are present; and

storing in digital form the image of the portion of the assessment area in which it was determined that marks are present.

22. A method of displaying no a human scorer images of handwritten nonobjective assessment materials recorded in an assessment area of an assessment form and recording scores recorded by the scorer, comprising the steps of:

capturing digitized images for a plurality of assessment materials;

storing the digitized images in a computer file;

displaying a first subset of the digitized images on a computer screen for scoring by the scorer to train the scorer;

displaying a second subset of the digitized images on the a computer screen for scoring by the scorer to test the scorer'"'"'s selection of scores; and

displaying additional digitized images on the computer screen for scoring by the scorer to provide scores for the assessment materials represented by the additional images;

said step of capturing digitized images comprising the further steps of;

determining by OMR the location of the assessment area on the assessment form, the presence within the assessment area of assessment materials and the location within the assessment area of said assessment materials; and

storing in electronic form an image of at least said location within the assessment area in which said assessment materials are present.

23. An apparatus for electronically displaying to a human scorer an image of handwritten assessment materials recorded by a test taker in an assessment area of an assessment form and for recording a score selected by scorer, comprising:

an electronic file for storing identifying information to identify the test taker;

input means for capturing from the assessment form a digitized image of the assessment materials to be scored, said means for capturing a digitized image comprising;

an electronic image file for storing the digitized image;

means operatively connected to said image files for selectively displaying the digitized image but not the identifying information on a computer screen for scoring by the scorer;

means for electronically recording a score entered by the scorer for the digitized image;

said input means having means for identifying by electronic optical scanning the assessment form;

means for determining by electronic optical scanning the location of the assessment area on the assessment form, the presence within the assessment area of assessment materials recorded by the test taker, and the location within the assessment area of said assessment materials; and

means for storing in electronic form as part of said image file an image of at least said location within the assessment area in which said assessment materials recorded by the test taker are present.

24. Apparatus in accordance with claim 23, and a second image file, said second image file comprising means for electronically storing training items, said training items comprising data for simulating the appearance of assessment materials recorded by a test taker;

and

means for selectively displaying a digitized image of said training items on a computer screen for scoring by the scorer, whereby the scorer may be trained by scoring said training items.

25. Apparatus in accordance with claim 23, and an additional image file, said additional image file comprising means for electronically storing assessment items to be scored by a human scorer, said assessment items comprising a subset of previously scored assessment materials used to train the human scorer and having the appearance of assessment materials recorded by a test taker;

and

means for selectively displaying a digitized image of said assessment items on a computer screen, whereby the scoring by said scorer of said assessment items can be used to verify the accuracy of the scorer'"'"'s assignment of scores.