Method for scoring and control of scoring open-ended assessments using scorers in diverse locations

• US 5,987,149 A
• Filed: 03/07/1996
• Issued: 11/16/1999
• Est. Priority Date: 07/08/1992
• Status: Expired due to Term

• Current Assignee: NCS Pearson Incorporated (Pearson plc)
• Original Assignee: Uniscore, Inc. (S&P Global Inc.)
• Inventors: Poor, David D. S.
• Primary Examiner(s): Chang, Jon
• Assistant Examiner(s): Patel, Jayanti K.
• Application Number:
  US08/610,584
• Time in Patent Office: 1,349 Days
• Field of Search: 382/100, 382/305, 434/322-325, 434/332-336, 434/339-341, 395/221, 395/244, 395/608, 395/615, 395/766
• US Class Current: 382/100
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G09B 7/066 with answer indicating cards, blocks

Abstract

Apparatus and methods are disclosed for scoring nonobjective assessment materials by displaying images of the assessment materials or of text in lieu of images on a computer screen, for scoring by a human scorer. A digitized image of the assessment material may be captured by scanning an assessment form and storing the image in an electronic file. The stored digitized image or other data is displayed to the scorer and a score entered by the scorer is stored. In disclosed embodiments, the apparatus provides for scoring by multiple scorers on a network located at a single, central site; scoring by multiple scorers on a network remote from the central site; and scoring by one or more individual scorers remote from the central site. Apparatus and methods are disclosed for controlling and controlling access to assessment materials; for matching the qualifications of the scorers to the assessment materials made available to them for scoring; and for evaluating the performance of the scorers. In remote scoring in accordance with the invention, data from various files may selectively be transferred from central to remote sites, using diverse techniques in accordance with the needs of the particular job and setting.

Claims

1. A method of electronically scoring nonobjective assessment materials recorded by test takers, comprising the steps of:

creating digitized electronic representations of the nonobjective assessment materials;

storing the digitized representations in a file;

creating and maintaining a file of data that relates each assessment material item to the test taker who recorded the material;

creating and maintaining a file of data that identifies a group of human scorers who are authorized to request the assessment materials and the qualifications of each of the group of human scorers to score assessment materials;

providing a means for a scorer in the group of human scorers to request assessment materials to be displayed on a monitor;

processing a request by the scorer for the display of assessment materials by selecting items of assessment material for the scorer to score by referring to the file data identifying the scorer and signifying the qualifications of the scorer, and selecting in response to the request only assessment material items which the scorer is authorized and qualified to score;

displaying the selected assessment material to the scorer, without displaying any information identifying the test taker who recorded the material;

providing the particular scorer a means to electronically record scores for assessment material; and

electronically storing scores recorded by the particular scorer for the assessment material.

2. A method as in claim 1, wherein the file storing the digitized representations is maintained at a control site, the file of data that relates each assessment material item to the test taker who recorded the material is maintained at the control site, and the scorer is located at a site remote from the control site.

3. A method as in claim 1, wherein the file of data that identifies human scorers and the qualifications of each scorer is maintained at a control site, and the scorer is located at a site remote from the control site.

4. A method as in claim 3, wherein the file storing the digitized representations is maintained at the control site, and the file of data that relates each assessment material item to the test taker who recorded the material is maintained at the control site.

5. A method as in claim 1, having further steps to monitor the validity of scores, comprising:

pre-scoring a set of assessment items as validity monitoring items;

storing the set of validity monitoring items, and the validation scores associated with each item, in a file;

in response to a scorer request for the display of assessment materials, creating a scorer batch file that includes regular assessment items and one or more validity monitoring items;

displaying items from the scorer batch file to the scorer without identifying any item as a validity monitoring item; and

, when scoring of the batch is completed,

electronically comparing the scores entered by the scorer for each validity monitoring item to the stored validation scores associated with the same item to determine whether the scores entered by the scorer fall within a pre-determined validation range.

6. A method as in claim 5, wherein the file storing the set of validity monitoring items and the validation scores associated with each item is located at a control site, and the scorer is located at a site remote from the control site.

7. A method as in claim 5, wherein the file storing the digitized representations is maintained at the control site and the file of data that relates each assessment material item to the test taker who recorded the material is maintained at the control site.

8. A method as in claim 1, having further steps to monitor the validity of scores, comprising:

pre-scoring a set of assessment items as validity monitoring items;

storing the set of validity monitoring items, and the validation scores associated with each item, in a file;

in response to a scorer request for the display of assessment materials, creating a scorer batch file that includes regular assessment items and one or more validity monitoring items;

displaying to the scorer assessment material items from the scorer batch file without identifying any item as a validity monitoring item; and

, when scoring of the batch is completed,

electronically comparing the scores entered by the scorer for the one or more validity monitoring items to the stored validation scores associated with the same items to determine whether the scores entered by the scorer fall within a pre-determined validation range.

9. A method as in claim 8, wherein the file storing the set of validity monitoring items and the validation scores associated with each item is located at a control site, and the scorer is located at a site remote from the control site.

10. A method as in claim 9, wherein the file storing the digitized representations is maintained at the control site, the file of data that relates each assessment material item to the test taker who recorded the material is maintained at the control site, and the file of data that identifies human scorers and the qualifications of each scorer is maintained at the control site.

11. A method as in claim 1 having further steps to train scorers to score assessment materials, comprising:

pre-scoring a set of assessment materials as training items to obtain validated scores for the training items;

storing the set of training items, and the validated scores associated with each, in a training file at the control site;

providing scorers a means to request the display of assessment material training items from the training file;

displaying to the scorer training items from the training file;

electronically storing scores recorded by the scorer for the training items; and

providing feedback to the scorer on the relationship of his or her scores to the validated scores.

12. A method as in claim 11, wherein the file storing the training items and the validated score associated with each training item is located at a control site, and the scorer is located at a site remote from the control site.

13. A method as in claim 12, wherein the file storing the digitized representations is maintained at a control site, and the file of data that relates each assessment material item to the test taker who recorded the material is maintained at the control site.

14. A method as in claim 11, having further steps to establish qualifications of scorers, comprising:

providing scorers a means to request the display of a qualifying batch of assessment materials from the training file;

processing a request by a human scorer for the display of assessment materials by selecting items of assessment material from the training file to create a qualifying batch and displaying to the scorer a sequence of items from the qualifying batch;

electronically recording and storing scores determined by the scorer for the qualifying batch items;

comparing the scores entered by the scorer to the validated scores for the same materials; and

determining whether the scores falls within a pre-determined qualification range when compared to the validated scores.

15. A method as in claim 14, wherein the file storing the training items and the validated score associated with each training item is located at a control site, and the scorer is located at a site remote from the control site.

16. A method as in claim 15, wherein the file storing the digitized representations is maintained at a control site, and the file of data that relates each assessment material item to the test taker who recorded the material is maintained at the control site.

17. A method as in claim 1, having further steps to establish qualifications of scorers, comprising:

pre-scoring a set of assessment materials as qualifying items to obtain validated scores for the qualifying items;

storing the set of qualifying items, and the validated scores associated with each, in a qualifying file;

providing scorers a means to request the display of a qualifying batch of assessment materials;

processing a request by a human scorer for the display of assessment materials by selecting items of assessment material from the qualifying file to create a qualifying batch and displaying to the scorer a sequence of items from the qualifying batch;

electronically recording and storing scores determined by the scorer for the qualifying batch items displayed to that scorer;

comparing the scores entered by the scorer to the validated scores for the same materials; and

determining whether the scores falls within a pre-determined qualification range when compared to the validated scores.

18. A method as in claim 17, wherein the file storing the qualifying items and the validated score associated with each is located at a control site, and the scorer is located at a site remote from the control site.

19. A method as in claim 18, wherein the file storing the training items and the validated score associated with each training item is located at a control site, and the scorer is located at a site remote from the control site.

20. A method as in claim 17, having the further steps to establish qualifications of scorers of:

storing in association with the data identifying the qualifications of the scorer additional data signifying the scores determined by the scorer, whereby the data signifying the scores is made a part of the data identifying the qualifications of the scorer.