System and method of distribution of digitized materials and control of scoring for open-ended assessments

• US 7,054,464 B2
• Filed: 10/15/2002
• Issued: 05/30/2006
• Est. Priority Date: 07/08/1992
• Status: Expired due to Fees

• Current Assignee: NCS Pearson Incorporated (Pearson plc)
• Original Assignee: NCS Pearson Incorporated (Pearson plc)
• Inventors: Poor, David D. S.
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US10/272,469
• Publication Number:
  US 20030086586A1
• Time in Patent Office: 1,323 Days
• Field of Search: 382/100, 382/305, 382/306, 382/181, 358/462, 358/453, 235/435, 345/700, 345/701, 345703-730, 434112-116, 434/188, 434/322, 434/365
• US Class Current: 382/100
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G09B 7/066 with answer indicating cards, blocks

Abstract

Methods are disclosed for distributing digitized electronic representations of representative responses to non-objective assessment tasks and established scores for the responses to readers in order to train the readers to determine scores for responses in accordance with a scoring criteria, to qualify the readers to determine scores for responses in accordance with the scoring criteria, and to monitor how well a reader is determining scores for such responses in accordance with the scoring criteria. In disclosed embodiments, the methods provide such distribution for multiple scorers on a network located at a single site, for multiple scorers on a network remote from a central site; and scoring by individual scorers remote from the central site. Distribution may be accomplished by transferring data from electronic files using diverse techniques such as magnetic storage discs, CD-ROM, cable or high-speed public communications network, in accordance with the needs of the particular job and setting.

Claims

1. A method of electronically displaying to a human scorer an image of handwritten nonobjective assessment materials recorded by a test taker in an assessment area of an assessment form, and recording a score entered by the scorer, comprising the steps of:

providing an assessment form having thereon at least one assessment area;

recording in an electronic file information identifying the test taker including at least one of the following;

the test taker'"'"'s name, the test taker'"'"'s gender, and the test taker'"'"'s grade level;

recording in an electronic file information identifying the assessment form;

capturing from the assessment form a digitized image of at least a portion of said assessment materials recorded in the at least one assessment area;

storing said digitized image in an electronic image file;

recording in an electronic file information to locate said stored image of said assessment materials;

selectively displaying electronically said stored image of said assessment materials on a computer screen for scoring by the scorer, but not displaying the test taker'"'"'s name, the test taker'"'"'s gender, or the test taker'"'"'s grade level; and

recording in an electronic file a score entered by the scorer for the stored digitized image of the assessment materials.

2. The method of claim 1 wherein a test taker code is displayed on the computer screen, the test taker code being selected so that the scorer is unable to deduce information about the identity of the test taker from the code.

3. The method of claim 2 wherein the code is a number.

4. A method of electronically displaying to a human scorer an image of handwritten nonobjective assessment materials recorded by a test taker in an assessment area of an assessment form, and recording a score entered by the scorer, comprising the steps of:

providing an assessment form having thereon at least one assessment area;

recording in an electronic file the test taker'"'"'s name;

recording in an electronic file information identifying the assessment form;

capturing from the assessment form a digitized image of said assessment materials recorded in the at least one assessment area;

storing said digitized image in an electronic image file;

recording in an electronic file information to locate said stored image of said assessment materials;

selectively displaying electronically said stored image of said assessment materials on a computer screen for scoring by the scorer, but not displaying the test taker'"'"'s name; and

recording in an electronic file a score entered by the scorer for the stored digitized image of the assessment materials.

5. A method of electronically scoring nonobjective assessment materials recorded by test takers, comprising the steps of:

creating digitized electronic representations of the nonobjective assessment materials, the nonobjective assessment materials including a plurality of assessment items;

storing the digitized representations in a file;

creating and maintaining a file of data that relates each assessment material item to the test taker who recorded the material;

creating and maintaining a file of data that identifies;

a group of human scorers who are authorized to request the assessment materials; and

the qualifications of each of the group of human scorers to score assessment materials;

providing a means for a scorer in the group of human scorers to request assessment materials to be displayed on a monitor;

processing a request by the scorer for the display of assessment materials by selecting items of assessment material for the scorer to score by;

referring to the file data that identifies the scorer and the qualifications of the scorer, and

selecting in response to the request only assessment material items which the scorer is authorized and qualified to score;

displaying the selected assessment material to the scorer, without displaying the test taker'"'"'s name, gender, or educational grade level;

providing the scorer a means to electronically record scores for assessment material; and

electronically storing scores recorded by the scorer for the assessment material.

6. A computer-readable medium embodying code for electronically scoring nonobjective assessment materials recorded by test takers by performing the steps of:

creating digitized electronic representations of the nonobjective assessment materials, the nonobjective assessment materials including a plurality of assessment items;

storing the digitized representations in a file;

creating and maintaining a file of data that relates each assessment material item to the test taker who recorded the material;

creating and maintaining a file of data that identifies;

a group of human scorers who are authorized to request the assessment materials and

the qualifications of each of the group of human scorers to score assessment materials;

providing a means for a scorer in the group of human scorers to request assessment materials to be displayed on a monitor;

processing a request by the scorer for the display of assessment materials by selecting items of assessment material for the scorer to score by;

referring to the file data that identifies the scorer and the qualifications of the scorer; and

selecting in response to the request only assessment material items which the scorer is authorized and qualified to score;

displaying the selected assessment material to the scorer, without displaying the test taker'"'"'s name, gender, or educational grade level;

providing the scorer a means to electronically record scores for assessment material; and

electronically storing scores recorded by the scorer for the assessment material.

7. A method of electronically displaying to a human scorer a nonobjective assessment material authored by a test taker in response to an assessment form and recording a score entered by the scorer, comprising the steps of:

electronically receiving from a test taker an assessment material comprising character strings of text authored by the test taker;

recording the assessment material in an electronic assessment file;

recording in an electronic file information identifying the test taker;

recording in an electronic file information identifying the assessment form;

selectively displaying on a computer screen the assessment material for scoring by a scorer and a code corresponding to the test taker, but not displaying said information identifying the test taker, whereby the scorer is prevented from considering information identifying the test taker; and

recording in an electronic file a score entered by the scorer for the stored digitized image of the assessment materials.

8. A method of electronically displaying to a human scorer a nonobjective assessment material authored by a test taker in response to an assessment form and recording a score entered by the scorer, comprising the steps of:

scanning a plurality of pages of responses to non-objective assessment items to produce a plurality of scanned images, each scanned image including at least one response to a nonobjective assessment item;

loading the scanned images into a processing system;

presenting the scorer with at least a portion of at least one of the scanned images, the portion of the scanned image including at least one response to a nonobjective assessment item, the portion of the scanned image further including coded information corresponding to the test taker, the coded information being provided in a manner such that the scorer cannot deduce the identity of the test taker, so as to preserve the test taker'"'"'s anonymity;

receiving a score from the scorer;

recording the score from the scorer.

9. A system for electronically displaying to a human scorer a nonobjective assessment material authored by a test taker in response to an assessment form and recording a score entered by the scorer, comprising:

a scanning system configured to scan a plurality of pages of responses to nonobjective assessment items to produce a plurality of scanned images, each scanned image including at least one response to a non-objective assessment item;

a processing system configured to receive and store the scanned images;

a presenting system configured to present the scorer with at least a portion of at least one of the scanned images, the portion of the scanned image including at least one response to a nonobjective assessment item, the portion of the scanned image further including information identifying the test taker that is provided in a manner such that the scorer cannot deduce the identity of the test taker;

a receiving system configured to receive a score from the scorer;

a recording system configured to record the score from the scorer.

10. The system of claim 9 wherein the processing system comprises the receiving system.

11. The system of claim 9 wherein the processing system comprises the recording system.

12. A system for electronically displaying to a human scorer a nonobjective assessment material authored by a test taker in response to an assessment form and recording a score entered by the scorer, comprising:

means for scanning a plurality of pages of responses to non-objective assessment items to produce a plurality of scanned images, each scanned image including at least one response to a nonobjective assessment item;

means for receiving the scanned images;

means for storing the scanned images;

means for presenting the scorer with at least a portion of at least one of the scanned images, the portion of the scanned image including at least one response to a nonobjective assessment item, the portion of the scanned image further including information corresponding to the test taker that is provided in a manner such that the scorer cannot deduce the identity of the test taker;

means for receiving a score from the scorer;

means for recording the score from the scorer.

13. A computer-readable medium embodying code for electronically displaying to a human scorer a nonobjective assessment material authored by a test taker in response to an assessment form and recording a score entered by the scorer by performing the steps of:

scanning a plurality of pages of responses to non-objective assessment items to produce a plurality of scanned images, each scanned image including at least one response to a nonobjective assessment item;

loading the scanned images into a processing system;

presenting the scorer with at least a portion of at least one of the scanned images, the portion of the scanned image including at least one response to a nonobjective assessment item, the portion of the scanned image further including coded information corresponding to the test taker, the coded information being provided in a manner such that the scorer cannot deduce the identity of the test taker, so as to preserve the test taker'"'"'s anonymity;

receiving a score from the scorer;

recording the score from the scorer.