Methods for range finding of open-ended assessments

• US 6,577,846 B2
• Filed: 02/12/2001
• Issued: 06/10/2003
• Est. Priority Date: 02/12/2001
• Status: Expired due to Term

• Current Assignee: McGraw Hill LLC
• Original Assignee: MHE Criterion LLC
• Inventors: Poor, David D. S.
• Primary Examiner(s): Cheng, Joe H.
• Assistant Examiner(s): MOSSER, KATHLEEN MICHELE
• Application Number:
  US09/781,334
• Publication Number:
  US 20020110797A1
• Time in Patent Office: 848 Days
• Field of Search: 434/322, 434/323, 434/350, 434/353, 434/362, 434/356, 434/365, 434/307.R, 434/169
• US Class Current: 434/353
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

The current invention discloses methods to expedite and simplify a range finding process with resulting savings in time and cost. The invention utilizes digitized representations of respondent answers instead of documents containing the student responses. The invention uses methods of using performance indicators other than the response itself to identify a targeted subset of individual student responses within the sample which are likely to be exemplars of any given score point. In the preferred implementation of this invention, scanned images or other digitized representations of the student responses, and electronically recorded performance indicators are used so that a computer system can quickly and efficiently identify and present the targeted subset to the expert readers of a range finding team. The invention allows expanding the sample of responses available to range finding, up to the entire test administration.

Claims

1. A method of presenting to an expert reader a digitized representation of a test taker'"'"'s response to an open-ended assessment for the purpose of enabling the reader to identify exemplar responses for range finding, the method comprising the steps of:

(a) electronically recording a digitized representation of the test taker'"'"'s response to the open-ended assessment;

(b) identifying an indicator other than the response as a predictor of whether the representation is likely to be graded within a limited range of score points for the assessment;

(c) linking data related to the indicator to the digital representation of the response;

(d) selecting the response for display to the expert reader only if the indicator is associated with the response; and

(e) electronically displaying the digitized representation to an expert reader for evaluation as a possible exemplar response.

2. The method of claim 1, in the event that the expert reader evaluates the digitized representation to be an exemplar response, comprising the further steps of:

(f) electronically recording the representation in a set of digital representations of responses for training readers;

(g) recording a score point for which the representation is an exemplar; and

(h) linking the score point to the recorded representation.

3. A method of presenting to an expert reader a digitized representation of a test taker'"'"'s response to an open-ended assessment for the purpose of enabling the reader to identify exemplar responses for range finding, the method comprising the steps of:

(a) electronically recording a digitized representation of the test taker'"'"'s response to the open-ended assessment;

(b) identifying an indicator other than the response as a predictor of the likely range of score points in which the response will be graded;

(c) linking data related to the indicator to the digital representation of the response;

(d) sequencing the digital representation for display to the reader in an order of representations based upon its relative ranking to other representations with respect to the indicator; and

(e) electronically displaying the digitized representations to an expert reader for evaluation as a possible exemplar response in the sequence as determined.

4. The method of claim 3, in the event that the expert reader evaluates the digitized representation to be an exemplar response, comprising the further steps of:

(f) electronically recording the representation in a set of digital representations of responses for training readers;

(g) recording a score point for which the representation is an exemplar; and

(h) linking the score point to the recorded representation.

5. A method of presenting to an expert reader a set of digitized representations of respondent answers to an open ended assessment for range finding, the set being targeted to identifying exemplars of answers within a limited range of score points in the assessment, the method comprising the steps of:

(a) electronically recording a digitized representation of the response of each of a plurality test takers of the open-ended assessment;

(b) selecting a performance indicator for test takers, other than the response, as a predictor of whether a response is likely to be graded within the limited range of score points;

(c) for each of such plurality of test takers, electronically recording data related to his or her performance relative to the performance indicator;

(d) for each of such plurality test takers, linking the test taker'"'"'s performance indicator data with the representation of his or her response;

(e) identifying by the performance indicator those responses that are likely to be graded within the limited range of score points;

(f) creating a set of responses from those identified as likely to be graded within the limited range of score points; and

(g) electronically displaying images from the set to the expert reader for evaluation as a possible exemplar response.

6. A method as in claim 5, wherein the indicator is derived from data from the group consisting of:

score on answers to objective test items in the same assessment, score on other assessments;

school attended, school placement section, academic performance, teacher performance rating; and

student performance rating.

7. A method as in claim 5, wherein the indicator is whether the test taker'"'"'s score on answers to objective test items in the same assessment is within a selected range of scores.

8. A method as in claim 7, wherein the digitized representation is a scanned image of response and the test taker'"'"'s score on answers to objective test items is determined by OMR.

9. A method as in claim 5, wherein the representations are displayed from the set in a sequence ranked by the indicator.

10. A method as in claim 5, wherein if the expert reader evaluates the digitized representation to be an exemplar response, comprising the further steps of:

(c) electronically recording the representation in a set of digital representations of responses for training readers;

(d) recording a score point for which the representation is an exemplar;

(e) linking the score point to the recorded representation.