Methods for improving certainty of test-taker performance determinations for assessments with open-ended items
First Claim
1. A method of allocating resources to score a test-taker'"'"'s response to an open-ended item of an assessment comprising:
- a. estimating the test-taker'"'"'s performance on the assessment, wherein estimating the test-taker'"'"'s performance on the assessment comprises;
1. estimating the test-taker'"'"'s overall score for all items of the assessment;
2. utilizing one or more cutoff scores which separate the set of all possible overall scores into discrete non-overlapping ranges;
3. treating each of said ranges of scores as an indicator of a performance level on the assessment;
4. assigning to the test-taker the performance level based on the range of scores which includes the test-taker'"'"'s estimated overall score; and
5. using the assigned performance level as the estimate of the test-taker'"'"'s performance on the assessment;
b. establishing a specified level of certainty required for a final determination of the test-taker'"'"'s performance;
c. estimating a level of certainty for the estimated performance of the test-taker;
d. allocating an additional scoring resource to the test-taker'"'"'s response to at least one open-ended item when the estimated level of certainty fails to meet the specified level of certainty ande. accepting the assigned performance level as the final determination of the test-taker'"'"'s performance when the estimated level of certainty meets the specified level of certainty.
20 Assignments
0 Petitions
Accused Products
Abstract
The current invention is designed to improve the certainty of test-taker classifications in testing programs utilizing open-ended items. The current invention discloses methods to improve certainty in estimates of test-taker performance for all test-takers while using a minimum of scoring resources. The invention selectively allocates resources to scoring by ceasing to allocate scoring resources to test-takers whose performance has been determined with sufficient certainty, and by having an additional reader or readers score responses from test-takers whose scores are close to cutoff levels. The invention further increases precision of scores for test-takers near a cutoff by selectively allocating more reliable and accurate readers to score responses most likely to influence a test-taker'"'"'s performance classification. Other scoring resources are also shown.
20 Citations
35 Claims
-
1. A method of allocating resources to score a test-taker'"'"'s response to an open-ended item of an assessment comprising:
-
a. estimating the test-taker'"'"'s performance on the assessment, wherein estimating the test-taker'"'"'s performance on the assessment comprises; 1. estimating the test-taker'"'"'s overall score for all items of the assessment; 2. utilizing one or more cutoff scores which separate the set of all possible overall scores into discrete non-overlapping ranges; 3. treating each of said ranges of scores as an indicator of a performance level on the assessment; 4. assigning to the test-taker the performance level based on the range of scores which includes the test-taker'"'"'s estimated overall score; and 5. using the assigned performance level as the estimate of the test-taker'"'"'s performance on the assessment; b. establishing a specified level of certainty required for a final determination of the test-taker'"'"'s performance; c. estimating a level of certainty for the estimated performance of the test-taker; d. allocating an additional scoring resource to the test-taker'"'"'s response to at least one open-ended item when the estimated level of certainty fails to meet the specified level of certainty and e. accepting the assigned performance level as the final determination of the test-taker'"'"'s performance when the estimated level of certainty meets the specified level of certainty. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25)
-
-
26. A method of allocating resources to score a test-taker'"'"'s response to an open-ended item of an assessment including a plurality of open-ended items and multiple choice items comprising:
-
a. scoring the test-taker'"'"'s responses to the multiple choice items; b. scoring the test-taker'"'"'s responses to the open-ended items a first time; c. estimating a performance level of the test-taker based on the test-taker'"'"'s score on the multiple choice items and the test-taker'"'"'s first time score on the open-ended items; d. determining whether an adiustment to the first time score of one or more of the open-ended item responses could chanae the estimated performance level of the test taker; e. allocating scoring resources to only those of the test-taker'"'"'s responses to open-ended items, which if the first time score thereof were adjusted, could change the test-taker'"'"'s estimated performance level; and f. accepting the estimated performance level for the test-taker determined in the estimating step as a final performance level if it is determined in the determining step that adiusting the first time score of one or more of the open-ended item responses would not change the estimated performance level. - View Dependent Claims (27, 28, 29, 30, 31, 32, 33, 34, 35)
-
Specification