Methods for improving certainty of test-taker performance determinations for assessments with open-ended items

• US 7,149,468 B2
• Filed: 07/25/2002
• Issued: 12/12/2006
• Est. Priority Date: 07/25/2002
• Status: Active Grant

• Current Assignee: McGraw Hill LLC
• Original Assignee: McGraw-Hill Companies Incorporated
• Inventors: Poor, David Deas Sinkler, Patz, Richard J.
• Primary Examiner(s): MOSSER, KATHLEEN MICHELE
• Application Number:
  US10/201,904
• Publication Number:
  US 20040018480A1
• Time in Patent Office: 1,601 Days
• Field of Search: 434/353, 434/362, 434/365, 434/118
• US Class Current: 434/353
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

The current invention is designed to improve the certainty of test-taker classifications in testing programs utilizing open-ended items. The current invention discloses methods to improve certainty in estimates of test-taker performance for all test-takers while using a minimum of scoring resources. The invention selectively allocates resources to scoring by ceasing to allocate scoring resources to test-takers whose performance has been determined with sufficient certainty, and by having an additional reader or readers score responses from test-takers whose scores are close to cutoff levels. The invention further increases precision of scores for test-takers near a cutoff by selectively allocating more reliable and accurate readers to score responses most likely to influence a test-taker'"'"'s performance classification. Other scoring resources are also shown.

Claims

1. A method of allocating resources to score a test-taker'"'"'s response to an open-ended item of an assessment comprising:

a. estimating the test-taker'"'"'s performance on the assessment, wherein estimating the test-taker'"'"'s performance on the assessment comprises;

1. estimating the test-taker'"'"'s overall score for all items of the assessment;

2. utilizing one or more cutoff scores which separate the set of all possible overall scores into discrete non-overlapping ranges;

3. treating each of said ranges of scores as an indicator of a performance level on the assessment;

4. assigning to the test-taker the performance level based on the range of scores which includes the test-taker'"'"'s estimated overall score; and

5. using the assigned performance level as the estimate of the test-taker'"'"'s performance on the assessment;

b. establishing a specified level of certainty required for a final determination of the test-taker'"'"'s performance;

c. estimating a level of certainty for the estimated performance of the test-taker;

d. allocating an additional scoring resource to the test-taker'"'"'s response to at least one open-ended item when the estimated level of certainty fails to meet the specified level of certainty and

e. accepting the assigned performance level as the final determination of the test-taker'"'"'s performance when the estimated level of certainty meets the specified level of certainty.

2. The method of claim 1, further comprising repeating steps (a) through (d) until the estimated level of certainty meets the specified level of certainty.

3. The method of claim 1, wherein the estimated level of certainty is deemed to fail to meet the specified level of certainty if the test-taker'"'"'s estimated assessment score is within a predefined range of scores around a cutoff score.

4. The method of claim 1, wherein estimating the level of certainty for the estimated performance of the test-taker comprises:

a. estimating a precision of the estimate of the test-taker'"'"'s assessment score; and

b. estimating the certainty of the estimate of the test-taker'"'"'s performance as a function of the estimated precision and the difference between the estimate of the test-taker'"'"'s assessment score and the cutoff score that is closest to the test-taker'"'"'s estimated assessment score.

5. The method of claim 1, wherein allocating a scoring resource comprises:

a. presenting the test-taker'"'"'s response to the open-ended item to a reader; and

b. recording a score assigned by the reader to the test-taker'"'"'s response to the open-ended item.

6. The method of claim 5, wherein allocating a scoring resource further comprises:

a. presenting the test-taker'"'"'s response to the open-ended item to one or more additional readers; and

b. recording a score assigned by the one or more additional readers to the test-taker'"'"'s response to the open-ended item.

7. The method of claim 6, wherein allocating a scoring resource further comprises:

a. identifying unresolved discrepant scores selected for the open-ended item response by the reader and the one or more additional readers; and

b. presenting the open-ended item response for which there are unresolved discrepant scores to a resolving reader in order to resolve the discrepancy.

8. The method of claim 7, wherein allocating a scoring resource further comprises:

a. determining characteristics of a set of available readers;

b. determining specific characteristics of a reader necessary to improve the estimated level of certainty for the estimated performance of the test-taker associated with the open-ended item response for which there are unresolved discrepant scores; and

c. selecting as the resolving reader a reader satisfying one or more of the specific characteristics to score the open-ended item response for which there are unresolved discrepant scores.

b. recording a score assigned by the one or more additional readers to the test-taker'"'"'s response to the open-ended item.

9. The method of claim 5, wherein selection of a reader for presentation of a particular open-ended item is based on scoring characteristics of the reader.

10. The method of claim 8, wherein said scoring characteristics include one or more characteristics selected from the group comprising:

(a) scoring accuracy, (b) scoring patterns, (c) the distribution of the quality of open-ended item responses assigned to a reader, and (d) the extent to which a reader'"'"'s scores have been reviewed by a lead reader.

11. The method of claim 10, wherein scoring accuracy is determined by one or more of monitoring items, lead reader review, and inter-reader agreement.

12. The method of claim 9, wherein selection of a reader further comprises:

a. determining scoring characteristics of a set of available readers;

b. determining specific scoring characteristics of a reader necessary to improve the estimated level of certainty for the estimated performance of the test-taker; and

c. selecting a reader satisfying one or more of the specific scoring characteristics determined in step (b) to score the test-taker'"'"'s response to the open-ended item.

13. The method of claim 5, wherein allocating a scoring resource further comprises:

a. estimating an overall distribution of the quality of open-ended item responses to be scored;

b. estimating a distribution of the quality of open-ended item responses presented to each reader to whom a specific open-ended item response may be assigned for scoring;

c. estimating the quality of the specific open-ended item response to be assigned to a reader; and

d. assigning the specific open-ended item response for scoring by a particular reader in order to lessen a difference between the estimated overall distribution of the quality of open-ended item responses to be scored and the estimated distribution of the quality of open-ended item responses presented to the particular reader.

14. The method of claim 5, wherein allocating a scoring resource further comprises:

a. identifying one or more scores having a precision to which the failure of the estimated level of certainty to meet the specified level of certainty may be attributed;

b. presenting to a lead reader the one or more open-ended item responses identified in step (a); and

c. recording a score selected by the lead reader for the one or more open-ended item responses presented to the lead reader.

15. The method of claim 14, wherein identifying the one or more scores comprises considering, as a characteristic of a previous reader, the extent to which scores selected by the previous reader have been reviewed by a lead reader.

16. The method of claim 1, wherein allocating a scoring resource comprises:

a. determining, for at least one of the test-taker'"'"'s open-ended item responses, possible scores that could result from presenting that open-ended item response for scoring to a reader;

b. estimating the probability of each possible score that could result by presenting that open-ended item response to the reader;

c. determining the effect that each possible score on that open-ended item response would have on the estimate of the test-taker'"'"'s performance; and

d. presenting the test-taker'"'"'s response to the open-ended item to the reader if the probability of a possible score that would affect the estimated performance is within a pre-defined threshold.

17. The method of claim 1, wherein said assessment includes multiple open-ended items, and allocating a scoring resource comprises:

a. estimating an increase in the level of certainty for the estimated performance of the test-taker that would be realized from scoring each of the open-ended item responses; and

b. allocating the scoring resource to the open-ended item response estimated to yield the largest increase in the level of certainty for the estimated performance of the test-taker.

18. The method of claim 1, wherein said assessment includes multiple-choice items, and estimating the test-taker'"'"'s performance on the assessment comprises determining the test-taker'"'"'s score on the multiple-choice items.

19. The method of claim 1, wherein estimating the test-taker'"'"'s performance on the assessment is based on one or more items of information selected from the items of information comprising:

(a) previously recorded scores to open-ended assessment items;

(b) teacher ratings of the test-taker; and

the (c) performance exhibited by the test-taker on one or more prior assessments.

20. The method of claim 1, wherein estimating the level of certainty comprises:

a. determining, for at least one open-ended item response, possible scores that could result from the allocation of one or more scoring resources to that open-ended item response;

b. estimating the probability of each possible score that could result on the open-ended item response; and

c. determining the effect that each possible score on that open-ended item response would have on the test-taker'"'"'s estimated performance level.

21. The method of claim 20, further comprising allocating a scoring resource to an open-ended item response if the probability of a possible score that would affect the estimated performance level is within a pre-defined threshold.

22. The method of claim 20, wherein allocating a scoring resource comprises:

a. presenting the test-taker'"'"'s response to the open-ended item to a reader; and

b. recording a score assigned by the reader to the test-taker'"'"'s response to the open-ended item, wherein selection of the reader for presentation of a particular open-ended item is based on scoring characteristics of the reader and the probability of a score that would affect the test-taker'"'"'s estimated performance.

23. The method of claim 1, wherein said scoring resource comprises allocating computational resources to at least one of estimating the test-taker'"'"'s performance on the assessment and estimating the level of certainty for the estimated performance of the test-taker.

24. The method of claim 23, wherein said computational resource for estimating the test-taker'"'"'s performance on the assessment comprises:

a. calculating the test-taker'"'"'s total score including scores achieved by the test-taker on the open-ended items;

b. estimating the standard error of the scores achieved by the test-taker on the open-ended items; and

c. estimating the likely range of the test-taker'"'"'s total score based on the standard error and the total score determined in step (a).

25. The method of claim 23, wherein said computational resource for estimating the level of certainty for the estimated performance comprises:

a. calculating the test-taker'"'"'s total score including scores achieved by the test-taker on the open-ended items;

b. estimating the standard error of the scores achieved by the test-taker on the open-ended items;

c. estimating the likely range of the test-taker'"'"'s total score based on the standard error and the total score determined in step (a); and

d. determining if the test-taker'"'"'s total score determined in step (a) is near a cutoff score separating one performance level from a next performance level relative to the likely range of the test-taker'"'"'s total score.

26. A method of allocating resources to score a test-taker'"'"'s response to an open-ended item of an assessment including a plurality of open-ended items and multiple choice items comprising:

a. scoring the test-taker'"'"'s responses to the multiple choice items;

b. scoring the test-taker'"'"'s responses to the open-ended items a first time;

c. estimating a performance level of the test-taker based on the test-taker'"'"'s score on the multiple choice items and the test-taker'"'"'s first time score on the open-ended items;

d. determining whether an adiustment to the first time score of one or more of the open-ended item responses could chanae the estimated performance level of the test taker;

e. allocating scoring resources to only those of the test-taker'"'"'s responses to open-ended items, which if the first time score thereof were adjusted, could change the test-taker'"'"'s estimated performance level; and

f. accepting the estimated performance level for the test-taker determined in the estimating step as a final performance level if it is determined in the determining step that adiusting the first time score of one or more of the open-ended item responses would not change the estimated performance level.

27. The method of claim 26, wherein estimating the performance level of the test-taker comprises:

a. utilizing one or more cutoff scores which separate the set of all possible scores into discrete non-overlapping ranges;

b. treating each of said ranges of scores as an indicator of a performance level on the assessment;

c. assigning to the test-taker the performance level based on the range of scores which includes the test-taker'"'"'s overall score as determined by the test-taker'"'"'s score on the multiple choice items and the test-taker'"'"'s first time score on the open-ended items; and

d. using the assigned performance level as the estimate of the test-taker'"'"'s performance on the assessment.

28. The method of claim 26, wherein allocating scoring resources comprises:

a. presenting the test-taker'"'"'s response to the open-ended item to a reader; and

b. recording a score assigned by the reader to the test-taker'"'"'s response to the open-ended item.

29. The method of claim 28, wherein selection of a reader for presentation of a particular open-ended item is based on scoring characteristics of the reader.

30. The method of claim 29, wherein selection of a reader further comprises:

a. determining scoring characteristics of a set of available readers;

b. determining specific scoring characteristics of a reader necessary to improve the precision of the score assigned to the open-ended item response; and

c. selecting a reader satisfying one or more of the specific scoring characteristics determined in step (b) to score the test-taker'"'"'s response to the open-ended item.

31. The method of claim 28, wherein allocating a scoring resource further comprises:

a. identifying unresolved discrepancies between scores selected for the open-ended item response by the reader and the first time score of the open-ended item response; and

b. presenting the open-ended item response for which there are unresolved discrepant scores to a resolving reader in order to resolve the discrepancy.

32. The method of claim 26, wherein said scoring resources comprise allocating computational resources to estimating the performance level of the test-taker on the assessment.

33. The method of claim 32, wherein said computational resource for estimating the performance level of the test-taker on the assessment comprises:

a. calculating the test-taker'"'"'s total score including first time scores achieved by the test-taker on the open-ended items;

b. estimating the standard error of the first time scores achieved by the test-taker on the open-ended items; and

c. estimating the likely range of the test-taker'"'"'s total score based on the standard error and the total score determined in step (a).

34. The method of claim 26, wherein allocating scoring resources comprises:

a. determining, for at least one of the test-taker'"'"'s open-ended item responses, possible scores that could result from presenting that open-ended item response for scoring to a reader;

b. estimating the probability of each possible score that could result by presenting that open-ended item response to the reader;

c. determining the effect that each possible score on that open-ended item response would have on the estimate of the test-taker'"'"'s performance; and

d. presenting the test-taker'"'"'s response to the open-ended item to the reader if the probability of a possible score that would affect the estimated performance is within a pre-defined threshold.

35. The method of claim 26, further comprising:

a. calculating the test-taker'"'"'s total score from the score achieved by the test-taker on the multiple choice items and the first time score achieved by the test-taker on the open-ended items;

b. estimating the standard error of the first time scores achieved by the test-taker on the open-ended items;

c. estimating the likely range of the test-taker'"'"'s total score based on the standard error and the total score determined in step (a); and

d. allocating scoring resources if the test-taker'"'"'s total score determined in step (a) is near a cutoff score separating one performance level from a next performance level relative to the likely range of the test-taker'"'"'s total score.