Constructed Response Scoring Mechanism

• US 20180225983A1
• Filed: 11/07/2017
• Published: 08/09/2018
• Est. Priority Date: 11/12/2008
• Status: Active Grant

• Current Assignee: Cambium Assessment, Inc.
• Original Assignee: American Institutes For Research
• Inventors: Cohen, Jon D., Dvorak, Joseph L., Albright, Larry E., Saba, Walid S.
• Granted Patent: US 10,643,489 B2
• Time in Patent Office: Days
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A system, method, and related techniques are disclosed for scoring user responses to constructed response test items. The system includes a scoring engine for receiving a user response to a test question and evaluating the response against a scoring rubric. The scoring rubric may include a binding stage, an assertion stage, and a scoring stage. Furthermore, the system includes a database for referencing elements used by the scoring engine which may comprise objects, object sets, attributes of objects, and transformations of any elements.

Claims

1. A system for scoring user responses to a test question, comprising:

a scoring engine for receiving a user response to a test question and evaluating the response against a scoring rubric, wherein the scoring rubric includes a binding stage, an assertion stage, and a scoring stage; and

a database for referencing elements used by the scoring engine which include at least one of objects, object sets, attributes of objects, and transformations of any elements.

2. The system of claim 1, wherein the user response includes one or more objects comprising at least one of points, line segments connecting points, geometric objects comprised of connected line segments, and user-defined atomic objects.

3. The system of claim 2, wherein the objects are characterized by an ordered set of points.

4. The system of claim 2, wherein the objects have properties including at least one of locations, names, labels, and values.

5. The system of claim 1, wherein the binding stage establishes at least one reference to one or more elements.

6. The system of claim 1, wherein at least one assertion about an element is evaluated and stored during the assertion stage.

7. The system of claim 6, wherein the at least one assertion is a predicate that is either true or false.

8. The system of claim 1, wherein the scoring stage for assigns a score to the response based on a result produced at the assertion stage.

9. The system of claim 1, wherein named assertions are collected in a set of And-Or trees.

10. The system of claim 9, wherein there is one tree for each numeric score point.

11. A method for scoring user responses to a test question, comprising:

receiving a user response wherein the response is a collection of objects with one or more attributes;

binding one or more variables identified in a binding tree based on the collection of objects;

evaluating at least one named assertion associated with an element; and

storing the named assertion.

12. The method of claim 11, further comprising commencing the evaluation of a scoring tree by starting with the highest possible score.

13. The method of claim 12, wherein the evaluation of the scoring tree is stopped when a subtree associated with a score evaluates to true.

14. A method of refining a scoring rubric, comprising:

receiving a plurality of user responses to a test item;

identifying a set of responses from the plurality of user responses;

providing a score for each of the identified responses;

generating one or more rules for differentiating a correct score from an incorrect score;

modifying a scoring rubric based on the one or more rules; and

applying the modified scoring rubric to multiple user responses.

15. The method of claim 14, wherein the set of responses are identified based on a random sample.

16. The method of claim 16, further comprising

identifying one or more scores of user responses that change using the modified scoring rubric;

and

reviewing a sample of the user responses and changed scores.

17. The method of claim 16, wherein the sample is a census.

18. The method of claim 16, further comprising making a decision to at least one of accept and reject the modified scoring rubric based on the review of the sample.

19. The method of claim 18, wherein the decision is based on the intentions of a third party.