Automated Scoring Using an Item-Specific Grammar

• US 20140295387A1
• Filed: 03/27/2014
• Published: 10/02/2014
• Est. Priority Date: 03/27/2013
• Status: Active Application

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Heilman, Michael, Blanchard, Daniel
• Application Number:
  US14/227,181
• Publication Number:
  US 20140295387A1
• Time in Patent Office: Days
• US Class Current: 434/169
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

Systems and methods are provided for scoring a constructed response. The constructed response is processed according to a set of grammar rules to generate a data structure. The grammar rules specify a set of preferred responses for the item. The grammar rules utilize a plurality of variables that specify legitimate word patterns for the constructed response. It is determined whether the data structure indicates that the constructed response is included in the set of preferred responses, and if so, a maximum score is assigned to the constructed response. If the data structure indicates that the constructed response is not included in the set of preferred responses, a partial credit score for the constructed response is determined by assessing from the data structure which ones of the concepts represented by the variables are present in the constructed response. The partial credit score is assigned based on the presence of the concepts.

Claims

1. A computer-implemented method for scoring a constructed response, the computer-implemented method comprising:

receiving a constructed response for an item;

processing the constructed response with a processing system according to a set of grammar rules to generate a data structure for use in scoring the constructed response, the grammar rules specifying a set of preferred responses for the item, each preferred response meriting a maximum score for the item, the grammar rules utilizing a plurality of variables that specify legitimate word patterns for the constructed response,

wherein the data structure comprises information regarding i) whether the constructed response is included in the set of preferred responses, and ii) for each of the variables, whether a concept represented by the variable is present in the constructed response;

determining with the processing system, based on the information included in the data structure, whether the constructed response is included in the set of preferred responses with the processing system, and if so, assigning the maximum score to the constructed response; and

if the constructed response is not included in the set of preferred responses, determining with the processing system a partial credit score for the constructed response by assessing from the data structure which ones of the concepts are present in the constructed response, and assigning the partial credit score based on the presence of the concepts.

2. The computer-implemented method of claim 1, wherein the grammar rules comprise production rules of a context-free grammar or a feature-based grammar.

3. The computer-implemented method of claim 1, wherein the plurality of variables include a low-score variable, and wherein the constructed response is assigned a lowest partial credit score if the concept represented by the low-score variable is present in the constructed response.

4. The computer-implemented method of claim 1, wherein the grammar rules utilize a second plurality of variables, wherein the plurality of variables is a subset of the second plurality of variables, and wherein the plurality of variables and the second plurality of variables are non-terminal symbols defined by grammar rules of the set of grammar rules.

5. The computer-implemented method of claim 1, wherein each of the concepts is a phrase or a sentence.

6. The computer-implemented method of claim 1, wherein the data structure indicates that the constructed response is included in the set of preferred responses if the constructed response parses completely according to the set of grammar rules.

7. The computer-implemented method of claim 1, wherein the partial credit score is determined based on a number of the concepts that are present in the constructed response.

8. A system for scoring a constructed response, the system comprising:

a processing system; and

a memory in communication with the processing system, wherein the processing system is configured to execute steps comprising;

receiving a constructed response for an item;

processing the constructed response according to a set of grammar rules to generate a data structure for use in scoring the constructed response, the grammar rules specifying a set of preferred responses for the item, each preferred response meriting a maximum score for the item, the grammar rules utilizing a plurality of variables that specify legitimate word patterns for the constructed response,

wherein the data structure comprises information regarding i) whether the constructed response is included in the set of preferred responses, and ii) for each of the variables, whether a concept represented by the variable is present in the constructed response;

determining, based on the information included in the data structure, whether the data the constructed response is included in the set of preferred responses with the processing system, and if so, assigning the maximum score to the constructed response; and

if the constructed response is not included in the set of preferred responses, determining with the processing system a partial credit score for the constructed response by assessing from the data structure which ones of the concepts are present in the constructed response, and assigning the partial credit score based on the presence of the concepts.

9. The system of claim 8, wherein the grammar rules comprise production rules of a context-free grammar or a feature-based grammar.

10. The system of claim 8, wherein the plurality of variables include a low-score variable, and wherein the constructed response is assigned a lowest partial credit score if the concept represented by the low-score variable is present in the constructed response.

11. The system of claim 8, wherein the grammar rules utilize a second plurality of variables, wherein the plurality of variables is a subset of the second plurality of variables, and wherein the plurality of variables and the second plurality of variables are non-terminal symbols defined by grammar rules of the set of grammar rules.

12. The system of claim 8, wherein each of the concepts is a phrase or a sentence.

13. The system of claim 8, wherein the data structure indicates that the constructed response is included in the set of preferred responses if the constructed response parses completely according to the set of grammar rules.

14. The system of claim 8, wherein the partial credit score is determined based on a number of the concepts that are present in the constructed response.

15. A non-transitory computer-readable storage medium for scoring a constructed response, the computer-readable storage medium comprising computer executable instructions which, when executed, cause a processing system to execute steps comprising:

receiving a constructed response for an item;

processing the constructed response according to a set of grammar rules to generate a data structure for use in scoring the constructed response, the grammar rules specifying a set of preferred responses for the item, each preferred response meriting a maximum score for the item, the grammar rules utilizing a plurality of variables that specify legitimate word patterns for the constructed response,

wherein the data structure comprises information regarding i) whether the constructed response is included in the set of preferred responses, and ii) for each of the variables, whether a concept represented by the variable is present in the constructed response;

determining, based on the information included in the data structure, whether the constructed response is included in the set of preferred responses with the processing system, and if so, assigning the maximum score to the constructed response; and

if the constructed response is not included in the set of preferred responses, determining with the processing system a partial credit score for the constructed response by assessing from the data structure which ones of the concepts are present in the constructed response, and assigning the partial credit score based on the presence of the concepts.

16. The non-transitory computer-readable storage medium of claim 15, wherein the grammar rules comprise production rules of a context-free grammar or a feature-based grammar.

17. The non-transitory computer-readable storage medium of claim 15, wherein the plurality of variables include a low-score variable, and wherein the constructed response is assigned a lowest partial credit score if the concept represented by the low-score variable is present in the constructed response.

18. The non-transitory computer-readable storage medium of claim 15, wherein the grammar rules utilize a second plurality of variables, wherein the plurality of variables is a subset of the second plurality of variables, and wherein the plurality of variables and the second plurality of variables are non-terminal symbols defined by grammar rules of the set of grammar rules.

19. The non-transitory computer-readable storage medium of claim 15, wherein each of the concepts is a phrase or a sentence.

20. The non-transitory computer-readable storage medium of claim 15, wherein the data structure indicates that the constructed response is included in the set of preferred responses if the constructed response parses completely according to the set of grammar rules.

21. The non-transitory computer-readable storage medium of claim 15, wherein the partial credit score is determined based on a number of the concepts that are present in the constructed response.