System for rating constructed responses based on concepts and a model answer

• US 8,380,491 B2
• Filed: 04/19/2002
• Issued: 02/19/2013
• Est. Priority Date: 04/19/2002
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Leacock, Claudia, Chodorow, Martin, Bolge, Eleanor, Wolska, Magdalena
• Primary Examiner(s): YEN, ERIC L
• Application Number:
  US10/125,440
• Publication Number:
  US 20030200077A1
• Time in Patent Office: 3,959 Days
• Field of Search: 434/335, 434/353, 704/1, 704/2, 704/4, 704/7, 704/8, 704/9
• US Class Current: 704/9
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

A concept rater module is utilized to automatically grade or score constructed responses based on a model answer. The concept rater module may be configured to accept a model answer as input. The model answer may be used as a grading key by the concept rater module. The concept rater module may be further configured to accept student responses in a file format. The file format may be ASCII text, a formatted word processing (e.g., WORDPERFECT, MICROSOFT WORD, etc.) and the like. The concept rater module may be further configured to process a student response into a canonical representation of the student response. The canonical representation of the student response is compared against the model answer by the concept rater module. From the comparison, a score is generated which represents that student'"'"'s ability to cover all the key concepts.

Claims

1. A method for grading constructed responses, said method comprising:

converting, using a processing system, a model answer into a tuple representation of the model answer including at least one topic with each topic including at least one concept, the tuple representation of the model answer comprising a predicate-argument data structure;

converting, using the processing system, a constructed response into a tuple representation of the constructed response;

comparing, using the processing system, said tuple representation of the model answer and said tuple representation of the constructed response; and

generating, using the processing system, a score based on matches of concepts between said tuple representation of the model answer and said tuple representation of the constructed response by determining whether a concept in each topic of the tuple representation of the model answer is present in the tuple representation of the constructed response;

wherein converting said constructed response into said tuple representation of the constructed response includes;

generating an intermediate output file based on said constructed response;

selecting a word in the intermediate output file;

determining whether a synonym or similar word for the selected word in the intermediate output file exists;

if said synonym or similar word exists, determining whether said synonym or similar word is included in the model answer;

if said synonym or similar word exists and is included in the model answer, substituting said synonym or similar word for said selected word in the intermediate output file; and

generating said tuple representation of the constructed response.

2. The method according to claim 1, wherein converting said constructed response into said tuple representation of the constructed response comprises:

preprocessing said constructed response; and

generating a formatted output file in response to the preprocessing.

3. The method according to claim 2, wherein converting said constructed response into said tuple representation of the constructed response further comprises:

performing a syntactic analysis on said formatted output file; and

generating a syntactic output file in response to said syntactic analysis.

4. The method according to claim 3, wherein performing a syntactic analysis comprises:

tagging a word in said formatted output me with a corresponding part-of-speech tag; and

organizing a plurality of tagged words according to a corresponding grammatical group.

5. The method according to claim 3, wherein converting said constructed response further comprises:

formulating a tuple from a sentence in said syntactic output file; and

generating a predicate-argument output file in response to said formulation.

6. The method according to claim 5, wherein the conversion of said constructed response further comprises:

resolving pronouns in each tuple of said predicate-argument output file; and

generating a pronoun resolved output file in response to said resolution.

7. The method according to claim 6, wherein converting said constructed response further comprises:

analyzing a word in said pronoun resolved output file to determine whether the word is a root word;

substituting said respective root word for said word; and

generating the intermediate output file in response to said substitution.

8. The method according to claim 1, wherein converting said constructed response into said tuple representation of the constructed response comprises:

assigning a topic number to said tuple representation of the constructed response.

9. A system for grading constructed responses, said system comprising:

at least one processor;

a memory coupled to said at least one processor; and

a concept rater module residing in said memory and executed by said at least one processor,

wherein said concept rater module is configured to convert a model answer into a tuple representation of the model answer that includes at least one topic with each topic including at least one concept, to convert a constructed response into a tuple representation of the constructed response, and to generate a score in response to matches of concepts between said tuple representation of the model answer and said tuple representation of the constructed response by determining whether a concept in each topic of the tuple representation of the model answer is present in the tuple representation of the constructed response, wherein the tuple representation of the model answer comprises a predicate-argument data structure;

wherein said concept rater module is further configured to;

generate an intermediate output file based on the constructed response;

select a word in the intermediate output file;

determine whether a synonym or similar word for the selected word in said intermediate output file exists;

if said synonym or similar word exists, determine whether said synonym or similar word is included in the model answer;

if said synonym or similar word exists and is included in the model answer, substitute said synonym or similar word for said selected word in the intermediate output file; and

generate said tuple representation of the constructed response.

10. The system according to claim 9, wherein said concept rater module is further configured to preprocess said constructed response;

and to generate a formatted output file with said preprocessed constructed response.

11. The system according to claim 10, wherein said concept rater module is further configured to perform a syntactic analysis on said formatted output file;

and to generate a syntactic output file in response to a completion of said syntactic analysis.

12. The system according to claim 11, wherein said concept rater module is further configured to tag a word in said formatted output file with a corresponding part-of-speech tag;

and to organize a plurality of tagged words according to a corresponding grammatical group.

13. The system according to claim 11, wherein said concept rater module is further configured to formulate a tuple from a sentence in said syntactic output file and to generate a predicate-argument output file in response to said formulation.

14. The system according to claim 13, wherein said concept rater module is further configured to resolve pronouns in each tuple of said predicate-argument output file and to generate a pronoun resolved output file in response to said resolution of said pronouns.

15. The system according to claim 14, wherein said concept rater module is further configured to:

analyze a word in said pronoun resolved output file for a respective root word;

substitute said respective root word for said word in response to finding said respective root word; and

generate the intermediate output file in response to said substitution.

16. The system according to claim 9, wherein said concept rater module is further configured to assign a topic number to said tuple representation of the constructed response.

17. A non-transitory computer readable storage medium on which is embedded one or more computer programs, said one or more computer programs implementing a method of grading constructed responses, said one or more computer programs comprising a set of instructions for:

converting a model answer into a tuple representation of the model answer including at least one topic with each topic including at least one concept, the tuple representation of the model answer comprising a predicate-argument data structure;

converting a constructed response into a tuple representation of the constructed response; and

generating a score based on matching of concepts between said tuple representation of the model answer and said tuple representation of the constructed response by determining whether a concept in each topic of the tuple representation of the model answer is present in the tuple representation of the constructed response;

wherein said one or more computer programs further comprising a set of instructions for;

generating an intermediate output file based on said constructed response;

selecting a word in the intermediate output file;

determining whether a synonym or similar word for the selected word in the intermediate output file exists;

if said synonym or similar word exists, determining whether said synonym or similar word is included in the model answer;

if said synonym or similar word exists and is included in the model answer, substituting said synonym or similar word for said selected word in the intermediate output file; and

generating said tuple representation of the constructed response.

18. The computer readable storage medium according to claim 17, said one or more computer programs further comprising a set of instructions for:

preprocessing said constructed response; and

generating a formatted output file in response to said preprocessing.

19. The computer readable storage medium according to claim 18, said one or more computer programs further comprising a set of instructions for:

performing a syntactic analysis on said formatted output file; and

generating a syntactic output file in response to said syntactic analysis.

20. The computer readable storage medium according to claim 19, said one or more computer programs further comprising a set of instructions for:

tagging a word in said formatted output file with a corresponding part-of-speech tag; and

organizing a plurality of tagged words according to a corresponding grammatical group.

21. The computer readable storage medium according to claim 19, said one or more computer programs further comprising a set of instructions for:

formulating a tuple from a sentence in said syntactic output file; and

generating a predicate-argument output file in response to said formulation.

22. The computer readable storage medium according to claim 20, said one or more computer programs further comprising a set of instructions for:

resolving pronouns in each tuple of said predicate-argument output file; and

generating a pronoun resolved output file in response to said resolution of said pronouns.

23. The computer readable storage medium according to claim 22, said one or more computer programs further comprising a set of instructions for:

analyzing a word in said pronoun resolved output file for a respective root word;

substituting said respective root word for said word; and

generating the intermediate output file in response to said substitution.

24. The computer readable storage medium according to claim 17, said one or more computer programs further comprising a set of instructions for:

assigning a topic number to said tuple representation of the constructed response.