System for rating constructed responses based on concepts and a model answer

• US 20030200077A1
• Filed: 04/19/2002
• Published: 10/23/2003
• Est. Priority Date: 04/19/2002
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Wolska, Magdalena, Bolge, Eleanor, Chodorow, Martin, Leacock, Claudia
• Granted Patent: US 8,380,491 B2
• Time in Patent Office: Days
• US Class Current: 704/1
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

A concept rater module is utilized to automatically grade or score constructed responses based on a model answer. The concept rater module may be configured to accept a model answer as input. The model answer may be used as a grading key by the concept rater module. The concept rater module may be further configured to accept student responses in a file format. The file format may be ASCII text, a formatted word processing (e.g., WORDPERFECT, MICROSOFT WORD, etc.) and the like. The concept rater module maybe further configured to process a student response into a canonical representation of the student response. The canonical representation of the student response is compared against the model answer by the concept rater module. From the comparison, a score is generated which represents the students ability to cover all the key concepts.

Claims

1. A method for grading constructed responses, said method comprising:

converting a model answer into a model answer canonical representation;

converting a constructed response into an answer canonical representation;

comparing said model answer canonical representation and said answer canonical representation; and

generating a score based on matches between said model answer canonical representation and said answer canonical representation.

2. The method according to claim 1, wherein the converting of said constructed response into said answer canonical representation further comprises:

preprocessing said constructed response; and

generating a formatted output file in response to the preprocessing.

3. The method according to claim 2, wherein the preprocessing of said constructed response further comprises:

reformatting said constructed response into a single sentence per line;

determining an existence of a coordination term for a sentence;

splitting said sentence into two sentences at said coordination term, said existence of said coordination term indicating multiple concepts in said sentence; and

tokenizing said sentence.

4. The method according to claim 2, wherein the step of converting said constructed response into said answer canonical representation further comprises:

performing a syntactic analysis on said formatted output file; and

generating a syntactic output file in response to said syntactic analysis.

5. The method according to claim 4, wherein said step of syntactic analysis further comprises:

tagging a word in said formatted output file with a corresponding part of a speech tag; and

organizing a plurality of tagged words according to a corresponding grammatical group.

6. The method according to claim 4, wherein the conversion of said constructed response further comprises:

formulating a tuple from a sentence in said syntactic output file; and

generating a predicate-argument output file in response said formulation.

7. The method according to claim 6, wherein the conversion of said constructed response further comprises:

resolving pronouns in each tuple of said predicate-argument output file; and

generating a pronoun resolved output file in response to said resolution.

8. The method according to claim 7, wherein the conversion of said constructed response further comprises:

analyzing a word in said pronoun resolved output file to determine whether the said word is a root word;

substituting said root word for said word in response to said respective root word not being on an exception list; and

generating a root word output file in response to said substitution.

9. The method according to claim 8, wherein said conversion of said constructed response further comprises:

determining whether a synonym/similar word for a word in said root word output file exists;

substituting said synonym/similar word for said word in response to said determination of said existence of said synonym/similar word; and

generating said answer canonical representation.

10. A system for grading constructed responses, said system comprising:

at least one processor;

a memory coupled to said at least one processor; and

a concept rater module residing in said memory and executed by said at least one processor, wherein said concept rater module is configured to convert a model answer into a model answer canonical representation, to convert a constructed response into an answer canonical representation, and to generate a score in response to matches of concepts between said model answer canonical representation and said answer canonical representation.

11. The system according to claim 10, wherein said concept rater module is further configured to preprocess said constructed response;

and

to generate a formatted output file with said preprocessed constructed response.

12. The system according to claim 11, wherein said concept rater module is further configured to:

reformat said constructed response into a single sentence per line;

determine an existence of a coordination term for a sentence;

split said sentence into two sentences at said coordination term, said existence of said coordination term indicating multiple concepts in said sentence; and

tokenize said sentence.

13. The system according to claim 11, wherein said concept rater module is further configured to perform a syntactic analysis on said formatted output file;

and

to generate a syntactic output file in response to a completion of said syntactic analysis.

14. The system according to claim 13, wherein said concept rater module is further configured to tag a word in said formatted output file with a corresponding part of speech tag;

and to organize a plurality of tagged words according to a corresponding grammatical group.

15. The system according to claim 13, wherein said concept rater module is further configured to formulate a tuple from a sentence in said syntactic output file and to generate a predicate-argument output file in response to said formulation.

16. The system according to claim 15, wherein said concept rater module is further configured to resolve pronouns in each tuple of said predicate-argument output file and to generate a pronoun resolved output file in response to said resolution of said pronouns.

17. The system according to claim 16, wherein said concept rater module is further configured to:

analyze a word in said pronoun resolved output file for a respective root word;

substitute said respective root word for said word in response to finding said respective root word; and

generate a root word output file in response to said substitution.

18. The system according to claim 17, wherein said concept rater module is further configured to:

determine whether a synonym/similar word for a word in said root word output file exists;

substituting said synonym/similar word for said word in response to a determination of existence of said synonym/similar word; and

generate said answer canonical representation.

19. A computer readable storage medium on which is embedded one or more computer programs, said one or more computer programs implementing a method of grading constructed responses, said one or more computer programs comprising a set of instructions for:

converting a model answer into a model answer canonical representation;

converting a constructed response into an answer canonical representation; and

generating a score based on matching of concepts between said model answer canonical representation and said answer canonical representation.

20. The computer readable storage medium in according to claim 19, said one or more computer programs further comprising a set of instructions for:

preprocessing said constructed response; and

generating a formatted output file in response to said preprocessing.

21. The computer readable storage medium in according to claim 20, said one or more computer programs further comprising a set of instructions for:

reformatting said constructed response into a single sentence per line;

determining an existence of a coordination term for a sentence;

splitting said sentence into two sentences at said coordination term in response to determination of multiple concepts in said sentence; and

tokenizing said sentence.

22. The computer readable storage medium in according to claim 20, said one or more computer programs further comprising a set of instructions for:

performing a syntactic analysis on said formatted output file; and

generating a syntactic output file in response to said syntactic analysis.

23. The computer readable storage medium in according to claim 22, said one or more computer programs further comprising a set of instructions for:

tagging a word in said formatted output file with a corresponding part of a speech tag; and

organizing a plurality of tagged words according to a corresponding grammatical group.

24. The computer readable storage medium in according to claim 22, said one or more computer programs further comprising a set of instructions for:

formulating a tuple from a sentence in said syntactic output file; and

generating a predicate-argument output file in response to said formulation.

25. The computer readable storage medium in according to claim 23, said one or more computer programs further comprising a set of instructions for:

resolving pronouns in each tuple of said predicate-argument output file; and

generating a pronoun resolved output file in response to said resolution of said pronouns.

26. The computer readable storage medium in according to claim 25, said one or more computer programs further comprising a set of instructions for:

analyzing a word in said pronoun resolved output file for a respective root word;

substituting said respective root word for said word in response to said respective root word not being on an exception list; and

generating a root word output file in response to said substitution.

27. The computer readable storage medium in according to claim 26, said one or more computer programs further comprising a set of instructions for:

determining whether a synonym/similar word for a word in said root word output file exists;

substituting said synonym/similar word for said word in response to a determination of existence of said synonym/similar word; and

generating said answer canonical representation.