Systems and Methods for Determining the Ecological Validity of An Assessment

• US 20150332599A1
• Filed: 05/19/2015
• Published: 11/19/2015
• Est. Priority Date: 05/19/2014
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Kaufer, David, Beigman Klebanov, Beata, Yeoh, Paul L.
• Granted Patent: US 10,699,589 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

Systems and methods are provided for determining whether an examination prompt elicits responses of a type for which the examination prompt is designed. Word types for each of a plurality of words in a plurality of responses to the examination prompt are identified. An evaluation distribution of the identified word types is determined. Predetermined distributions stored on a computer-readable medium associated with a plurality of different response types are accessed, and the evaluation distribution is compared to each of the predetermined distributions to determine which predetermined distribution the evaluation distribution is most similar. A determination is made as to whether the most similar distribution is associated with a same response type as the type for which the examination prompt is designed. An indication is made as to whether the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed.

Claims

1. A computer-implemented method of determining whether an examination prompt elicits responses of a type for which the examination prompt is designed, comprising:

identifying, using a processing system, word types for each of a plurality of words in a plurality of responses to the examination prompt, each word being a member of one of a plurality of predefined word types;

determining, using the processing system, an evaluation distribution of the identified word types;

accessing, using the processing system, predetermined distributions stored on a computer-readable medium associated with a plurality of different response types;

comparing, using the processing system, the evaluation distribution to each of the predetermined distributions to determine which predetermined distribution the evaluation distribution is most similar;

determining, using the processing system, whether the most similar distribution is associated with a same response type as the type for which the examination prompt is designed;

indicating, using the processing system, whether the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed.

2. The method of claim 1, wherein the plurality of responses are from a trial administration of the examination prompt, wherein when the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed, the examination prompt is included in a high-stakes examination.

3. The method of claim 2, wherein when the examination prompt is deemed not to elicit responses of the type for which the examination prompt is designed, the examination prompt is adjusted prior to inclusion on any high-stakes examination.

4. The method of claim 1, wherein the evaluation distribution is determined using identified word types from responses to a plurality of examination prompts of a trial examination, wherein the method determines whether the plurality of evaluation prompts elicit responses of the type for which they are designed.

5. The method of claim 1, wherein the examination prompt is designed to elicit argumentative responses.

6. The method of claim 1, wherein the examination prompt is designed to elicit a response of a type selected from the group comprising:

argumentative, informational, instructional, narrative, scenic, other-portrait, or self-portrait.

7. The method of claim 1, wherein the predefined word types include two or more of:

personal, emotion, assertive, description, public, academic, future, past, personal relations, reasoning, interactive, elaboration, reporting, directing, narrative, character, no match.

8. The method of claim 1, wherein the word types are identified by accessing a word-type repository stored on a computer-readable medium that identifies one of the predefined word types for each of a plurality of words and phrases.

9. The method of claim 8, wherein the word-type repository is a repository containing more than 1 million words and phrases categorized according to rhetorical category.

10. The method of claim 1, wherein said comparison is performed via a Kullback-Leibler divergence operation.

11. The method of claim 1, wherein said comparison is performed via a vector-space comparison.

12. The method of claim 1, wherein the predetermined distributions for the different response types are determined based on essays of the different response types generated in a non-examination context.

13. The method of claim 12, wherein the essays are generated over a multi-week period of time.

14. A computer-implemented system for determining whether an examination prompt elicits responses of a type for which the examination prompt is designed, comprising:

a processing system;

one or more computer-readable mediums encoded with;

a word type repository identifying one of a plurality of predetermined word types for each of a plurality of words;

a predetermined word type distribution associated with each of a plurality of different response types;

instructions for commanding the processing system to;

identify word types for each of a plurality of words in a plurality of responses to the examination prompt, each word being a member of one of a plurality of predefined word types using the word type repository;

determine an evaluation distribution of the identified word types;

access the predetermined distributions associated with the plurality of different response types;

compare the evaluation distribution to each of the predetermined distributions to determine which predetermined distribution the evaluation distribution is most similar;

determine whether the most similar distribution is associated with a same response type as the type for which the examination prompt is designed;

indicate whether the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed.

15. The system of claim 14, wherein the plurality of responses are from a trial administration of the examination prompt, wherein when the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed, the examination prompt is included in a high-stakes examination.

16. The system of claim 15, wherein when the examination prompt is deemed not to elicit responses of the type for which the examination prompt is designed, the examination prompt is adjusted prior to inclusion on any high-stakes examination.

17. The system of claim 14, wherein the evaluation distribution is determined using identified word types from responses to a plurality of examination prompts of a trial examination, wherein the method determines whether the plurality of evaluation prompts elicit responses of the type for which they are designed.

18. The system of claim 14, wherein the examination prompt is designed to elicit argumentative responses.

19. The system of claim 14, wherein the examination prompt is designed to elicit a response of a type selected from the group comprising:

argumentative, informational, instructional, narrative, scenic, or self-portrait.

20. A computer-readable medium encoded with instructions for commanding a processing system to execute steps of a method for determining whether an examination prompt elicits responses of a type for which the examination prompt is designed, the method comprising:

identifying, using the processing system, word types for each of a plurality of words in a plurality of responses to the examination prompt, each word being a member of one of a plurality of predefined word types;

determining, using the processing system, an evaluation distribution of the identified word types;

accessing, using the processing system, predetermined distributions stored on a computer-readable medium associated with a plurality of different response types;

comparing, using the processing system, the evaluation distribution to each of the predetermined distributions to determine which predetermined distribution the evaluation distribution is most similar;

determining, using the processing system, whether the most similar distribution is associated with a same response type as the type for which the examination prompt is designed;

indicating, using the processing system, whether the examination prompt is deemed to elicit responses of the type for which the examination prompt is designed.