Methods and systems for presentation and evaluation of constructed responses assessed by human evaluators

• US 6,526,258 B2
• Filed: 07/27/2001
• Issued: 02/25/2003
• Est. Priority Date: 03/21/1997
• Status: Expired due to Term

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Bejar, Isaac I., Whalen, Sean J.
• Primary Examiner(s): Cheng, Joe H.
• Application Number:
  US09/916,810
• Publication Number:
  US 20020001795A1
• Time in Patent Office: 578 Days
• Field of Search: 434/118, 434/307.R, 434/308, 434/322, 434/323, 434/327, 434/347, 434/350, 434/362, 434/365, 273/429-432, 706/927, 707/500, 708/167
• US Class Current: 434/350
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: Y10S 706/927 Education or instruction

Abstract

The present invention provides systems and methods for controlled presentation of constructed responses through a variety of computer applications to human evaluators for assessment. The systems and methods further provide for controlled presentation of the constructed responses to minimize the influences of psychometric factors on the accuracy of the human evaluators'"'"' assessments and to maximize the ability to efficiently generate data for use in analyzing the accuracy of the human evaluators'"'"' assessments and the difficulty of the constructed response categories or questions. The systems and methods of the present invention utilize a storage means which relationally stores data regarding the human evaluators, the constructed responses, the scores awarded and the computer applications to utilize. The systems also comprise assessment stations and processing means for selectively transmitting constructed responses to the assessment stations and receiving scores awarded by the human evaluators assigned to the assessment stations.

Claims

1. An application independent assessment system having a number of assessment stations at which human evaluators are assigned to assess constructed responses, comprising:

a data storage means for storing each of the constructed responses in relation to selected ones of the human evaluators and in relation to a specified computer application adapted to present the related constructed response, said specified computer application being one of at least a plurality of computer applications available for presentation of the constructed response; and

a processing means for selecting one constructed response stored in the storage means for assessment, for identifying an active assessment station at which one of the selected human evaluators is assigned, and for enabling the specified computer application to present the selected constructed response at the active assessment station so identified.

2. The system of claim 1, wherein the storage means contains assignments relating each human evaluator to at least one computer application.

3. The system of claim 1, wherein each constructed response is categorized within one of a plurality of constructed response categories and wherein the human evaluators are assigned to assess at least one predetermined constructed response category.

4. The system of claim 3, wherein each human evaluator is assigned to assess constructed responses within at least two constructed response categories.

5. The system of claim 3 further comprising:

a communication link between the processing means and the assessment stations for transmitting constructed responses from the storage means to the assessment stations and for transmitting scores awarded by the human evaluators operating from the assessment stations to the processing means; and

the storage means further storing the scores awarded by the human evaluators to the constructed responses in relation to the constructed responses and the human evaluators.

6. The system of claim 1, wherein the computer applications include word processing programs, drawing programs, audio programs and video programs.

7. The system of claim 1, wherein the constructed responses include at least one of the following:

written answers, figural responses, scanned images, drawings, verbal responses, and videotaped performances.

8. The system of claim 1, wherein each constructed response comprises a computer file.

9. The system of claim 8, wherein a name extension for the computer file is indicative of the related computer applications adapted to present the constructed response.

10. The system of claim 1, wherein the at least one assessment station comprises:

a monitor for presenting video and textual images to the human evaluator assigned to the assessment station; and

an audio means for presenting audio signals to the human evaluator assigned to the assessment station.

11. A method of controlling presentation of constructed responses to human evaluators during an assessment session, wherein each constructed response is assigned to at least one constructed response category and is assessed by at least two human evaluators, the method comprising the steps of:

assigning each human evaluator to at least two constructed response categories;

ordering the constructed responses to be presented to the human evaluators at an assessment station, such that the human evaluators assigned to assess the same constructed responses receive the constructed responses in a different order during the assessment session.

12. The method of claim 11, wherein the constructed responses are ordered by time shifting the presentation of the constructed responses to the human evaluators.

13. The method of claim 11, wherein the constructed responses assigned to a first category are presented to a first human evaluator before the constructed responses assigned to a second category and wherein the constructed responses assigned to the second category are presented to a second human evaluator before the constructed responses assigned to the first category.

14. The method of claim 11, wherein the constructed responses assigned to a category are presented in random order to the human evaluators.

15. The method of claim 11, wherein each constructed response may be presented to a human evaluator through at least one computer application, the method further comprising the steps of:

determining the computer application to use for presenting each constructed response; and

enabling the computer application at the assessment station and importing the constructed response into the computer application.

16. The method of claim 15, further comprising the steps of:

identifying a first assessment station associated with a first human evaluator;

identifying a second assessment station associated with a second human evaluator;

electronically transmitting constructed responses of at least two categories to the first and the second assessment stations such that at least one of the categories transmitted to each of the first and the second assessment stations is the same;

electronically receiving assessments for the constructed responses provided by the first and the second human evaluators from the first and the second assessment station, respectively; and

comparing the assessments received from the first human evaluator and the assessments received from the second human evaluator.