Consolidated on-line assessment system

• US 8,374,540 B2
• Filed: 12/29/2006
• Issued: 02/12/2013
• Est. Priority Date: 03/15/2002
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Kuntz, David L., Cody, Preston, Ivanov, Georgi Stefanov, Perlow, John E.
• Primary Examiner(s): Thai, Xuan
• Assistant Examiner(s): Fletcher, Jerry-Daryl
• Application Number:
  US11/617,782
• Publication Number:
  US 20070105084A1
• Time in Patent Office: 2,237 Days
• Field of Search: 434/353, 434/322, 434/323, 434/350
• US Class Current: 434/353
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A Consolidated Online Assessment System (COLA System) that creates, manipulates, and distributes an objected-oriented paradigm that represents the scoring and related activities as a unified and integrated family of loosely coupled objects, most notably a “Case” representing a state-machine that replaces the constructed response from the test taker as the unit of work. The Scoring Model and associated Properties that encapsulate the business rules associated with what actions are appropriate or required for a unit of work and the Responses to the unit of work which represent the test-taker-contributed content (e.g., essays or other text-based responses, audio responses, digitized video responses, scanned images, diagrams, lessons plans, etc.) and ties that content to its creator are linked to a Distinct Scorable Unit (DSU) which represents a tree-based mechanism that connects and provides inheritability for the other primary system objects. The abstraction of the DSU allows the system to distinguish between the constructed response as test-taker contributed content and the constructed response as the carrier of state or status information as it passes through the assessment process.

Claims

1. A computer-implemented method for benchmarking responses to test prompts, the method comprising:

during a benchmarking process, assigning with a computer a temporary score to each of a number of candidate responses, wherein said assigning is based upon a scorer review of each of the number of candidate responses, and wherein the number of candidate responses is at least a number of score measures on a score scale;

identifying an official score response from the number of candidate responses for each score measure on the score scale based on the temporary scores assigned to the candidate responses, the official score response being a benchmark response providing a basis for scoring other responses during a scoring process;

assigning with the computer an official score to each official score response based on another scoring review performed during the benchmarking process by the first scorer of the respective temporary score;

assigning with the computer an electronic scoring record for each official score response, wherein the electronic scoring record includes a justification for an official score assigned to an official score response, and wherein the electronic scoring record is associated with the first scorer; and

distributing the official score responses to one or more scorers.

2. The method of claim 1, further comprising:

annotating an official score response.

3. The method of claim 2, wherein the annotating comprises:

selecting text within the official score response;

entering the annotation pertaining to the selected text; and

storing the annotation and a pointer to the annotation in a computer-readable memory separate from the official score response.

4. The method of claim 1, comprising:

identifying additional candidate responses to be scored by multiple scorers;

selecting a particular response of the additional candidate responses for conflict resolution based upon a difference between scores provided by the multiple scorers for the particular response; and

resolving the conflict by a scorer other than the multiple scorers to provide an official score for the particular response.

5. The method of claim 1, comprising assigning labels identifying the official score responses as benchmark responses and training responses, and reviewing the labeled official score responses by a second scorer other than the first scorer for validation of the assigned labels.

6. The method of claim 1, further comprising sorting the number of candidate responses based on the temporary scores assigned to the candidate responses to generate a sorted list, and identifying an official score response based on the sorted list.

7. A computer-implemented method for assessing responses to test questions, the method comprising:

establishing communication between a server computer and a client computer;

communicating, from the server computer to the client computer, instructions for permitting entry of user-generated scoring input at an interface of the client computer, said instructions configured to cause the client computer to permit entry of a temporary score for each of a number of candidate responses to a given training question based on review of the candidate responses by a first scorer performed during a benchmarking process, to permit selection of benchmark responses to the given training question for each score measure on a score scale based on the temporary scores input for candidate responses, and to permit entry of an official score for each benchmark response based upon another scoring review by the first scorer performed during the benchmarking process for the given training question, wherein the number of candidate responses is at least a number of score measures on the score scale;

receiving from the client computer information identifying the given training question, the associated benchmark responses to the given training question, and the associated official scores for the benchmark responses, the official scores providing a basis for scoring the other cases during a scoring process; and

distributing scoring information including the given training question, the associated benchmark responses to the given training question, and the associated official scores for the benchmark responses to one or more scorers.

8. The method of claim 7, further comprising:

receiving an electronic scoring record including an annotation for an official score providing a justification for the official score, wherein the electronic scoring record is associated with the first scorer.

9. The method of claim 8, further comprising storing the annotation and a pointer to the annotation in a computer-readable memory.

10. The method of claim 8, wherein distributing the scoring information to one or more scorers includes distributing instructions for displaying the scoring information to a scorer with the assigned official score and the annotation.

11. The method of claim 10, wherein distributing the scoring information comprises distributing instructions for displaying the annotation as a footnote associated with the official score.

12. The method of claim 8, wherein distributing the scoring information comprises distributing instructions for displaying the annotation as a text pop-up display.

13. The method of claim 8, comprising receiving assigned labels identifying the candidate responses as benchmark and training responses, and receiving validation of the assigned labels by a second scorer other than the first scorer.

14. The method of claim 7, comprising:

identifying additional questions to be scored by multiple scorers;

selecting a particular question of the additional questions for conflict resolution based upon a difference between scores provided by the multiple scorers for the additional question; and

resolving the conflict by a scorer other than the multiple scorers to provide an official score for the particular question.

15. The method of claim 7, further comprising sorting the number of candidate responses based on the temporary scores assigned to the candidate responses to generate a sorted list, and identifying an official score response based on the sorted list.

16. A computer-based system for assessing responses to test questions, comprising:

a server computer including a memory;

the server computer configured to;

establish communication between a server computer and a client computer;

communicate to a client computer instructions for permitting entry of user-generated scoring input at an interface of the client computer, said instructions configured to cause the client computer to permit entry of a temporary score for each of a number of candidate responses to a given training question based on review of the candidate responses by a first scorer performed during a benchmarking process, said instructions further configured to cause the client computer to permit selection of benchmark responses to the given training question for each score measure on a score scale based on the temporary scores entered for the candidate responses and to permit entry of an official score for each benchmark response based upon another scoring review by the first scorer performed during the benchmarking process for the given training question, the official scores providing a basis for scoring other responses during a scoring process, wherein the number of candidate responses is at least a number of score measures on the score scale;

receive from the client computer information identifying the given training question, the associated benchmark responses to the given training question, and the associated official scores for the benchmark responses; and

distribute scoring information including the given training question, the associated benchmark responses to the given training question, and the associated official scores for the benchmark responses to one or more scorers.

17. The system of claim 16, the server computer configured to receive an electronic scoring record including an annotation for an official score providing a justification for the official score, wherein the electronic scoring record is associated with the first scorer.

18. The system of claim 17, the server computer configured to store the annotation and a pointer to the annotation in a computer-readable memory.

19. The system of claim 17, the server computer configured to distribute instructions for displaying the scoring information to a scorer with the assigned official score and the annotation.

20. The system of claim 19, the server computer configured to distribute instructions for displaying the annotation as a footnote associated with the official score.

21. The system of claim 17, the server computer configured to distribute instructions for displaying the annotation as a text pop-up display.

22. The system of claim 17, comprising receiving assigned labels identifying the candidate responses as benchmark and training responses, and receiving validation of the assigned labels by a second scorer other than the first scorer.

23. The system of claim 16, the server computer configured to:

identify additional cases to be scored by multiple scorers;

select a particular case of the additional cases for conflict resolution based upon a difference between scores provided by the multiple scorers for the additional case; and

resolve the conflict by a scorer other than the multiple scorers to provide an official score for the particular case.

24. The system of claim 16, the server computer configured to sort the number of candidate responses based on the temporary scores assigned to the candidate responses to generate a sorted list, and to identify an official score response based on the sorted list.

25. A system for benchmarking responses to test prompts, the system comprising:

a computer; and

a database,

wherein the computer is configured access said database and to execute steps comprising;

during a benchmarking process, assigning with a computer a temporary score to each of a number of candidate responses, wherein said assigning is based upon a scorer review of each of the number of candidate responses, and wherein the number of candidate responses is at least a number of score measures on a score scale;

identifying an official score response from the number of candidate responses for each score measure on the score scale based on the temporary scores assigned to the candidate responses, the official score response being a benchmark response providing a basis for scoring other responses during a scoring process;

assigning with the computer an official score to each official score response based on another scoring review performed during the benchmarking process by the first scorer of the respective temporary score;

assigning with the computer an electronic scoring record for each official score response, wherein the electronic scoring record includes a justification for an official score response, and wherein the electronic scoring record is associated with the first scorer; and

distributing the official score responses to one or more scorers.

26. The system of claim 25, wherein the computer is configured to permit annotating an official score response.

27. The system of claim 26, wherein the annotating comprises:

selecting text within the official score response;

entering the annotation pertaining to the selected text; and

storing the annotation and a pointer to the annotation in a computer-readable memory separate from the official score response.

28. The system of claim 25, wherein the computer is configured to permit assigning labels identifying the official score responses as benchmark and training responses, and reviewing the labeled official score responses by a second scorer other than the first scorer for validation of the assigned labels.

29. The system of claim 25 wherein the computer is configured to permit:

identifying additional candidate responses to be scored by multiple scorers;

selecting a particular response of the additional candidate responses for conflict resolution based upon a difference between scores provided by the multiple scorers for the particular response; and

resolving the conflict by a scorer other than the multiple scorers to provide an official score for the particular response.

30. The system of claim 25, wherein the computer is configured to execute steps comprising:

sorting the number of candidate responses based on the temporary scores assigned to the candidate responses to generate a sorted list; and

identifying an official score response based on the sorted list.