Computerized system for scoring constructed responses and methods for training, monitoring, and evaluating human rater's scoring of constructed responses

• US 5,991,595 A
• Filed: 03/20/1998
• Issued: 11/23/1999
• Est. Priority Date: 03/21/1997
• Status: Expired due to Term

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Grant, Mark, Farnum, Marisa D., Romano, Frank J.
• Primary Examiner(s): Cheng, Joe H.
• Application Number:
  US09/045,621
• Time in Patent Office: 613 Days
• Field of Search: 434/118, 434/156, 434/157, 434/169, 434/307 R, 434/308, 434/322, 434/323, 434/327, 434/347, 434/350, 434/353, 434/362, 358/450, 704/10, 706/45, 706/927, 707/104, 707/500, 707/501, 707/512, 382/10, 382/48, 382/321, 273/429-432, 345/302, 345/348
• US Class Current: 434/353
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

Systems and methods for presentation to raters of constructed responses to test questions in electronic workfolders. The systems and methods provide for selection and designation of constructed responses into categories for use in quality control functions and training applications. A storage device is used to relationally store data regarding the raters, the constructed responses, and the scores awarded for use in selecting constructed responses for use in quality control and training. The system also comprises rater stations and processing systems for selectively transmitting constructed responses to the rater stations and receiving scores awarded by the raters assigned to the rater stations during both training and actual production scoring. The system further comprises rater stations and processing systems for displaying rater performance data, topic performance data, constructed response volume data, and candidate sample volume data.

Claims

1. A workfolder creation and distribution system for use in a computer test scoring system having at least one rater station at which constructed responses to test questions are displayed for an authorized rater and where said authorized rater awards a score to each of the constructed responses so displayed, each constructed response being associated with a predetermined topic, the workfolder creation and distribution system comprising:

a memory which stores the constructed responses in electronic form keyed to a test section associated therewith;

means for electronically grouping a plurality of the constructed responses in said memory into a production workfolder for evaluation of the plurality of constructed responses by the at least one rater; and

workfolder transmission means for accessing the production workfolder and transmitting the plurality of constructed responses in said production workfolder to at least one rater station operable to display the constructed responses and operable to accept scores awarded by an authorized rater at said at least one rater station in response to the constructed responses so displayed, said workfolder transmission means transmitting the scores associated with the plurality of constructed responses to the memory for storage in association with the constructed responses in the production workfolder.

2. The system of claim 1, wherein a rater station is operable to display the plurality of constructed responses in any order selected by the authorized rater.

3. The system of claim 1, wherein the grouping means provides a workfolder summary including a list of the constructed responses in the workfolder along with the score assigned thereto and allows the rater to select and redisplay any of the listed constructed responses for further scoring.

4. The system of claim 1, wherein the workfolder transmission means transmits the scores in response to a commit signal generated by the rater station after the authorized rater has completed the scoring of each of the plurality of constructed responses, the rater station being operable to display and redisplay any of the plurality of constructed responses and to accept and to replace scores awarded by the authorized rater prior to the generation of said commit signal.

5. The system of claim 1, wherein the grouping means further electronically groups constructed responses into at least one of the following:

calibration workfolders for training new raters in applying scoring guidelines and certification workfolders for testing trainee raters.

6. The system of claim 1, wherein a common test section is associated with each of the plurality of constructed responses.

7. The system of claim 5, further comprising:

a constructed response classification means for selecting some of the constructed responses and defining the same as calibration constructed responses and for selecting some of the other constructed responses and defining the same as certification constructed responses, each said calibration constructed responses and each said certification constructed responses being stored in the memory in association with one of the predetermined topics, the grouping means further electronically grouping, based on the topics associated therewith, a plurality of said calibration constructed responses into a calibration workfolder and a plurality of said certification constructed responses into a certification workfolder.

8. The system of claim 1, further comprising:

a constructed response classification means for selecting some of the constructed responses and defining the same as monitoring constructed responses, each monitoring constructed response being stored in the memory in association with one of the predetermined topics, the grouping means electronically including, based on the topics associated therewith, a number of the monitoring constructed responses into the production workfolder.

9. The system of claim 8, wherein the number of monitoring constructed responses included in the production workfolder is preselected.

10. The system of claim 1, wherein the memory stores qualification information electronically related to a rater identification code, said qualification information being indicative of related rater'"'"'s status as trainee, certified rater, experienced rater, or scoring leader.

11. The system of claim 10, wherein the qualification information is defined for a number of test sections for each authorized rater such that at least one authorized rater is qualified to score more than one test section, wherein the qualification information for said rater is test section independent.

12. The system of claim 11, further comprising:

a constructed response classification means for selecting some of the constructed responses and defining the same as calibration constructed responses and for selecting some of the other constructed responses and defining the same as certification constructed responses, each said calibration constructed response and each said certification constructed response being stored in the memory in association with one of the predetermined topics, the grouping means further electronically grouping, based on the topics associated therewith, a plurality of said calibration constructed responses into a calibration workfolder and a plurality of said certification constructed responses into a certification workfolder, the workfolder transmission means being enabled to transmit calibration workfolders and certification workfolders to rater stations at which the authorized rater status is trainee and calibration workfolders and production workfolders to rater stations at which the authorized rater status is experienced or certified.

13. The system of claim 12, wherein the constructed response classification means is additionally operable for selecting some other of the constructed responses and defining the same as benchmark constructed responses and for selecting others of the constructed responses and defining the same as rangefinder constructed responses, each benchmark constructed response and each rangefinder constructed response being stored in the memory in association with one of the predetermined topics, said workfolder transmission means transmitting said benchmark constructed responses and said rangefinder constructed responses to the rater station where said benchmark constructed responses and said rangefinder constructed responses are displayed upon a request initiated by the authorized rater.

14. The system of claim 11, wherein the grouping means includes each constructed response in at least two different production workfolder and transmits each different production workfolder to a different rater station with a different authorized rater, the grouping means comparing scores awarded to the same constructed response by different raters to determine whether said constructed response requires adjudication, and if so, defining said constructed response as an adjudication constructed response, the grouping means electronically grouping a plurality of adjudication constructed responses, based on the topics associated with said constructed responses, into an adjudication workfolder, the workfolder transmission means being enabled to transmit production workfolders and adjudication workfolders to adjudicators and adjudication workfolders to scoring leaders.

15. The system of claim 1 wherein the authorized rater is one of the following:

a trainee having access only to training materials and calibration and certification workfolders, an experienced or certified rater having successfully completed training and having access to read and score production constructed responses; and

a scoring leader having responsibility for training, assisting, and monitoring.

16. In a computer test scoring system comprising at least one rater station at which constructed responses to test questions are displayed to an authorized rater and at which the authorized rater awards a score in response to constructed responses so displayed, a method for selecting and designating constructed responses for training and quality control, a method comprising the steps of:

electronically distributing one constructed response to at least two rater stations for scoring by at least two different authorized raters;

electronically comparing the scores awarded by the different raters, and if the scores satisfy predetermined criteria, defining the constructed response as a qualified sample constructed response; and

classifying each qualified sample constructed response in one of a number of categories of constructed responses wherein said classification is determinative of how said qualified sample constructed response is used by the computer scoring system.

17. The method of claim 16, wherein the step of electronically distributing said one constructed response comprises the steps of:

electronically distributing said one constructed response to a first rater station and a second rater station whereby a first rater and a second rater award a first score and a second score, respectively, for said one constructed response; and

electronically transmitting said one constructed response to a third rater station and a fourth rater station whereby a third rater and a fourth rater award a third score and a fourth score, respectively, for said one constructed response if a difference between the first score and the second score satisfies said predetermined criteria.

18. The method of claim 17, wherein the first rater and the second rater are located in different scoring centers.

19. The method of claim 17, wherein the third and fourth rater are designated as scoring leaders with responsibilities for training assisting, assigning, and monitoring the work of other raters.

20. The method of claim 19, wherein the third rater and fourth rater are located in different scoring centers from each other.

21. The method of claim 16, wherein the number of categories of constructed responses include calibration constructed responses for use in training of raters and practice for raters, training sample constructed responses for use in training raters, certification constructed responses for use in qualifying raters for scoring constructed responses associated with certain topics, monitoring constructed responses for use in monitoring the quality of the raters'"'"' scoring, benchmark constructed responses and rangefinder constructed responses for use in establishing criteria for particular scores awarded by the raters.

22. The method of claim 16, wherein the computer test scoring system comprises a classification station, and said classifying step includes the step of electronically transmitting said qualified sample constructed responses to the classification station for classification by an authorized individual.

23. The method of claim 22, said classifying step further comprising the steps of:

reviewing the qualified sample constructed responses at the classification station;

assigning one of the number of categories of constructed responses to said qualified sample constructed responses; and

storing each said qualified sample constructed response so classified with the category assigned thereto in a data storage device accessed by the computer test scoring system during rater training and scoring.

24. A fully integrated computer test scoring system for use in training raters for scoring constructed responses to test questions for use in controlling the quality of the scoring by raters, the system comprising:

at least one rater station operative to display constructed responses for review by raters and operative to accept scores awarded by the raters in response to the constructed responses so displayed;

a data storage device which stores constructed responses and relationships between the constructed responses and raters;

a processor which accesses the data storage device and distributes the constructed responses to the at least one rater station based on the relationships between the constructed responses and the raters operating from the at least one rater station, wherein the raters are qualified based on level of expertise; and

a number of classification stations interfaced with the processor and the data storage device for use in classifying some of the constructed responses, based on the scores awarded thereto, as one of a predetermined category of constructed responses, wherein said processing means distributes only certain categories of constructed responses to certain raters based on the raters'"'"' level of expertise.

25. The system of claim 24, wherein the predetermined categories of constructed responses include calibration constructed responses for use in training of raters and practice for raters, training sample constructed responses for use in training raters, certification constructed responses for use in qualifying raters for scoring constructed responses associated with certain topics, monitoring constructed responses for use in monitoring the quality of the raters'"'"' scoring, benchmark constructed responses and rangefinder constructed responses for use in establishing criteria for particular scores awarded by the raters.

26. The system of claim 25, wherein said rater stations are operative to allow raters to request for a test section at least one of a training sample constructed response, calibration constructed response, rangefinder constructed response, and benchmark constructed response.

27. The system of claim 24, wherein each constructed response is associated with a predetermined test section and wherein the raters are qualified by test section.

28. The system of claim 27, wherein the categories of constructed responses include calibration constructed responses and certification constructed responses, the processing means selecting certification constructed responses and calibration constructed responses to transmit to the rater station when the rater assigned to said rater station is a trainee.

29. The system of claim 27, wherein the categories of constructed responses include monitoring constructed responses having qualified scores associated therewith, the processing means selecting a number of monitoring constructed responses to be transmitted to the rater station when the rater assigned to said rater station is experienced or certified.

30. The system of claim 27, wherein the categories of constructed responses include benchmark constructed responses and rangefinder constructed responses having predetermined scores associated therewith, the processing means selecting benchmark constructed responses and rangefinder constructed responses to transmit to the rater station in response to a benchmark request and a rangefinder request from the rater station.

31. The system of claim 24, wherein each constructed response is associated with a predetermined topic, the system further comprising:

an electronic workfolder distribution means interfaced with the processor for grouping the constructed responses into workfolders based upon the test section associated with the constructed response and the category of the constructed response and for transmitting said workfolders to the rater stations.

32. A computer test scoring system for use in monitoring the quality and performance of scoring of constructed responses to test questions by a plurality of raters evaluating particular test sections within tests, the system comprising:

means for storing in a production workfolder a plurality of constructed responses in an electronic workfolder for evaluation by a rater;

a number of rater stations, located at at least one scoring center, operative to display rater performance data, topic performance data for monitoring rater and topic performance across scoring centers, constructed response volume data, and candidate sample volume data;

a data storage device which stores constructed responses and related rater performance data, topic performance data, essay volume data, and candidate sample volume data; and

a processor which distributes rater performance data, topic performance data, constructed response volume data, and candidate sample volume data stored in the data storage device to the rater stations based on the relationships among the rater performance data, topic performance data, constructed response volume data, and candidate sample volume data and the raters operating from the respective rater stations, wherein the raters are qualified to view the data based on system status.

33. The system of claim 32, wherein the rater performance data comprise:

the total number of constructed responses scored by each rater;

the total number of constructed responses for each rater with scores exactly equal to scores of another rater;

the total number of constructed responses for each rater with scores adjacent to scores of another rater;

the total number of constructed responses for each rater with scores discrepant to scores of another rater; and

the percentage distribution of awarded scores for each rater, wherein the data is compiled for transactions made by each rater during a given period of time and wherein the data is grouped by test section and test program.

34. The system of claim 33 wherein the rater performance data further comprise:

a beginning and ending date indicating a period on which is being reported;

a date on which the rater scored each essay;

a testing program identifier indicating the testing program for which each constructed response is being scored;

an item code uniquely identifying a specific test section to which the constructed response belongs;

an item name identifying the test section; and

a folder number identifying a specific workfolder in which the constructed response is stored.

35. The system of claim 32 wherein the topic performance data comprise:

scoring center rater data which provide the capability to review rater performance at a particular scoring center;

network topics data which provide the capability to review overall performance summary across scoring centers for all topics scored for a selected workfolder type; and

scoring center topics data which provide the capability to review the performance summary for all topics scored for a selected workfolder type for each scoring center that is active during a specific date range.

36. The system of claim 35 wherein the scoring center rater data comprises:

the total number of constructed responses scored by each rater;

the total number of constructed responses scored by each rater with scores exactly equal to scores of another rater,

the total number of constructed responses scored by each rater with scores adjacent to scores of another rater,

the total number of constructed responses scored by each rater with scores discrepant to scores of another rater; and

the percentage distribution of awarded scores for each rater, wherein the data is compiled for transactions occurring at a specific site during a specified time period and grouped by topic name, test section, and test program.

37. The system of claim 35, wherein the network topic data comprise:

the total number of constructed responses scored by each rater;

the total number of constructed responses scored by each rater with scores exactly equal to scores of another rater;

the total number of constructed responses scored by each rater with scores adjacent to scores of another rater;

the total number of constructed responses scored by each rater with scores discrepant to scores of another rater; and

the percentage distribution of awarded scores for each rater, wherein the data is compiled for transactions occurring across all scoring sites over a period of time and is grouped by topic, test section, test program, and scoring center.

38. The system of claim 35 wherein the topic performance data comprises:

the total number of constructed responses scored by each rater;

the total number of constructed responses scored by each rater with scores exactly equal to scores of another rater;

the total number of constructed responses scored by each rater with scores adjacent to scores of another rater;

the total number of constructed responses scored by each rater with scores discrepant to scores of another rater; and

the percentage distribution of awarded scores for each rater, wherein the data is compiled from transactions occurring across sites and grouped by topic, test section, and test program.

39. The system of claim 32 wherein the essay volume data for reporting on the volume of essays comprise:

the number of open and pending constructed responses for each topic;

the number of constructed responses scored for each topic;

the number of constructed responses for which scores have been sent for each topic; and

the total number of constructed responses for each topic, wherein the data is compiled for transactions related to a particular testing program and occurring over a specified period of time and grouped by test program, test section, and topic.

40. The system of claim 39 wherein the constructed response volume data further comprise:

the minimum number of readings required for production constructed responses for a testing program and test section;

an indicator of whether a rater must be an experienced rater for a testing program and test section;

the number of constructed responses for a topic that were handwritten by examinees and that must be scanned into the system before they can be scored online;

an indicator of whether a constructed response is awaiting assignment to a rater;

an indicator of whether a constructed response is in the process of being read by a rater;

the number of production constructed responses that have been read and scored by the required number of raters for a particular topic and test date;

the number of production constructed responses for which scores have been sent to the testing program; and

the number of production constructed responses that have been processed in the system for a particular topic and test date.

41. The system of claim 32 wherein the candidate sample volume statistics for reporting on the volume of candidate samples comprise:

the number of open candidate samples for each topic;

the number of pending candidate sample constructed responses for each topic;

the number of candidate sample constructed responses designated as qualified for each topic;

the number of candidate sample constructed responses rejected for each topic; and

the total number of candidate sample constructed responses;

wherein data is compiled for transactions related to a particular testing program and occurring between a specified range of dates and the data is organized by test program, test section, and topic.

42. The system as recited in claim 41 wherein the candidate sample data further comprise:

the number of candidate sample constructed responses that have been qualified by the system as sample constructed responses for a particular topic and test date;

the number of candidate sample constructed responses that were rejected by the system because the scores awarded were different for that topic; and

the total number of rejected and qualified candidate sample constructed responses for a topic and test date in a test section.