Free response test grading method

• US 4,978,305 A
• Filed: 06/06/1989
• Issued: 12/18/1990
• Est. Priority Date: 06/06/1989
• Status: Expired due to Term

• Current Assignee: Educational Testing Service Princeton NJ 08541
• Original Assignee: Educational Testing Service
• Inventors: Kraft, William D.
• Primary Examiner(s): Apley, Richard J.
• Assistant Examiner(s): Doyle, Jennifer L.
• Application Number:
  US07/362,262
• Time in Patent Office: 560 Days
• Field of Search: 434/353, 434/359, 434/358, 235/435, 235/454-456, 235/462-466, 235/472, 382/58, 382/59
• US Class Current: 434/353
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

A method and system for grading free response tests including the use of highly accurate machine-readable data codes to uniquely associate test-taker, test and reader/grader, and a portable sensing device which stores codes read by the device for subsequent entry into a host computer. The method permits multiple readers/graders to evaluate the same test without one reader/grader influencing another, while reducing paper handling and key entry of data inherent in large volume paper and pencil testing techniques.

Claims

1. A method for processing free response test questions while maintaining the independence of grades and reducing paper handling comprising:

selectively associating with a test-taker machine-readable data code associated with the test-taker;

retrievably storing said association;

assembling test-taker materials which bear machine-readable data code associated with the test-taker;

selectively associating with a reader machine-readable data code associated with the reader;

retrievably storing said association;

assembling reader materials which bear the machine-readable data code associated with the reader;

selectively associating with a grade machine-readable data code associated with the grade;

assembling grade materials which bear the machine-readable data code associated with the grade;

selectively associating with a test machine-readable data codes associated with the test;

retrievably storing said association;

assembling test materials which bear the machine-readable data code associated with the test;

selectively associating with a test question machine-readable data code associated with the test question;

assembling test question materials which bear the machine-readable data associated with the test question;

assembling for all test-takers at a grading location test-taker materials, reader materials, grade materials, test materials and test question materials;

receiving selectively generated reader input for assigning a grade to said test materials;

selectively scanning, with a portable sensing device located at the grading location, the machine-readable data codes associated with the test-taker, the reader, the grade for the test question, the test, and the test question, for each question graded by a reader;

wherein the data code associated with the grade is scanned as the grade is assigned;

retrievably storing the data codes scanned by the sensing device and an indication of the time of the scanning; and

processing the stored data codes and the stored data code associations on a computer.

2. The method of claim 1 wherein the assembling of the test-taker materials further comprises distributing machine-readable data code labels associated with the test-taker to the test-taker and affixing the labels to the test materials.

3. The method of claim 2 wherein the labels are distributed to the test-taker during a registration for the test.

4. The method of claim 2 wherein the labels are distributed to the test-taker at a testing location.

5. The method of claim 1 wherein the portable sensing device is provided with data code storage and time of scan determining means.

6. The method of claim 5 wherein the portable sensing device retrievably stores the data codes scanned by the sensing device and the time of the scanning;

7. The method of claim 6 wherein periodically a sensing-device-sweep is performed for a table of readers.

8. The method of claim 7 wherein periodically a table of readers is provided with a table sweep report.

9. The method of claim 8 wherein the table sweep report comprises identifying the table and selectively associating readers and time interval, between successive entry of grades for test questions, with grades and test questions for a specified period of time.

10. The method of claim 9 wherein the table sweep report further comprises:

readers'"'"' names and numbers;

a percentage of scores each reader awarded at each possible grade;

a total number of test questions graded by readers at a table in a current sweep;

the mean scores awarded by each reader at the table for the last sensing-device-sweep and for all prior sweeps;

a scoring confidence interval for each reader at the table;

a mean score awarded by all the readers at the table; and

an identifying mark and number of questions graded by a reader if the mean score awarded by the reader falls outside the readers'"'"' confidence interval.

11. A system for processing free response test questions while maintaining the independence of grades and reducing paper handling comprising:

printing indicia of and retrievably storing data code identifying each test-taker;

distributing the indicia to each test-taker and affixing the indicia to test-taker materials used by the test-taker;

printing indicia of and retrievably storing data code identifying each reader and distributing the indicia to each reader;

printing indicia of each grade which can be awarded for a test question response and distributing the indicia to each reader;

printing indicia of and retrievably storing data code identifying each test which is to be graded and distributing the indicia to each reader;

printing indicia of each test question which is to be graded and distributing the indicia to each reader;

printing indicia of and retrievably storing data code identifying each folder which is to be graded and distributing the indicia to each reader;

distributing portable data code sensing devices and test-taker materials to the reader;

selectively scanning with the portable sensing device the data code indicia associated with the test-taker, the reader, the grade, the test, and the test question, for each question graded by a reader;

wherein the data code indicia associated with the grade is scanned as the grade is assigned;

means for retrievably storing the data codes scanned by the sensing device; and

means for processing the stored data codes and the stored identifying data codes so as to provide a score for each test question associated with a test-taker.

12. The system of claim 11 wherein the sensing device is programmed to silently acknowledge a successful data code scan.

13. The system of claim 12 wherein processing of the data codes scanned by the sensing device further comprises means for periodically providing a table of readers with a table sweep report.

14. The system of claim 13 wherein the table sweep report comprises:

identification of the table and an indication of a mean time for readers at a table to grade a test question;

readers'"'"' names and numbers;

a percentage of scores each reader awarded at each possible grade;

a total number of test questions graded by readers at a table in a current sweep;

the mean scores awarded by each reader at the table for the last sensing-device-sweep and for all prior sweeps;

a scoring confidence interval for each reader at the table;

a mean score awarded by all the readers at the table; and

an identifying mark and number of questions graded by a reader if the mean score awarded by the reader falls outside the readers'"'"' confidence interval.

15. A method for scoring and grading a plurality of test materials containing a plurality of free response test questions wherein the responses to the questions are independently assigned a grade by a plurality of readers within predetermined grade scales depending on the response comprising:

providing a plurality of test-taker material units, each unit being provided with a distinct machine-readable data code indicia means associated with and identifying a test-taker;

providing a plurality of reader material units each unit being provided with a distinct machine-readable data code indicia means associated with and identifying a reader;

providing a plurality of grading material units each unit being provided with a distinct machine-readable data code indicia means associated with and identifying each assignable grade within the predetermined grade scale;

providing a plurality of test material units each unit being provided with a distinct machine-readable data code indicia means associated with and identifying a test;

providing a plurality of test question material units each unit being provided with a distinct machine-readable data code indicia means associated with and identifying a test question;

associating and retrievably storing data codes identifying test-takers with data code indicia identifying the test-takers on a first computer database;

receiving selectively generated reader input for assigning, for each test-taker test question response, a specific grade to be associated with the test-taker, reader, test and test question;

selectively scanning with a portable data code indicia sensing device means, for each test-taker test question response, the data code indicia for the specific grade, test-taker, reader, test and test question;

retrievably storing said selectively scanned data codes on a second computer database; and

processing the stored first and second databases on a computer to provide a score for each test question associated with a test-taker;

wherein processing of the stored second database comprises periodically providing a table of readers with a table sweep report;

wherein the table sweep report comprises identification of the table and an identification of the time for readers at a table to grade a test question.

16. The method of claim 15 wherein the table sweep report further comprises:

readers'"'"' names and numbers;

a percentage of scores each reader awarded at each possible grade;

a total number of test questions graded by readers at a table in a current sweep;

the mean scores awarded by each reader at the table for the last sensing-device-sweep and for all prior sweeps;

a scoring confidence interval for each reader at the table;

a mean score awarded by all the readers at the table; and

an identifying mark and number of questions graded by a reader if the mean score awarded by the reader falls outside the readers'"'"' confidence interval.

17. A method for scoring and grading a plurality of test materials associated with a test and test-takers and containing a plurality of free response test questions wherein the responses to the questions are independently assigned a grade by a plurality of readers within predetermined grade scales depending on the response comprising:

associating each test-taker with a test-taker number, means for retrievably storing the association, storing the association, printing the examinee number in the form of a bar code on self-adhering labels, distributing the bar code labels to the test-takers and selectively affixing the labels to test-takers test materials;

associating with each reader a reader number, means for retrievably storing the association, storing the association, printing the reader number in the form of a bar code on self-adhering labels;

associating with each test a test number, means for retrievably storing the association, storing the association and selectively printing the test number in the form of a bar code on test-taker test materials and on folder control sheets;

associating with a test question a test question number, means for retrievably storing the association, storing the association and selectively printing the test question number in the form of a bar code on test-taker test materials and on the folder control sheets;

associating with a folder used to hold a selection of test-taker materials a folder number, means for retrievably storing the association, storing the association, selectively printing the folder number in the form of a bar code on self-adhering labels;

affixing the folder number label on the folder control sheet and affixing the folder control sheet to the folder;

identifying each grade within the predetermined grade scale for each test question and printing each grade in the form of a bar code on an essay scoring card;

affixing reader number bar code labels to the essay scoring card;

selectively assembling at grading tables, test-taker test materials in folders with affixed folder control sheet, readers and essay scoring cards with affixed reader number labels;

selectively scanning with a portable sensing device means bar code labels associated with the test-taker, the reader, the grade for the test question, the test, and the test question, for each question graded by a reader;

retrievably storing bar codes scanned by the sensing device and the time of scanning; and

processing the stored scanned bar codes and the stored bar code associations on a computer to generate a test-taker report.

18. The method of claim 17 wherein processing of stored scanned bar codes comprises periodically providing a table of readers with a table sweep report.

19. The method of claim 18 wherein the table sweep report comprises identification of a table and an indication of time for readers to grade a test question.

20. The method of claim 19 wherein the table sweep report further comprises:

readers'"'"' names and numbers;

a percentage of scores each reader awarded at each possible grade;

a total number of test questions graded by readers at a table in a current sweep;

the mean scores awarded by each reader at the table for the last sensing-device-sweep and for all prior sweeps;

a scoring confidence interval for each reader at the table;

a mean score awarded by all the readers at the table; and

an identifying mark and number of questions graded by a reader if the mean score awarded by the reader falls outside the readers'"'"' confidence interval.