On-line essay evaluation system

• US 5,987,302 A
• Filed: 03/20/1998
• Issued: 11/16/1999
• Est. Priority Date: 03/21/1997
• Status: Expired due to Term

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Johnson, Arlene A., Hatfield, Lou A., Kessler, Theodore E., Pocino, Cheryl A., Kahn, Helen D., Rosenthal, Michele, Williams, Patricia G., Kuntz, David L., Driscoll, Gary
• Primary Examiner(s): Cheng, Joe H.
• Application Number:
  US09/045,622
• Time in Patent Office: 606 Days
• Field of Search: 434/118, 434/156, 434/157, 434/169, 434/307 R, 434/308, 434/322, 434/323, 434/327, 434/347, 434/350, 434/353, 434/365, 704/10, 358/450, 706/45, 706/927, 707/104, 707/500, 707/501, 707/512, 707/100, 707/102, 382/10, 382/48, 382/321, 273/429-432, 345/302, 345/348
• US Class Current: 434/353
• CPC Class Codes: G09B 5/065 Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems
• CPC Class Codes: G09B 5/14 with provision for individual teacher-student communication
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

Systems and methods for on-line essay evaluation offer students the opportunity to prepare practice essays, submit the essays to trained, expert readers, and retrieve an evaluation at the student'"'"'s convenience. The system provides the capability for a user or test taker to submit essays at any time during the year, independent of the timing of an actual testing event, and to receive prompt, consistent evaluations of the submitted essays. Further, the system provides the capability to prioritize essays and schedule readers so that essays can be evaluated on a rush basis. The essays are evaluated in a manner that provides useful instructional feedback to students about their skills relative to any assessment or test that the student wishes to take.

Claims

1. A method for on-line evaluation of a constructed response to an examination question, comprising the steps of:

accessing a web site via the Internet, ordering an evaluation account, and selecting an examination question;

an examinee constructing an essay response to said examination question;

electronically submitting said essay response to said web site for evaluation;

positioning the submitted essay response in a queue of essay responses that need to be evaluated by a grader, the essay being placed in the queue based upon at least one of the time of submission of said essay response and the date an evaluation of the essay response is due to the examinee;

the grader evaluating the essay response in accordance with calibrated grading guidelines, selecting an overall evaluation, at least one pre-defined feedback comment, and a score; and

releasing for retrieval by the examinee the overall evaluation and the pre-defined feedback comments regarding said essay response.

2. An on-line essay evaluation system for submitting essay responses for evaluation by graders and retrieving an evaluation of the essay response from the grader for submission to the examinee, comprising:

a database which stores a plurality of questions which elicit essay responses from an examinee;

an essay response submission system which selects a question from said database and which submits an essay response to the selected question;

an essay response administrative system which stores essay responses from a plurality of examinees in a queue and selectively distributes the queued essay responses to graders for evaluation;

an automated grading system which enables a grader to evaluate an essay response, to select an overall evaluation, at least one pre-defined additional comment, and a score; and

an evaluation delivery system which releases to the examinee for the examinee'"'"'s retrieval said overall evaluation and any pre-defined additional comments from the grader.

3. The system of claim 2 wherein said essay response administrative system schedules graders to evaluate queued essay responses so that essay responses are evaluated and feedback is stored and released for retrieval by the examinee within a predetermined time period.

4. The system of claim 2 wherein said essay response administrative system prioritizes the presentation and distribution of essay responses to graders so that essay responses are evaluated and feedback is stored and released for retrieval by examinees within a predetermined time period.

5. The system of claim 2 wherein said essay response administrative system automatically calculates any compensation for the graders based upon the number of constructed responses the grader has evaluated.

6. The system of claim 2 wherein said automated grading system enables a grader leader to view the overall evaluation and pre-defined additional comments of the grader and assign a new overall evaluation and pre-defined additional comments if desired.

7. The system of claim 2 wherein said evaluation delivery system releases to the examinee for the examinee'"'"'s retrieval at least one exemplary essay response, said overall evaluation, and said pre-defined additional comments.

8. The system of claim 2 wherein said essay response administrative system includes software which:

identifies the number of essay responses which need to be evaluated in an upcoming time period;

identifies the number of essay responses which are scheduled to be evaluated in the upcoming time period;

if the number of essay responses that need to be evaluated in the upcoming time period is less than the number of essay responses that are scheduled to be evaluated in the upcoming time period, identifies the number of essay responses which need to be evaluated in the upcoming time period; and

if the number of essay responses that need to be evaluated in the upcoming time period is greater than the number of essays that are scheduled to be evaluated in the upcoming time period, electronically notifies backup graders of the need to provide assistance in the upcoming time period.

9. The system of claim 2 wherein said essay response administrative system includes software which:

identifies the date and time an essay response was submitted;

identifies whether the essay response is to be graded on a regular schedule or a rush schedule;

calculates a date for which an evaluation of the essay response is due based upon the date the essay response was submitted and whether the essay response is to be evaluated on a regular schedule or a rush schedule; and

places the essay response in a queue of essay responses that is ordered by the date on which an evaluation of each response is due.

10. The system of claim 2 wherein said essay response administrative system includes software which:

identifies a grader;

identifies the total number of essay responses read by the grader; and

calculates the compensation for the grader as function of the number of essay responses read.

11. The system of claim 2 wherein said essay response administrative system includes software which:

identifies that a grader has submitted an overall evaluation and score for an essay response;

forwards the essay response and overall evaluation to a grader leader for review; and

if the overall evaluation and score are not consistent with scoring and evaluation standards, allows grader leader to assign a new score and evaluation, so long as the supervisory grader performs the review in a predetermined time period.

12. The system of claim 2 wherein said automated grading system presents an essay response to a grader so that the grader may evaluate the essay response;

and presents evaluation categories to the grader so that the grader may select an overall evaluation.

13. The system of claim 2 wherein said evaluation delivery system groups the examination question, scoring guidelines, sample essay responses, the overall evaluation, and pre-defined additional comments into a feedback package;

stores said feedback package; and

releases said feedback package for retrieval by the examinee.

14. An Internet based system for submitting and evaluating essay responses to examination questions, comprising:

a server computer which forwards examination questions to examinees and accepts essay responses to said examination questions for evaluation, said server being connected to the Internet;

a first computer connected to said server computer so as to permit examinees to set up an account for purchasing access to an examination question and evaluation of an essay response submitted in response to said examination question; and

a second computer connected to said server computer so as to permit graders to read an essay response that was submitted by an examinee, evaluate the essay response by selecting an overall comment and at least one pre-defined feedback comment, store said overall comment and said at least one re-defined feedback comments, and release said overall comment and said at least one pre-defined feedback comment for retrieval by the examinee via said server computer.

15. A method for on-line evaluation of constructed responses to an examination question over a system having a server computer connected to the Internet, a first computer electronically connected to said server computer, and second computer connected to said server computer, comprising the following steps:

an examinee accessing the server computer from said first computer to set up an account for purchasing access to an examination question and evaluation of an essay response which is submitted in response to said examination question;

an examinee accessing the server computer from said first computer to submit an essay response in response to said examination question;

accepting said essay responses at the server computer; and

a grader accessing the server computer from the second computer to read the essay response submitted by the examinee, evaluate the essay response by providing an overall comment and at least one pre-defined feedback comment, storing said overall comment and said at least one pre-defined feedback comment, and releasing said overall comment and said at least one pre-defined feedback comment for retrieval by the examinee.