System and method for computer based creation of tests formatted to facilitate computer based testing

• US 7,845,950 B2
• Filed: 10/18/2004
• Issued: 12/07/2010
• Est. Priority Date: 03/27/1997
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Tyler, Linda Louise, William, Kenneth, Driscoll, Gary F., Wichert, Victor E., SooHoo, Paul, Keener, Barbara, Burns, Richard, Snieckus, Alison H., Sanchez-Lazer, Teresa M.
• Primary Examiner(s): Saadat; Cameron
• Application Number:
  US10/967,642
• Publication Number:
  US 20050153269A1
• Time in Patent Office: 2,241 Days
• Field of Search: 434/323, 434/350, 434/362, 434/322
• US Class Current: 434/322
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A computer-based test creation system and method for generating test items. The test creation process includes creating items in electronic form and allowing access by a plurality of reviewers via a network. The reviewers may review the item for content, fairness and format, and collaborate with each other before indicating that the item is acceptable for inclusion in a test. After the administration of the items to test takers, item performance may be evaluated on-line by reviewing electronic feedback from the administrations and creating electronic evaluation reports. Since the items are created in an electronic form, the items may further include new forms of stimuli such as graphics and multimedia.

Claims

1. A method for generating a test item in electronic form using an automated test creation system, comprising the steps of:

electronically associating components of a test item into an extensible markup language based electronic item form using a test creation system;

storing the electronic item form in an item repository of the test creation system;

any of a plurality of test creators accessing the electronic item form from the item repository and utilizing the test creation system to review the content and format of the electronic item form and to update the content and format of the electronic item form as appropriate, including reviewing the electronic item form in sequence for content, fairness, and format by at least one reviewer for each of content, fairness, and format, respectively, where reviewing for fairness includes guidelines that pertain to words and phrases that are inappropriate for a population, the method further comprising the steps of;

when a content reviewer has completed the content review, automatically routing the electronic item form to a fairness reviewer;

when the fairness reviewer has completed the fairness review, automatically routing the electronic item form to a format reviewer; and

when the format reviewer has completed the format review, automatically routing the electronic item form to a final resolver for a final review to determine whether the content, fairness, and format are acceptable for test assembly;

storing the updated electronic item form in the item repository as a locked item when the content and format are deemed acceptable by the test creators for inclusion in a test, where a locked item is not to be further reviewed by any of said test creators.

2. The method of claim 1, further comprising administering the test.

3. A method for generating a test item in electronic form using an automated test creation system, comprising the steps of:

electronically associating components of test item into an extensible markup language based electronic item form using a test creation system;

storing the electronic item form in an item repository of the test creation system;

any of a plurality of test creators accessing the electronic item form from the item repository and utilizing the test creation system to review the content and format of the electronic item form and to update the content and format of the electronic item form as appropriate;

storing the updated electronic item form in the item repository as a locked item when the content and format are deemed acceptable by the test creators for inclusion in a test, where a locked item is not to be further reviewed by any of said test creators;

wherein a number of reviewers are assigned to review the content, fairness and format of the electronic item form, where reviewing for fairness includes guidelines that pertain to words and phrases that are inappropriate for a population, and wherein at least two of the reviewers electronically collaborate concerning at least one of the content, fairness, and format of the electronic item form.

4. The method of claim 3, including the additional steps of reviewing statistical information obtained after an administration of said test item and, if said test item does not meet predetermined performance criteria, initiating an electronic evaluation report for said test item.

5. The method of claim 4, including the additional steps of providing a statistical flag which, when set, automatically initiates said electronic evaluation report,

identifying problems in said test item;

and

resolving said problems.

6. The method of claim 3, further comprising administering the test.

7. A method for generating a test item in electronic form using an automated test creation system, comprising the steps of:

electronically associating components of a test item into an extensible markup language based electronic item form using a test creation system;

storing the electronic item form in an item repository of the test creation system;

any of a plurality of test creators accessing the electronic item form from the item repository and utilizing the test creation system to review the content and format of the electronic item form and to update the content and format of the electronic item form as appropriate;

storing the updated electronic item form in the item repository as a locked item when the content and format are deemed acceptable by the test creators for inclusion in a test, where a locked item is not to be further reviewed by any of said test creators; and

monitoring the progress and checking the status of electronic item forms in the test creation system by measuring metrics statistics and management statistics for said electronic item forms, said metrics statistics calculated by quality and count, and said management statistics calculated by said count, wherein said count measures the number of items at a particular stage in a particular subsystem, and said quality measures a percentage of at least one of;

locked items;

pre-test items;

items with problems; and

specific item problems present in all problems out of an overall group of items that could potentially be designated as a locked item.

8. The method of claim 7, further comprising administering the test.