System and method for computer based test creation

• US 6,259,890 B1
• Filed: 03/27/1997
• Issued: 07/10/2001
• Est. Priority Date: 03/27/1997
• Status: Expired due to Term

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Willian, Kenneth, SooHoo, Paul, Keener, Barbara, Burns, Richard, Driscoll, Gary, Tyler, Linda, Snieckus, Alison, Sanchez-Lazer, Teresa, Wichert, Victor
• Primary Examiner(s): Martin-Wallace, Valencia
• Assistant Examiner(s): Harris, Chanda
• Application Number:
  US08/826,417
• Time in Patent Office: 1,566 Days
• Field of Search: 434/350, 434/118, 434/322
• US Class Current: 434/350
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A computer-based test creation system and method for generating test items. The test creation process includes creating items in electronic form and allowing access by a plurality of reviewers via a network. The reviewers may review the item for content, fairness and format, and collaborate with each other before indicating that the item is acceptable for inclusion in a test. After the administration of the items to test takers, item performance may be evaluated on-line by reviewing electronic feedback from the administrations and creating electronic evaluation reports. Since the items are created in an electronic form, the items may further include new forms of stimuli such as graphics and multimedia.

Claims

1. A method for generating a test item in electronic form using an automated test creation system, comprising the steps of:

electronically associating components of the test item into an electronic item form using the test creation system;

storing the electronic item form in an item repository of the test creation system;

any of a plurality of test creators accessing the electronic item form from the item repository and utilizing the test creation system to review the content and format of the electronic item form and to update the content and format of the electronic item form as appropriate; and

storing the updated electronic item form in the item repository as a locked item when the content and format are deemed acceptable by the test creators for inclusion in a test, where a locked item is not to be further reviewed by any of said test creators.

2. The method of claim 1, wherein the components of the test item include a written text representative of a test question and an answer key.

3. The method of claim 2, wherein the components of the test item further includes a stimulus to which the test question refers.

4. The method of claim 1, wherein the step of electronically associating components of the test item comprises the steps of:

providing the test creation system with identification information relating to a selected test program, a selected test section, a selected delivery mode, and a selected item type for which the electronic item form is being created;

retrieving an item template based on the information so provided from a template storage facility of the test creation system;

completing the item template by entering written text representative of a test question, an answer key, and a stimulus to which the test question refers; and

releasing the item template so completed to the item repository where the written text, stimulus, and answer key are electronically associated as the electronic item form.

5. The method of claim 4, wherein the stimulus comprises at least one of:

written text, graphics, video, audio, audiovisual material, animation and scanned images.

6. The method of claim 5, further comprising the steps of:

accessing an on-line source for locating stimulus material via a communications interface of the test creation system;

selecting certain stimulus material; and

importing the selected stimulus material into the item template as the stimulus being associated with the written text and answer key as the electronic item form.

7. The method of claim 4, wherein the stimulus comprises graphics, and the method further comprises the steps of:

accessing an object database of the test creation system having stored therein predefined graphical objects;

selecting one of the predefined graphical objects; and

importing the one selected graphical object into the item template as the stimulus being associated with the written text and answer key as the electronic item form.

8. The method of claim 4, wherein the stimulus comprises graphics, and the method further comprises the steps of:

generating a graphical object using a drawing application supported by the test creation system; and

importing the graphical object into the item template as the stimulus being associated with the written text and answer key as the electronic item form.

9. The method of claim 1, wherein the electronic item form is reviewed in said accessing and utilizing step in sequence for content, fairness, and format by at least one reviewer for each of content, fairness, and format, respectively, the method further comprising the steps of:

when a content reviewer has completed the content review, automatically routing the electronic item form to a fairness reviewer;

when the fairness reviewer has completed the fairness review, automatically routing the electronic item form to a format reviewer; and

when the format reviewer has completed the format review, automatically routing the electronic item form to a final resolver for a final review to determine whether the content, fairness, and format are acceptable for test assembly.

10. The method of claim 9, wherein a number of reviewers are assigned to review the content, fairness and format of the electronic item form, and wherein at least two of the reviewers electronically collaborate concerning at least one of the content, fairness, and format of the electronic item form.

11. The method of claim 9, further comprising the step of generating a workflow log listing the progress of each electronic item form as it is reviewed by each of the reviewers.

12. The method of claim 1, including the additional steps of reviewing statistical information obtained after an administration of said test item and, if said test item does not meet predetermined performance criteria, initiating an electronic evaluation report for said test item.

13. The method of claim 12, including the additional step of providing a statistical flag which, when set, automatically initiates said electronic evaluation report.

14. The method of claim 1, further comprising the step of automatically initiating copyright approval for said test item before the test item is stored as a locked item by initiating a process of automatically initiating copyright approval for the test item during review of the electronic item form.

15. The method of claim 1, including the additional steps of monitoring the progress and checking the status of electronic item forms in the test creation system by measuring metrics statistics and management statistics for said electronic item forms.

16. The method of claim 15, comprising the further steps of calculating the metrics statistics by cycle time, quality, and count, and calculating the management statistics by said cycle time and said count, wherein said count measures the number of items at a particular stage in a particular subsystem, said cycle time measures a number of days a particular item has been at a particular stage in a particular subsystem of said test creation system, and said quality measures a percentage of at least one of locked items, pre-test items, items with problems, or specific item problems, and present in all problems out of an overall group of items that could potentially be designated as a locked item.

17. An automated method of evaluating performance of a test item after an administration of the test item to test takers, comprising the steps of:

electronically compiling feedback from the administration of the test item;

electronically reviewing the feedback from the administration of the test item;

identifying a problem in the test item from the reviewed feedback; and

resolving the problem in the test item, if present.

18. The method of claim 17, wherein after a test item has been identified as having a problem and the problem has been resolved, the method further comprises the step of modifying an electronic representative of the test item so as to resolve said problem.

19. The method of claim 17, wherein the feedback from the administration of said test item comprises:

(a) statistical information;

(b) feedback from test takers comprising at least one of;

(i) student irregularity reports, and

(ii) candidate inquiries; and

(c) reader evaluations.

20. The method of claim 19, wherein an electronic evaluation report is initiated or information is added to an existing electronic evaluation report based on at least one of said feedback from test takers and said reader evaluations.

21. An automated test creation system, comprising:

an item repository for storing electronic items for use in creation of a test for administration to test takers;

an item creation and review subsystem for creating new electronic items for storage in said item repository;

an item migration and review subsystem for converting preexisting items into converted electronic items and storing the converted electronic items in the item repository;

an analysis feedback review subsystem for identifying electronic items that do not meet predetermined performance criteria after the administration of said items to test takers; and

an item performance evaluation subsystem for evaluating the performance of electronic items after said administration of said items to test takers, where said electronic items are modified by the item performance evaluation subsystem based on the performance of said electronic items and said modified electronic items stored in said item repository for use in creation of said test.

22. The system of claim 21, wherein said preexisting items comprise paper-and-pencil items, items created in whole or in part by a different test creation system, and electronic test items created for computer-based test administrations.

23. The test creation system of claim 22, wherein the item creation and review subsystem classifies the new electronic items and the item migration and review subsystem classifies the converted electronic items so that said new electronic items and said converted electronic items are uniformly accessible from said item repository for the creation of said test.

24. The test creation system of claim 23, wherein the item creation and review subsystem and the migration and review subsystem classify the new electronic items and converted electronic items, respectively, according to criteria of classification, sub-classification and keywords for subject, and stimulus description and context for each electronic item, said criteria being based on identification information relating to a selected test program, a selected test section, and a selected delivery mode for each said electronic item.

25. The test creation system of claim 21, wherein the item creation and review subsystem includes means for creating a plurality of multi-media item types.

26. The test creation system of claim 25 wherein the plurality of multi-media item types comprise at least one of written items, graphical items, braille items, video items, audio items, audiovisual items, animation items, and any combination thereof.

27. The test creation system of claim 21, further comprising a plurality of workstations connected to a network so as to enable a plurality of reviewers to electronically collaborate on any aspect of an electronic item.

28. The test creation system of claim 27, wherein a number of reviewers are assigned to review the content, fairness and format of each electronic item, and wherein at least some of the reviewers electronically collaborate concerning at least one of the content, fairness, and format of the electronic item via said network.

29. The test creation system of claim 21, wherein said analysis feedback review subsystem uses statistical information for each administered electronic item to identify the electronic items that do not meet the predetermined performance criteria and to initiate electronic evaluation reports for those electronic items that do not meet the predetermined performance criteria.

30. The test creation system of claim 29, further comprising a plurality of statistical flags which, when set, automatically initiate the electronic evaluation reports.

31. The test creation system of claim 21, wherein the item performance evaluation subsystem comprises means for compiling and reviewing feedback from test administrations of the electronic items for evaluating the performance of said electronic items, identifying problems in said electronic items, and resolving said problems.

32. The test creation system of claim 31, wherein the feedback from the test administrations of the electronic items comprises at least one of statistical information, feedback from test takers comprising at least one of student irregularity reports and candidate inquiries, and reader evaluations.

33. The test creation system of claim 32, wherein electronic evaluation reports are initiated or information is added to existing evaluation reports based on said feedback from test takers and said reader evaluations.

34. The test creation system of claim 21, further comprising an object database for storing predefined graphical objects for use in said electronic items.

35. The test creation system of claim 21, further comprising an automated metrics and management subsystem for monitoring the progress and checking the status of the electronic items in terms of metrics statistics and management statistics.

36. The test creation system of claim 35, wherein the metrics statistics are calculated by cycle time, quality, and count, and the management statistics are calculated by said cycle time and said count, wherein said count measures the number of electronic items at a particular stage in a particular subsystem, said cycle time measures the number of days a particular electronic item has been at a particular stage in a particular subsystem, and quality measures a percentage of at least one of locked items, pre-test items, items with problems, and specific item problems present in all problems out of an overall group of electronic items that could potentially be used in creation of a test.

37. The test creation system of claim 21, further comprising a copyright database which is accessed to automatically initiate copyright approval for an electronic item by the time the electronic item is ready for administration in a test.

38. The test creation system of claim 21, further comprising a test form database for storing and providing information regarding a test form used for a test administration.

39. The test creation system of claim 21, further comprising a test type database for storing and providing generic information regarding a test including a number of sections, types of sections, medium for the test, and allowable response types.

40. The test creation system of claim 21, further comprising a categories database for storing and providing information on each electronic item including dimensions, category, structural category, and workflow.

41. The test creation system of claim 21, further comprising a statistics database for storing and providing statistical information on each electronic item including at least one of:

its history, differential item performance, item analysis, and item response theory.

42. The test creation system of claim 21, wherein the item performance evaluation subsystem further comprises means for providing an alert notification if an electronic item in a test about to be administered is found to not meet said predetermined performance criteria.