System and method for computer based testing

• US 5,565,316 A
• Filed: 06/22/1993
• Issued: 10/15/1996
• Est. Priority Date: 10/09/1992
• Status: Expired due to Term

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Kershaw, Roger C., Ward, William C. Jr., Romano, Frank J., Swanson, Leonard C.
• Primary Examiner(s): Cheng, Joe H.
• Application Number:
  US08/082,058
• Time in Patent Office: 1,211 Days
• Field of Search: 434/118, 434/307 R, 434/308, 434/322, 434/323, 434/327, 434/347, 434/350, 434/353, 434/362, 364/419.08, 364/419.19, 364/419.2, 395/100, 395/118, 395/144, 395/152, 395/327, 395/154
• US Class Current: 434/322
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: Y10S 706/927 Education or instruction

Abstract

The present invention provides a system and method for computer based testing. The system of the present invention comprises a test development system for producing a computerized test, a test delivery system for delivering the computerized test to an examinee, and a workstation on which the computerized test is delivered to the examinee. The method of the present invention comprises producing a computerized test, delivering the computerized test to an examinee and recording examinee responses to questions presented to the examinee during the delivery of the computerized test. A method of producing a computerized test is also provided. This method comprises preparing a test document of items, computerizing each item, preparing scripts and other test components and packaging the items, scripts and other test components together to form the computerized test. A method of delivering a computerized test is also provided in which a standardized test is created, an electronic form of the test is then prepared, the items of the test are presented to an examinee on a workstations display and the examinee'"'"'s responses are accepted and recorded. A method of administering a computerized test is further provided in which a computerized test is installed on a workstation and then the delivery of the test to an examinee is initiated. A data distribution system is further provided. The system comprises an examinee performance database and a file processing component for receiving files from the computer based testing system and updating the examinee performance database with information from these files.

Claims

1. A computer based testing system comprising:

a test development system for creating a computerized test having a number of questions to be answered by an examinee, a plurality of related test screens containing messages and directions providing information for the examinee, and a computerized test script defining rules for determining a sequence of questions and related test screens to be presented;

a workstation operable to present the questions to at least one said examinee and operable to accept examinee responses to the questions so presented; and

a test delivery system operatively coupled to both said test development system and said workstation for delivering said computerized test by presenting the questions and related test screens according to the rules defined by the test script to each said examinee on said workstation.

2. The system of claim 1, further comprising:

a test administration system operatively coupled to said test development system, said test delivery system and said workstation for initiating and terminating the delivery of said computerized test by said test delivery system.

3. The system of claim 2, wherein said test administration system further comprises security means for securing said computerized test from access by unauthorized persons.

4. The system of claim 2, wherein a plurality of events indicative of errors of said computer based testing system and related to security of said computerized test and said computer based testing system are monitored by said test administration system, and a plurality of predetermined events occurring during the delivery of said computerized test are monitored by said test delivery system, each event and related information being stored within a log record, a combination of said log records storing events related to errors defining an error log file, a combination of said log records storing events related to the security of said computerized tests and computer based testing system defining a security log file, and a combination of said log records storing events occurring during the delivery of said computerized test defining an examinee performance file.

5. The system of claim 4, wherein said workstations are located at at least one test center, the system further comprising:

a data distribution system having an interface with said test administration system for processing said error log files, said security log files and said examinee performance files and for generating from these files reports indicative of events occurring at each workstation and at each of said test centers.

6. The system of claim 1, wherein said test development system further comprises:

a test document creation means for defining the questions to be presented to the examinee based on a test specification and for determining a manner of presentation of the questions on said workstation during the delivery of said computerized test;

an item preparation system interfaced with said test document creation means for computerizing the questions for delivery by said test delivery system;

a test preparation system interfaced with said test document creation means and said item preparation system for preparing the computerized test script and the related test screens; and

a test packaging utility interfaced with said item and test preparation systems for combining the questions, the test script and the related test screens to form the computerized test.

7. The system of claim 6, wherein the test specification specifies a number of test sections of the computerized test in which the questions are assigned, each test script defining the rules for determining the sequence of questions and related test screens to be presented in one of said test sections and wherein said test preparation system is operative to create, and the test delivery system is operative to deliver, at least one of:

tutorials for providing instructions to the examinee for interacting with said workstation during the delivery of said computerized test;

directions for explaining how to take said computerized test;

a help facility selectable by the examinee for accessing additional instructions describing how to interact with said workstation and for accessing said directions for explaining how to take the computerized test;

a review facility selectable by the examinee during the delivery of said computerized test for providing a list of substantially all of the questions of said computerized test and for allowing the examinee to select a specific question from the list to be presented; and

said test preparation system creating a session script defining a sequence in which the tutorials, the directions, and the test scripts are to be invoked by the test delivery system and wherein the test delivery system is operable to interrupt the sequences defined by the session script and the sequences determined based on the rules defined by the test scripts to invoke the help and review facilities upon selection by the examinee.

8. The system of claim 1, wherein a plurality of predetermined events are monitored by said test delivery system, each event and related information being stored within a log record, a combination of log records created during the delivery of said computerized test defining a performance file.

9. The system of claim 8, wherein said events include said examinee responses, the system further comprising:

a data distribution system having an interface with said test delivery system for formatting said performance files so that said examinee responses can be evaluated.

10. The system of claim 1, wherein said computerized test is a standardized test.

11. The system of claim 1, wherein said workstations are located at at least one test center, the system further comprising:

a data distribution system located at a central processing site remote from said test centers, said data distribution system having a data connection with each said test center for receiving information related to activities at the test centers and information indicative of said examinee responses and for processing said information to generate at least one report presenting said information in human readable form and for formatting at least some of said information for subsequent scoring of examinee responses.

12. The system of claim 1, wherein the rules for determining the sequence of questions to be presented are based on a measurement model for at least one of a linear test, an adaptive test, a randomized parallel test, and a mastery test.

13. A method of computer based testing, comprising the steps of:

producing a computerized test having a plurality of questions to be answered by an examinee, a plurality of related test screens containing messages and directions comprising information for the examinee, and a computerized test script defining rules for determining a sequence of the questions and related test screens to be presented;

delivering said computerized test to the examinee by presenting said plurality of questions and related test screens to the examinee at a computer workstation according to the rules defined by the test script; and

recording responses provided by the examinee to the questions presented.

14. The method of claim 13, further comprising the step of:

administering at least one said computerized test to at least one examinee.

15. The method of claim 13, further comprising the step of:

securing said computerized test and responses from access by an unauthorized person.

16. The method of claim 13, wherein said step of delivering said computerized test further comprises the step of:

providing user controls selectable by the examinee for controlling said delivery of said computerized test.

17. The method of claim 13, wherein said workstation comprises a memory, the method further comprising the steps of:

registering at least some examinees to take a specified computerized test prior to the delivery of the computerized test, each examinee so registered being authorized to take said specified computerized test;

storing at least one specified computerized test in said memory prior to delivery of said specified computerized test.

18. The method of claim 17, further comprising the step of:

safeguarding each computerized test stored in said memory from access by unauthorized persons.

19. The method of claim 17, further comprising the step of:

confirming the authorization of each examinee seeking to take said specified computerized test prior to delivering said specified computerized test.

20. The method of claim 17, further comprising the steps of:

selecting one specified computerized test stored in memory to be delivered, the test selected being said specified computerized test the examinee is authorized to take; and

initiating the delivery of said computerized test so selected.

21. The method of claim 13, wherein said computerized test is standardized and wherein said computerized test is based on a test specification being defined independent of said test development system used to produce the computerized test.

22. The method of claim 21, wherein the test specification specifies a number of test sections of the computerized test in which the questions are assigned, each test script defining rules for determining the sequence of questions and related test screens to be presented in one of said test sections, wherein said step of delivering said computerized test further comprises at least one of the following steps:

displaying tutorials to the examinee for instructing the examinee how to interact with said workstation;

displaying directions related to the questions to explain how the examinee is intended to respond to each question presented;

providing a help facility selectable by the examinee to access instructions describing how to interact with said workstation and for accessing directions describing how to respond to questions presented; and

providing a review facility selectable by the examinee to review questions presented by providing a list of substantially all questions included in said computerized test and for allowing the examinee to return and to advance to a question listed; and

said method further comprising the steps of;

creating a session script defining a sequence in which the tutorials, the directions, and the test scripts are to be invoked; and

interrupting the sequence defined by the session script and the sequence determined based on the rules defined by the test scripts to invoke the help and review facilities upon selection by the examinee.

23. The method of claim 13, wherein each said workstation is located at a test center.

24. The method of claim 23, further comprising the steps of:

transferring information indicative of activities occurring at each said test center from said test centers to a central processing site being remote from said test centers; and

processing said transferred information so that said activities at said test centers are monitored at said central processing site.

25. The method of claim 23, further comprising the steps of:

transferring said recorded responses from each workstation to a central processing site being remote from said test centers; and

formatting said recorded responses for subsequent scoring.

26. The method of claim 13, wherein the rules for determining the sequence of questions to be presented are based on a measurement model for at least one of a linear test, an adaptive test, a randomized parallel test, and a mastery test.

27. A method of producing a computerized test, comprising the steps of:

preparing a test document of items, each item having at least a test question, related presentation information indicative of at least the text to be presented to an examinee and a response type associated with said item;

preparing an electronic form of each item according to said presentation information and said response type and defining said electronic form so prepared as a computerized item;

preparing at least one session script having information indicative of predetermined test options and information indicative of a sequence in which said computerized items are to be presented; and

packaging said computerized items and said session scripts together to form said computerized test.

28. The method of claim 27, wherein computerizing each item comprises the steps of:

editing the items according to the presentation information using a word processor; and

inserting custom codes into the items according to both the presentation information and the response type.

29. The method of claim 27, wherein the items further comprise stimulus material to be referenced by the items, the method further comprising the step of:

demarcating blocks of the stimulus material by inserting custom codes into the stimulus material;

said insertion of the custom codes causing the demarcated blocks to be one of i) highlighted upon presentation to the examinees, and ii) selectable by the examinees.

30. The method of claim 27, wherein each computerized test has at least one test section, each test section having at least one item, further comprising the steps of:

preparing for each said section, at least one delivery unit defining options to be effective for that said section and for specifying an organization of said computerized items to be presented within that said section.

31. The method of claim 30, further comprising the steps of:

preparing at least one of the following units;

(1) a general information screen (GIS) unit having information related to at least one of said test, said section, and said item, and

(2) a tutorial unit having information to familiarize examinees with said computerized test;

combining GIS units, tutorial units and delivery units to form a testing unit; and

generating a test script defining a sequence in which said GIS units, tutorial units and delivery units are to be presented, at least one test script being included within each of said session scripts.

32. The method of claim 31, further comprising the steps of:

preparing at least one of the following units;

(1) an examinee data collection unit having at least one inquiry to be presented to examinees to obtain information about said examinees, and

(2) a scoring and reporting unit having information related to scoring and reporting of at least one computerized test; and

defining a sequence in which said examinee data collection units, scoring and reporting units, testing units, GIS units, and tutorial units are to be presented, said session script being based on said sequence.

33. The method of claim 27, further comprising the step of:

generating computerized graphics to be presented with the computerized items.

34. The method of claim 27, wherein each item has a key indicative of a correct response to the test question associated therewith, the method further comprising the step of:

computerizing the key; and

packaging the computerized key with the computerized test.

35. The method of claim 27, further comprising the step of:

presenting each computerized item for review; and

modifying the computerized item based on said presentation.

36. A computer based testing system comprising:

a test development system for creating a computerized test for standardized testing, said computerized test having a number of questions to be answered by an examinee and being based on a test specification which specifies at least one of the following information;

the number of questions to be presented in the computerized test, the number of test sections in which the questions are grouped, the number of questions in each section, an amount of time for taking the test, an amount of time allotted for responding to each of the questions, and criteria for selecting questions to be presented, said test specification being defined independent of said test development system;

a workstation operable to present the questions to at least one said examinee and operable to accept examinee responses to the questions so presented; and

a test delivery system operatively coupled to both said test development system and said workstation for delivering said computerized test to each said examinee on said workstation in a manner independent of the information specified by the test specification.

37. The system of claim 36, further comprising:

a test administration system operatively coupled to said test development system, said test delivery system and said workstation for initiating and terminating the delivery of said computerized test by said test delivery system.

38. The system of claim 37, wherein said test administration system further comprises security means for securing said computerized test from access by unauthorized persons.

39. The system of claim 37, wherein a plurality of events indicative of errors of said computer based testing system and related to security of said computerized test and said computer based testing system are monitored by said test administration system, and a plurality of predetermined events occurring during the delivery of said computerized test are monitored by said test delivery system, each event and related information being stored within a log record, a combination of said log records storing events related to errors defining an error log file, a combination of said log record storing events related to the security of said computerized tests and computer based testing system defining a security log file, and a combination of said log records storing events occurring during the delivery of said computerized test defining an examinee performance file.

40. The system of claim 39, wherein said workstations are located at at least one test center, the system further comprising:

a data distribution system having an interface with said test administration system for processing said error log files, said security log files and said examinee performance files and for generating from these files reports indicative of events occurring at each workstation and at each of said test center.

41. The system of claim 36, wherein said test development system further comprises:

a test document creation means for defining the questions to be presented to the examinee and for determining a manner of presentation of the questions on said workstation during the delivery of said computerized test;

an item preparation system interfaced with said test document creation means for computerizing the questions for delivery by said test delivery system;

a test preparation system interfaced with said test document creation means and said item preparation system for preparing a computerized test script defining a sequence of the questions to be presented and for preparing related test screens containing of messages and directions to be presented during the delivery of the computerized test; and

a test packaging utility interfaced with said item and test preparation systems for combining the test script and related test screens to form the computerized test.

42. The system of claim 41, wherein each test script defines rules for determining the sequence of questions and related test screens to be presented in one of said test sections and wherein said test preparation system is operative to create and said test delivery system is operative to deliver at least one of:

tutorials for providing instructions to the examinee for interacting with said workstation during the delivery of said computerized test;

directions for explaining how to take said computerized test;

a help facility selectable by each examinee for accessing additional instructions describing how to interact with said workstation and for accessing said directions for explaining how to take the computerized test;

a review facility selectable by each examinee during the delivery of said computerized test for providing a list of substantially all of the questions of said computerized test and for allowing the examinee to select a specific question from the list to be presented; and

said test preparation system creating a session script defining a sequence in which the tutorials, the directions, and the test scripts are to be invoked by the test delivery system and wherein the test delivery system is operable to interrupt the sequence defined by the session script and the sequences determined based on the rules defined by the test scripts to invoke the help and review facilities upon selection by the examinee.

43. The system of claim 42, wherein the rules for determining the sequence of questions to be presented are based on a measurement model for at least one of a linear test, an adaptive test, a randomized parallel test, and a mastery test.

44. The system of claim 36, wherein a plurality of predetermined events are monitored by said test delivery system, each event and related information being stored within a log record, a combination of log records created during the delivery of said computerized test defining a performance file.

45. The system of claim 44, wherein said events include said examinee responses, the system further comprising:

a data distribution system having an interface with said test delivery system for formatting said performance files so that said examinee responses can be evaluated.

46. The system of claim 36, wherein said workstations are located at at least one test center, the system further comprising:

a data distribution system located at a central processing site remote from said test centers, said data distribution system having a data connection with each said test center for receiving information related to activities at the test centers and information indicative of said examinee responses and for processing said information to generate at least one report presenting said information in human readable form and for formatting at least some of said information for subsequent scoring of examinee response.

47. A method of computer based testing, comprising the steps of:

producing a computerized test for standardized testing using a test development system, said computerized test having a plurality of questions to be answered by an examinee and being based on a test specification which specifies at least of the following information;

the number of questions to be presented in the computerized test, the number of test sections in which the questions are grouped, the number of questions in each section, an amount of time for taking the test, an amount of time allotted for responding to each of the questions, and criteria for selecting questions to be presented, said test specification being defined independent of said test development system;

delivering said computerized test to the examinee by presenting said plurality of questions to the examinee at a computer workstation in a manner independent of the information specified by the test specification; and

recording responses provided by the examinee to the questions presented.

48. The method of claim 47, further comprising the step of:

administering at least one said computerized test to at least one examinee.

49. The method of claim 47, further comprising the step of:

securing said computerized test and responses from access by an unauthorized person.

50. The method of claim 47, wherein said computerized test comprises a plurality of related test screens containing messages and directions providing information for the examinee and a computerized test script defining rules for determining a sequence of the questions and related test screens to be presented and wherein said step of delivering said computerized test further comprises at least one of the following steps:

displaying tutorials to the examinee for instructing the examinee how to interact with said workstation;

displaying directions related to the questions to explain how the examinee is intended to respond to each question presented;

providing a help facility selectable by the examinee to access instructions describing how to interact with said workstation and for accessing directions describing how to respond to questions presented; and

providing a review facility selectable by the examinee to review questions presented by providing a list of substantially all questions included in said computerized test and for allowing the examinee to return and to advance to a question listed; and

said method further comprising the steps of;

creating a session script defining a sequence in which the tutorials, the directions, and the test scripts are to be invoked; and

interrupting the sequence defined by the session script and the sequences determined based on the rules defined by the test scripts to invoke the help and review facilities upon selection by the examinee.

51. The method of claim 50, wherein the rules for determining the sequence of questions to be presented are based on a measurement model for at least one of a linear test, an adaptive test, a randomized parallel test, and a mastery test.

52. The method of claim 47, wherein said step of delivering said computerized test further comprises the step of:

providing user controls selectable by the examinee for controlling said delivery of said computerized test.

53. The method of claim 47, wherein said workstation comprises a memory, the method further comprising the steps of:

registering at least some examinees to take a specified computerized test prior to the delivery of the computerized test, each examinee so registered being authorized to take said specified computerized test;

storing at least one specified computerized test in said memory prior to delivery of said specified computerized test.

54. The method of claim 53, further comprising the step of:

safeguarding each computerized test stored in said memory from access by unauthorized persons.

55. The method of claim 53, further comprising the step of:

confirming the authorization of each examinee seeking to take said specified computerized test prior to delivering said specified computerized test.

56. The method of claim 53, further comprising the steps of:

selecting one specified computerized test stored in memory to be delivered, the test selected being said specified computerized test the examinee is authorized to take; and

initiating the delivery of said computerized test so selected.

57. The method of claim 47, wherein each said workstation is located at a test center.

58. The method of claim 57, further comprising the steps of:

transferring information indicative of activities occurring at each said test center from said test centers to a central processing site being remote from said test centers; and

processing said transferred information so that said activities at said test centers are monitored at said central processing site.

59. The method of claim 57, further comprising the steps of:

transferring said recorded responses from each workstation to a central processing site being remote from said test centers; and

formatting said recorded responses for subsequent scoring.