Computerized mastery testing system, a computer administered variable length sequential testing system for making pass/fail decisions

• US 5,059,127 A
• Filed: 10/26/1989
• Issued: 10/22/1991
• Est. Priority Date: 10/26/1989
• Status: Expired due to Term

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Lewis, Charles, Sheehan, Kathleen M., DeVore, Richard N., Swanson, Leonard C.
• Primary Examiner(s): Apley, Richard J.
• Assistant Examiner(s): Cheng, Joe H.
• Application Number:
  US07/428,383
• Time in Patent Office: 726 Days
• Field of Search: 434/118, 434/322, 434/327, 434/335, 434/353, 434/362, 434/236, 364/419, 364/200, 364/900 MS File
• US Class Current: 434/353
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/04 characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying a further explanation

Abstract

A computerized mastery testing system providing for the computerized implementation of sequential testing in order to reduce test length without sacrificing mastery classification accuracy. The mastery testing system is based on Item Response Theory and Bayesian Decision Theory which are used to qualify collections of test items, administered as a unit, and determine the decision rules regarding examinee'"'"'s responses thereto. The test item units are randomly and sequentially presented to the examinee by a computer test administrator. The administrator periodically determines, based on previous responses, whether the examinee may be classified as a nonmaster or master or whether more responses are necessary. If more responses are necessary it will present as many additional test item units as required for classification. The method provides for determining the test specifications, creating an item pool, obtaining IRT statistics for each item, determining ability values, assembling items into testlets, verifying the testlets, selecting loss functions and prior probability of mastery, estimating cutscores, packaging the test for administration, randomly and sequentially administering testlets to the examinee until a pass/fail decision can be made.

Claims

1. A method of computerized mastery testing for deciding whether an examinee has attained a specified mastery of a given subject by using a computer system means, said method comprising:

a. determining test specifications;

b. creating a pool of test items which meet the test specifications;

c. determining the dimensionally of the pool of test items;

d. obtaining item response theory (IRT) statistics and assessing IRT model fit for each test item;

e. determining mastery ability values;

f. assembling the test items into testlets;

g. verifying all of the testlets meet testing criteria;

h. selecting loss functions and prior probability of mastery;

i. estimating computerized mastery testing (CMT) cutscores which minimize expected posterior loss;

j. estimating expected error rates and average test length;

k. authoring and packaging the test for administration; and

l. administering presentation and scoring of the testlets in a sequential manner to the examinee.

2. A method according to claim 1 wherein the pool of test items is unidimensional and said IRT statistics are estimated for each said item by using a marginal maximum likelihood estimation procedure.

3. A method according to claim 2 wherein the marginal maximum likelihood estimation procedure is implemented by using the BILOG calibration program.

4. A method according to claim 1 wherein the pool of test items is unidimensional and said IRT statistics are estimated for each said item by using a joint maximum likelihood logistic estimation procedure.

5. A method according to claim 4 wherein the joint maximum likelihood logistic estimation procedure is implemented by using the LOGIST calibration program.

6. A method according to claim 1 wherein the pool of test items is multidimensional and said IRT statistics are estimated for each test item by using a multidimensional IRT model estimation procedure.

7. A method according to claim 6 wherein the multidimensional IRT model estimation procedure is implemented by using the TESTFACT program software.

8. A method according to claim 1 wherein the number of on-line calculations performed during the step of administrating presentation and scoring of the testlets is reduced by determining the probability metric cutscore values prior to the step of administrating presentation and scoring of the testlets.

9. A method according to claim 8 wherein the number of on-line calculations performed during the step of administrating presentation and scoring of the testlets is reduced by using the CMT cutscore estimation program.

10. A method according to claim 1 wherein the number of on-line calculations performed during the step of administrating presentation and scoring of the testlets is reduced by determining cumulative number of right cutscore values prior to the step of administrating presentation and scoring of the testlets.

11. A method according to claim 10 wherein the number of on-line calculations performed during the step of administrating presentation and scoring of the testlets is reduced by using the CMT cutscore estimation program.

12. A method according to claim 1 wherein the testlets are substantially parallel.

13. A method according to claim 1 wherein the testlets are nonparallel.

14. A method according to claim 1 wherein the testlets are administered to the examinee in a random manner for all testlets not as yet administered to the examinee so that a testlets is not administered more than once to the same examinee.

15. A method according to claim 1 wherein the CMT cutscore is determined for a fixed-length test.

16. A method according to claim 15 wherein the CMT cutscore is determined by using the CMT cutscore estimation program.

17. A method according to claim 1 wherein the CMT cutscores are determined for a variable-length test.

18. A method according to claim 17 wherein the CMT cutscores are determined by using the CMT cutscore examination program.

19. A method according to claim 1 wherein the test items are assembled into testlets by using the CMT testlet assembly procedure.

20. A method according to claim 1 wherein the step of presentation and scoring is determined by using the CMT-administration system program.

21. A method according to claim 1 wherein the step of determining mastery ability values comprises:

a. performing an Angoff-design cutscore study comprising;

(1) estimating a percentage of borderline examinee expected to give a correct response for each test item in the pool of the test items; and

(2) determining a proportion correct cutscore for a pool of selected test items;

b. translating the proportion correct cutscore into an ability value (Θ

_c) on the IRT scale by using a test characteristic curve of the selected test items; and

c. determining the lowest ability value at which an examinee will be considered a master (Θ

_m) and the highest ability value at which an examinee will be considered a nonmaster (Θ

_n) for the pool of test items in a method comprising;

(1) estimating the standard error of measurement at Θ

_c ; and

(2) selecting Θ

_m and Θ

_n to lie outside a certain error interval centered about Θ

_c.

22. A method according to claim 21 wherein the test characteristic curve is determined by using the CMT cutscore validation program.

23. A method according to claim 1 wherein the step of determining mastery ability values comprises:

a. performing an Ebel-design cutscore study for determining a proportion correct cutscore;

b. translating the proportion correct cutscore into an ability value (Θ

_c) on the IRT scale by using a test characteristic curve of the selected test items; and

c. determining the lowest ability value at which an examinee will be considered a master (Θ

_m) and the highest ability value at which an examinee will be considered a nonmaster (Θ

_n) for the pool of test items in a method comprising;

(1) estimating the standard error of measurement at Θ

_c ; and

(2) selecting Θ

_m and Θ

_n to lie outside a certain error interval centered about Θ

_c.

24. A computerized mastery testing system having a computer system means for testing whether an examinee has attained specified mastery of a given subject comprising:

a. means for inputing test specifications and test items into a computer database;

b. means for applying item response theory (IRT) so as to obtain IRT statistics for the test items and determine mastery ability values;

c. means for assembling the test items into testlets;

d. means for verifying the testlets meet testing criteria;

e. means for selecting loss functions and the prior probability of mastery;

f. means for applying Bayesian decision theory so as to estimate computerized mastery testing (CMT) cutscores which minimize the expected posterior loss and determine the associated operating characteristics;

g. means for authoring and packaging the test for administration; and

h. means for administering presentation and scoring of an examinee response dependent variable number of the testlets in a random, sequential manner to the examinee.

25. A method of computerized mastery testing for deciding whether an individual examinee has attained a specified mastery of a given subject by using a computer system means, said method comprising:

determining test specifications;

creating a pool of test items which meet the test specifications;

determining the dimensionality of the pool of test items;

obtaining item response theory (IRT) statistics and assessing IRT model fit for each test item, wherein the pool of test items is unidimensional and said IRT statistics are estimated for each test item by using a joint maximum likelihood logistic estimation procedure implemented by executing LOGIST calibration program;

determining mastery ability values;

assembling the test items into testlets by using the computerized mastery testing (CMT) testlet assembly procedure;

verifying all of the testlets meet testing criteria;

selecting loss functions and prior probability of mastery;

estimating CMT cutscores which minimize expected posterior loss, wherein the CMT cutscores are determined for a variable-length test by executing the CMT cutscore estimation program;

estimating expected error rates and average test length;

authoring and packaging the test for administration; and

administering presentation and scoring of the testlets in a sequential manner and wherein the testlets are chosen in a random manner for all testlets not as yet administered to the examinee by executing the CMT testlets presentation and scoring procedure.