Method for estimating examinee attribute parameters in cognitive diagnosis models

• US 20060040247A1
• Filed: 08/23/2004
• Published: 02/23/2006
• Est. Priority Date: 08/23/2004
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Templin, Jonathan
• Granted Patent: US 7,628,614 B2
• Time in Patent Office: Days
• US Class Current: 434/362
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

A method of determining a mastery level for an examinee from an assessment is disclosed. The method includes receiving one or more of an overall skill level for an examinee, a weight for the overall skill level, a covariate vector for an examinee, and a weight for the covariate vector. An examinee attribute value is computed using one or more of the received values for each examinee and each attribute. The computation of the examinee attribute values can include estimating the value using a Markov Chain Monte Carlo estimation technique. Examinee mastery levels are then assigned based on each examinee attribute level. Dichotomous or polytomous levels can be assigned based on requirements for the assessment.

Claims

1. A method for determining one or more examinee attribute mastery levels from an assessment, the method comprising:

receiving an overall skill level for an examinee, wherein the overall skill level comprises a rating of the examinee'"'"'s performance on at least a portion of an assessment; and

for each of one or more attributes;

receiving a weight for the overall skill level,

computing an examinee attribute value based on at least the overall skill level, the weight corresponding to the attribute and one or more responses made by the examinee to one or more questions pertaining to the attribute on the assessment, and

assigning an examinee attribute mastery level for the examinee with respect to the attribute based on whether the examinee attribute value surpasses one or more thresholds.

2. The method of claim 1 wherein computing an examinee attribute value comprises:

computing a product of the overall skill level and the weight; and

summing the product and a value based on the one or more responses.

3. The method of claim 1, further comprising:

receiving a covariate vector for an examinee, wherein the covariate vector includes a value for each of one or more covariates for the examinee; and

receiving, for each attribute, a weighting vector for the covariate vector,

wherein the examinee attribute value is further based on the covariate vector and the weighting vector.

4. The method of claim 3 wherein computing an examinee attribute value comprises:

computing a product of the overall skill level and the weight;

computing a vector product of the covariate vector and the weighting vector for the attribute; and

summing the product, the vector product and a value based on the one or more responses.

5. A method for determining one or more examinee attribute mastery levels from an assessment, the method comprising:

receiving a covariate vector for an examinee, wherein the covariate vector includes a value for each of one or more covariates for the examinee; and

for each of one or more attributes;

receiving a weighting vector for the covariate vector,

computing an examinee attribute value based on at least the covariate vector, the weighting vector and one or more responses made by the examinee to one or more questions pertaining to the attribute on an assessment, and

assigning an examinee attribute mastery level for the examinee with respect to the attribute based on whether the examinee attribute value surpasses one or more thresholds.

6. The method of claim 5 wherein computing an examinee attribute value comprises:

computing a vector product of the covariate vector and the weighting vector; and

summing the vector product and a value based on the one or more responses.

7. A method for determining examinee attribute mastery levels, the method comprising:

(a) for each item on an assessment, determining an estimated value for each of one or more item parameters;

(b) for each proficiency space parameter, determining an estimated value for the proficiency space parameter;

(c) for each examinee parameter, determining an estimated value for the examinee parameter for each examinee;

(d) repeating (a) through (c) a predetermined number of iterations; and

(e) determining one or more examinee attribute mastery levels based on the item parameters, proficiency space parameters, and examinee parameters.

8. The method of claim 7 wherein step (a) comprises:

performing a Metropolis-Hastings within Gibbs step for updating an estimate of a probability of correctly applying all attributes described in a Q-matrix for an item if each attribute for the item is mastered;

performing a Metropolis-Hastings within Gibbs step for updating an estimate of a penalty imposed on the probability of correctly applying all attributes in the Q-matrix if an examinee has not mastered a particular attribute; and

performing a Metropolis-Hastings within Gibbs step for updating an estimate of a measure of whether the attributes in the Q-matrix adequately describe attributes necessary for answering the item.

9. The method of claim 8 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of a probability comprises:

selecting a value for the probability from a moving window distribution.

10. The method of claim 8 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of a probability comprises:

accepting a value for the estimated probability with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated probability if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated probability if the examinee answered the item incorrectly, ii) a prior probability of the estimated probability and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed probability if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed probability if the examinee answered the item incorrectly, v) a prior probability of the previous computed probability and vi) a second proposal probability.

11. The method of claim 8 wherein the prior distribution for an estimated probability comprises a uniform distribution between 0 and 1.

12. The method of claim 8 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of a penalty comprises:

selecting a value for the estimated penalty from a moving window distribution.

13. The method of claim 8 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of a penalty comprises:

accepting a value for the estimated penalty with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated penalty if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated penalty if the examinee answered the item incorrectly, ii) a prior probability of the estimated penalty and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed penalty if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed penalty if the examinee answered the item incorrectly, v) a prior probability of the previous computed penalty and vi) a second proposal probability.

14. The method of claim 8 wherein the prior distribution for an estimated penalty comprises a uniform distribution between 0 and 1.

15. The method of claim 8 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of a measure comprises:

selecting a value for the estimated measure from a moving window distribution.

16. The method of claim 8 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of a measure comprises:

accepting a value for the estimated measure with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated measure if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated measure if the examinee answered the item incorrectly, ii) a prior probability of the estimated measure and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed measure if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed measure if the examinee answered the item incorrectly, v) a prior probability of the previous computed measure and vi) a second proposal probability.

17. The method of claim 8 wherein the prior distribution for an estimated measure comprises a uniform distribution between 0 and 3.

18. The method of claim 8

wherein step (a) further comprises:

for each parameter specified by the Q-matrix, performing a Metropolis-Hastings within Gibbs step for updating an estimate of a penalty exponent, which links an examinee attribute mastery level and a Q-matrix entry, for a penalty imposed on the probability of correctly applying all attributes in the Q-matrix if an examinee has not mastered a particular attribute, and

wherein step (c) comprises;

for each attribute, performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level weighting factor;

for each attribute and each of one or more mastery ranges, performing a Metropolis-Hastings within Gibbs step for updating an estimate for each of one or more mastery thresholds;

for each examinee, performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level for an examinee; and

for each covariate, attribute and continuous examinee parameter, performing a Metropolis-Hastings within Gibbs step for updating an estimate for each element of a covariate weighting vector.

19. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of a penalty exponent comprises:

selecting an estimated penalty exponent from a moving window distribution.

20. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of a penalty exponent further comprises:

accepting a value for the estimated penalty exponent with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated penalty exponent if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated penalty exponent if the examinee answered the item incorrectly, ii) a prior probability of the estimated penalty exponent and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed penalty exponent if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed penalty exponent if the examinee answered the item incorrectly, v) a prior probability of the previous computed penalty exponent and vi) a second proposal probability.

21. The method of claim 18 wherein the prior distribution for an estimated penalty exponent comprises a uniform distribution from 0 to 1.

22. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level weighting factor comprises:

selecting an estimated overall skill level weighting factor from a moving window distribution.

23. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level weighting factor comprises:

accepting a value for the estimated skill level weighting factor with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated skill level weighting factor if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated skill level weighting factor if the examinee answered the item incorrectly, ii) a prior probability of the estimated skill level weighting factor and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed skill level weighting factor if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed skill level weighting factor if the examinee answered the item incorrectly, v) a prior probability of the previous computed skill level weighting factor and vi) a second proposal probability.

24. The method of claim 18 wherein the prior distribution for an estimated overall skill level weighting factor comprises a uniform distribution between −

1 and 1.

25. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of one or more mastery thresholds comprises:

for each mastery threshold, selecting an estimated mastery threshold from a moving window distribution.

26. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of one or more mastery thresholds comprises:

accepting a value for the estimated mastery threshold with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated mastery threshold if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated mastery threshold if the examinee answered the item incorrectly, ii) a prior probability of the estimated mastery threshold and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed mastery threshold if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed mastery threshold if the examinee answered the item incorrectly, v) a prior probability of the previous computed mastery threshold and vi) a second proposal probability.

27. The method of claim 18 wherein the prior distribution for each estimated mastery threshold comprises a uniform distribution between −

4 and 4.

28. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level for the examinee comprises:

selecting an estimated overall skill level from a moving window distribution.

29. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level for the examinee comprises:

accepting a value for the estimated overall skill level with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated overall skill level if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated overall skill level if the examinee answered the item incorrectly, ii) a prior probability of the estimated overall skill level and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed overall skill level if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed overall skill level if the examinee answered the item incorrectly, v) a prior probability of the previous computed overall skill level and vi) a second proposal probability.

30. The method of claim 18 wherein the prior distribution for an estimated overall skill level comprises a normal distribution having a mean equal to 0 and a variance equal to 1.

31. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate for each element of a covariate weighting vector comprises:

for each element of a covariate weighting vector, selecting an estimate for the element from a moving window distribution.

32. The method of claim 18 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate for each element of a covariate weighting vector comprises:

accepting an estimate for an element of a covariate weighting vector with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated element if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated element if the examinee answered the item incorrectly, ii) a prior probability of the estimated element and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed element if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed element if the examinee answered the item incorrectly, v) a prior probability of the previous computed element and vi) a second proposal probability.

33. The method of claim 18 wherein a prior distribution for the estimated element of the covariate weighting vector comprises a uniform distribution between −

4 and 4.

34. The method of claim 7 wherein step (c) comprises:

for each attribute, performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level weighting factor;

for each attribute, performing a Metropolis-Hastings within Gibbs step for updating an estimate of a mastery threshold;

for each examinee, performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level for the examinee; and

for each covariate, attribute and continuous examinee parameter, performing a Metropolis-Hastings within Gibbs step for updating an estimate of each element of a covariate weighting vector.

35. The method of claim 34 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level weighting factor comprises:

selecting an estimated overall skill level weighting factor from a moving window distribution.

36. The method of claim 34 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level weighting factor comprises:

accepting a value for the estimated overall skill level weighting factor with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated overall skill level weighting factor if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated overall skill level weighting factor if the examinee answered the item incorrectly, ii) a prior probability of the estimated overall skill level weighting factor and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed overall skill level weighting factor if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed overall skill level weighting factor if the examinee answered the item incorrectly, v) a prior probability of the previous computed overall skill level weighting factor and vi) a second proposal probability.

37. The method of claim 34 wherein the prior distribution for an estimated overall skill level weighting factor comprises a uniform distribution between −

1 and 1.

38. The method of claim 34 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate for each of one or more mastery thresholds comprises:

selecting, for each mastery threshold, an estimated mastery threshold from a moving window distribution.

39. The method of claim 34 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate for each of one or more mastery thresholds comprises:

for each mastery threshold, accepting a value for the estimated mastery threshold with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated mastery threshold if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated mastery threshold if the examinee answered the item incorrectly, ii) a prior probability of the estimated mastery threshold and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed mastery threshold if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed mastery threshold if the examinee answered the item incorrectly, v) a prior probability of the previous computed mastery threshold and vi) a second proposal probability.

40. The method of claim 34 wherein the prior distribution for each estimated mastery threshold comprises a uniform distribution between −

4 and 4.

41. The method of claim 34 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level for the examinee comprises:

selecting an estimated overall skill level from a moving window distribution.

42. The method of claim 34 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of an overall skill level for the examinee comprises:

accepting a value for the estimated overall skill level with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated overall skill level if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated overall skill level if the examinee answered the item incorrectly, ii) a prior probability of the estimated overall skill level and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed overall skill level if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed overall skill level if the examinee answered the item incorrectly, v) a prior probability of the previous computed overall skill level and vi) a second proposal probability.

43. The method of claim 34 wherein the prior distribution for an estimated overall skill level comprises a normal distribution having a mean equal to 0 and a variance equal to 1.

44. The method of claim 34 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of each element of a covariate weighting vector comprises:

for each element of a covariate weighting vector, selecting an estimate for the element from a moving window distribution.

45. The method of claim 34 wherein performing a Metropolis-Hastings within Gibbs step for updating an estimate of each element of a covariate weighting vector comprises:

accepting an estimate for an element of a covariate weighting vector with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated element if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated element if the examinee answered the item incorrectly, ii) a prior probability of the estimated element and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed element if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed element if the examinee answered the item incorrectly, v) a prior probability of the previous computed element and vi) a second proposal probability.

46. The method of claim 34 wherein a prior distribution for the estimated element of the covariate weighting vector comprises a uniform distribution between −

4 and 4.

47. The method of claim 7 wherein step (d) comprises, for each examinee:

performing a Gibbs step for updating an estimate of an examinee attribute mastery level; and

performing a Metropolis-Hastings within Gibbs step for updating an estimate of a latent ability value.

48. The method of claim 47 wherein performing the Gibbs step comprises:

selecting an examinee attribute value from a normal distribution having a mean equal to the sum of a vector product of the covariate weighting vector and a covariate vector and a product of the overall skill level weighting factor and the overall skill level, and a variance equal to one minus the overall skill level weighting factor squared; and

determining an examinee attribute master level based on the examinee attribute value and one or more mastery thresholds.

49. The method of claim 47 wherein the prior distribution for an estimated examinee attribute mastery level comprises a standard normal cumulative distribution function of a product of a covariate weighting factor vector and a covariate vector plus the product of an overall skill value weighting factor and an overall skill value minus a mastery threshold all divided by the square root of one minus the overall skill value weighting factor squared.

50. The method of claim 49 wherein the examinee attribute mastery level is drawn from a Bernoulli distribution with probability of success equal to the product of a probability that the examinee answers a question correctly given that the examinee is a master of the particular attribute, a set of item parameters and attribute parameters and the prior distribution that the examinee is a master of the particular attribute divided by the sum of i) the product of a probability that the examinee answers the question correctly given that the examinee is not a master of the particular attribute, a set of item parameters and attribute parameters and the prior distribution that the examinee is not a master of the particular attribute and ii) the product of a probability that the examinee answers a question correctly given that the examinee is a master of the particular attribute, a set of item parameters and attribute parameters and the prior distribution that the examinee is a master of the particular attribute.

51. The method of claim 47 wherein the prior distribution for an estimated examinee attribute mastery level comprises the difference between i) a standard normal cumulative distribution function of a product of a covariate weighting factor vector and a covariate vector plus the product of an overall skill value weighting factor and an overall skill value minus a first mastery threshold all divided by the square root of one minus the overall skill value weighting factor squared and ii) a standard normal cumulative distribution function of a product of the covariate weighting factor vector and the covariate vector plus the product of the overall skill value weighting factor and the overall skill value minus a second mastery threshold all divided by the square root of one minus the overall skill value weighting factor squared.

52. The method of claim 51 wherein the examinee attribute mastery level is drawn from a Bernoulli distribution with probability of success equal to the product of a probability that an examinee answers a question correctly given that the particular attribute is mastered for a set of item parameters and attribute parameters and the prior distribution for the estimated examinee attribute mastery level divided by the sum of the products, for each examinee attribute mastery level, of the probability that the examinee answers the question correctly if the examinee has the examinee attribute mastery level, a set of item parameters and attribute parameters and the prior distribution for the examinee attribute mastery level.

53. The method of claim 47 wherein performing the Metropolis-Hastings within Gibbs step comprises:

selecting an estimated latent ability value from a moving window distribution.

54. The method of claim 47 wherein performing the Metropolis-Hastings within Gibbs step comprises:

accepting a value for the estimated latent ability value with a likelihood that is the minimum of one and the product, for all items and examinees, of i) either a) a probability of an examinee correctly answering an item with the estimated latent ability value if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the estimated latent ability value if the examinee answered the item incorrectly, ii) a prior probability of the estimated latent ability value and iii) a first proposal probability divided by the product, for all items and examinees, of iv) either a) a probability of the examinee correctly answering an item with a previous computed latent ability value if the examinee answered the item correctly or b) one minus a probability of the examinee incorrectly answering the item with the previous computed latent ability value if the examinee answered the item incorrectly, v) a prior probability of the previous computed latent ability value and vi) a second proposal probability.

55. The method of claim 47 wherein the prior distribution for an estimated latent ability value comprises a normal distribution having a mean equal to the sum of the vector product of a covariate weighting factor vector and a covariate vector and the product of an overall skill value weighting factor and an overall skill value and a variance equal to one minus the overall skill value weighting factor squared.

56. A method of updating an estimate of a value from a moving window distribution, the method comprising:

receiving a prior estimate of a value;

receiving a base interval for a moving window distribution, wherein the interval includes an absolute lower limit and an absolute upper limit;

receiving a width for the moving window distribution;

determining a minimum selectable value equal to the maximum of i) the prior estimate of the value minus half of the width of the moving window distribution and ii) the absolute lower limit;

determining a maximum selectable value equal to the minimum of i) the prior estimate of the value plus half of the width of the moving window distribution and ii) the absolute upper limit;

selecting the estimate for the value from a uniform distribution between the minimum selectable value and the maximum selectable value.