Method for interacting with a test subject with respect to knowledge and functionality

• US 6,301,571 B1
• Filed: 11/16/1998
• Issued: 10/09/2001
• Est. Priority Date: 09/13/1996
• Status: Expired due to Term

• Current Assignee: Curtis M. Tatsuoka
• Original Assignee: Curtis M. Tatsuoka
• Inventors: Tatsuoka, Curtis M.
• Primary Examiner(s): Davis, George B.
• Application Number:
  US09/193,061
• Time in Patent Office: 1,058 Days
• Field of Search: 706/45, 706/50, 434/322, 705/7, 707/501
• US Class Current: 706/45
• CPC Class Codes: G01R 31/318371 Methodologies therefor, e.g. algorithms, procedures
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0203 Market surveys; Market polls

Abstract

The invention is a method for interacting with a test subject with respect to knowledge or functionality characterized by a plurality of states in one or more domains. A domain is a set of facts, a set of values, or a combination of a set of facts and a set of values. The set of facts for a knowledge domain is any set of facts. The set of facts for a functionality domain is a set of facts relating to the functionality of a test subject. A state is denoted as a fact state, a value state, or a combination state, a fact state being characterized by a subset of facts, a value state being characterized by a subset of values, and a combination state being characterized by a combination of a subset of facts and a subset of values. The method consists of specifying one or more domains, specifying a domain pool for each domain comprising a plurality of test item blocks consisting of one or more test items, specifying a class conditional density for each test item in each test item block for each state in each domain, selecting one or more test item blocks from the one or more domain pools to be administered to a test subject, and processing the responses of the test subject to the one or more test item blocks administered to the test subject.

Claims

1. A method for interacting with a test subject with respect to knowledge or functionality characterized by a plurality of states in one or more domains, a domain being a set of facts, a set of values, or a combination of a set of facts and a set of values, the set of facts for a knowledge domain being any set of facts, the set of facts for a functionality domain being a set of facts relating to the functionality of a test subject, a state being denoted as a fact state, a value state or a combination state, a fact state being characterized by a subset of facts, a value state being characterized by a subset of values, a combination state being characterized by a combination of a subset of facts and a subset of values, a first state being higher than or equal to a second state and a second state being lower than or equal to a first state if (1) the subset of facts or a subset of values associated with the first state respectively includes the subset of facts or is greater than or equal to the subset of values associated with the second state or (2) the subset of facts and the subset of values associated with the first state respectively includes the subset of facts and is greater than or equal to the subset of values associated with the second state, the method comprising the steps:

(a) specifying one or more domains where each domain comprises a plurality of states and determining the higher-lower-neither relationships for each state in each domain, the higher-lower-neither relationships for a state being a specification of which states are higher, which states are lower, and which states are neither higher or lower, the plurality of states for at least one domain including a first, second, and third fact state characterized by subsets of facts wherein (1) the first and second fact states are higher than the third fact state and the first fact state is neither higher nor lower than the second fact state or (2) the first fact state is higher than the second and third fact states and the second fact state is neither higher nor lower than the third fact state;

(b) specifying a domain pool for each domain comprising a plurality of test item blocks, a test item block consisting of one or more test items, a test item administered to a test subject resulting in one of a plurality of possible responses;

(c) specifying a class conditional density f_ibd(x|s) for each test item i in test item block b for domain d for each state s in each domain, a class conditional density being a specification of the probability of a test subject in state s of domain d providing a response x to the test item i in the test item block b, each test item partitioning one or more domains into a plurality of partitions according to the class conditional densities associated with the test item, a partition being a subset of states for which the class conditional densities are the same or the union of such subsets;

(d) selecting one or more test item blocks from the one or more domain pools to be administered to a test subject;

(e) processing the responses of the test subject to the one or more test item blocks administered to the test subject, the relationship of the test subject to domains being representable by a state probability set (SPS); and

(z) repeating method from step (d) until method termination criteria are satisfied.

2. The method of claim 1 wherein step (a) comprises the steps:

(a1) determining the intersections of the partitions of states by one or more hypothetical test item blocks with hypothetical partitions.

3. The method of claim 1 wherein step (a) comprises the steps:

(a1) determining the intersections of the partitions of states by the test item blocks in the domain pool; and

(a2) replacing a first domain configuration with a second domain configuration, the second domain configuration states being the intersections of the partitions of the first domain configuration states by the test item blocks, the higher-lower-neither relationships of the second domain configuration states being derived from the higher-lower-neither relationships of the first domain configuration states.

4. The method of claim 3 wherein step (b) further comprises the step:

(b2) adding new types of test item blocks to the test item pool to increase the number of intersections of the partitions.

5. The method of claim 1 wherein in step (a) a state is removed from a domain if the number of test subjects in a specified population satisfying a condition is less than a specified number, the condition being that a test subject'"'"'s posterior probability for the state is less than a specified threshold.

6. The method of claim 1 wherein step (b) comprises the step:

(b1) determining the intersections of the partitions of states by one or more test item blocks in a domain pool.

7. The method of claim 1 wherein step (b) comprises the steps:

(b1) determining the partition of states by test item block 1 in a domain pool; and

(b2) determining intersections of partition of states by test item block N in a domain pool with the intersections of partitions of states by test item blocks 1 through N−

-1 in the domain pool, N taking on successive values of 2 through N, N being an integer.

8. The method of claim 1 wherein step (b) comprises the steps:

(b1) determining the sharpness of a test item block from a domain pool, sharpness being a measure of the capability of a test item block to discriminate between test subjects in different states, sharpness being measured by use of one or more discrepancy measures; and

(b2) removing the test item block from the domain pool if its sharpness does not satisfy a predetermined criterion.

9. The method of claim 1 wherein step (b) comprises the step:

(b1) administering hypothetically hypothetical test item blocks with hypothetical partitions and hypothetical class conditional densities.

10. The method of claim 1 wherein step (c) comprises the steps:

(c1) specifying one or more prior parameter distribution functions for each of a collection of test items, the class conditional densities for the test items being determinable from the parameter distribution functions;

(c2) obtaining a sequence of responses to a sequence of test item blocks from the domain pool by each of a plurality of training-sample test subjects;

(c3) updating the SPS of each of one or more of the plurality of training-sample test subjects based on a sequence of responses using an initial SPS and the class conditional densities;

(c4) determining training-sample test subject'"'"'s tentative classification in at least one domain;

(c5) updating the parameter distribution functions utilizing the one or more training-sample test subjects'"'"' tentative classifications to obtain the current parameter distribution functions; and

(c6) repeating steps (c3), (c4), (c5), and (c6) for active parameter distribution functions, an active parameter distribution function being a parameter distribution function for which a repeat termination rule has not been satisfied, random sampling from an SPS being used at least once in determining a training-sample test subject'"'"'s tentative classification while repeating steps (c3), (c4), (c5), and (c6).

11. The method of claim 1 wherein step (c) comprises the steps:

(c1) identifying test items having questionable class conditional densities, a questionable class conditional density being indicated by a sharpness criterion not being satisfied; and

(c2) changing a class conditional probability density of one or more test items to achieve greater sharpness.

12. The method of claim 1 wherein in step (c) class conditional densities are dependent on test subject-related factors in addition to a test subject'"'"'s knowledge or functionality.

13. The method of claim 1 wherein step (e) further comprises the step:

(c1) specifying an initial SPS for the test subject with respect to a domain.

14. The method of claim 1 wherein a domain pool includes a multi-item test item block consisting of a plurality of test items.

15. The method of claim 14 wherein the total number of multi-item test item blocks administered for one domain or a combination of two or more domains equals a predetermined number.

16. The method of claim 1 wherein in step (c) the class conditional density for a test item is a function of a difficulty parameter which is a measure of the difficulty that a test subject will have in providing the best response to the test item, the probability of a test subject providing the best response to the test item decreasing as the difficulty parameter varies in the direction of greater difficulty of the test item.

17. The method of claim 1 wherein in step (d) the selection of a test item block is in accordance with a test item block sequence generated in accordance with specified sequence generation rules.

18. The method of claim 1 wherein one or more strategy trees are defined for each of one or more domains, a strategy tree comprising a plurality of paths with each path beginning with the first test item block to be administered, continuing through a sequence alternating between a particular response to the last test item block and the specification of the next test item block, and ending with a particular response to the final test item block in the path, step (d) comprising the steps:

(d1) selecting a strategy tree based on a comparative evaluation of a plurality of the defined strategy trees utilizing one or more item objective functions, an item objective function providing a measure of effectiveness of a test item in classifying a test subject in a domain; and

(d2) selecting the test item block by consulting the strategy tree selected in step (d1).

19. The method of claim 1 wherein one or more strategy trees are defined for each of one or more domains, a strategy tree comprising a plurality of paths with each path beginning with the first test item block to be administered, continuing through a sequence alternating between a particular response to the last test item block and the specification of the next test item block, and ending with a particular response to the final test item block in the path, step (d) comprising the steps:

(d1) selecting in a random manner a strategy tree from a plurality of the defined strategy trees; and

(d2) selecting the test item block by consulting the strategy tree selected in step (d1).

20. The method of claim 1 wherein in step (d) the domain pool from which a test item block is to be selected is chosen from the group consisting of (1) the domain pool associated with the next domain in a specified domain sequence, (2) the domain pool associated with a domain chosen randomly, (3) the domain pool associated with a domain chosen on the basis of one or more uncertainty measures, (4) the domain pool associated with a domain chosen on the basis of one or more ranking measures, (5) the domain pool associated with a domain chosen on the basis of the values of one or more loss functions, (6) the domain pool associated with a domain chosen on the basis of the values of one or more SPS'"'"'s, and (7) the domain pool associated with a domain chosen by a process dependent on the prior satisfaction of one or more stopping rules.

21. The method of claim 1 wherein in step (d) the selection of a test item block is based on an objective function that is a function of one or more objective functions.

22. The method of claim 1 wherein step (d) comprises the steps:

(d1) selecting the test item block by consulting a strategy tree if a strategy tree is available, a strategy tree comprising a plurality of paths with each path beginning with the first test item block to be administered, continuing through a sequence alternating between a particular response to the last test item block and the specification of the next test item block, and ending with a particular response to the final test item block in the path, the specification of each test item block in a strategy tree being based on a comparative evaluation of specified collections of test item blocks in one or more domain pools;

otherwise,

(d2) performing a comparative evaluation of specified collections of test item blocks in one or more domain pools; and

(d3) selecting the test item block based on the results of the comparative evaluation of step (d2).

23. The method of claim 22 wherein in step (d2) multi-item test item blocks are compared, a multi-item test item block consisting of a plurality of test items.

24. The method of claim 22 wherein the specified collection for a domain are those test item blocks that have not yet been selected for administration to the test subject.

25. The method of claim 22 further comprising the step:

(d4) determining for a test item block in a domain pool the weighted frequency and/or the probability of being selected; and

(d5) removing a test item block from the domain pool if the weighted frequency and/or the probability of being selected is less than a predetermined value.

26. The method of claim 22 wherein a truncated strategy tree is obtained by removing one or more test item blocks at the path ends of a specified strategy tree if the weighted loss in administering test items for the truncated strategy tree is less than the weighted loss for the specified strategy tree, the weighted loss for a strategy tree being obtained by weighting a loss function over paths in the strategy tree and test subject states, the loss function being a measure of the loss associated with administering the test items in a path of the strategy tree.

27. The method of claim 26 wherein the loss function is a function of (1) the state of a domain, (2) a classification decision action that specifies a state, and (3) the number of test item blocks administered.

28. The method of claim 26 wherein the loss function consists of two additive components, the first component being a measure of the loss associated with the classification of the test subject after administering one or more additional test item blocks, the loss associated with an incorrect classification being higher than the loss associated with a correct classification, the second component being the cost of administering the one or more additional test item blocks.

29. The method of claim 28 wherein the first component of the loss function is (1) a constant A₁(s) if the test subject would be classified correctly after administering the one or more additional test item blocks and (2) a constant A₂(s) if the test subject would be classified incorrectly after administering the one or more additional test item blocks, the constants A₁(s) and A₂(s) having a possible dependence on the state s, the second component of the loss function being the sum of the individual costs of administering the one or more additional test item blocks.

30. The method of claim 22 wherein there are a plurality of domains and the comparative evaluation utilizes a domain objective function, the domain objective function being a function of one or more block objective functions, a block objective function being a function of one or more item objective functions, a second function being a function of a first function includes the second function being identical to the first function, an item objective function providing a measure of effectiveness of a test item in classifying a test subject in a domain, a block objective function providing a measure of effectiveness of a test item block in classifying a test subject in a domain, a domain objective function providing a measure of effectiveness of a test item block in classifying a test subject in a plurality of domains.

31. The method of claim 30 wherein at least one of the item objective functions is a weighted loss function given the hypothetical administration of a sequence of k test items, k being an integer, a loss function being a function of (1) a state in the domain, (2) a classification decision action that specifies a state, and (3) the number k of test items to be administered.

32. The method of claim 30 wherein at least one of the item objective functions is a function of a test item difficulty parameter and a state, the difficulty parameter being a measure of the difficulty that a test subject will have in providing the best response to a test item.

33. The method of claim 30 wherein at least one of the item objective functions is a weighted uncertainty measure, an uncertainty measure gauging the uncertainty as to which of the test item'"'"'s partitions the test subject is in, an uncertainty measure being smallest when all but one of the partition probabilities are near 0, a partition probability being the probability of the test subject being in the partition.

34. The method of claim 30 wherein at least one of the item objective functions is a weighted uncertainty measure, an uncertainty measure being smallest and the test item being best when all but one of the SPS probability density values are near 0.

35. The method of claim 30 wherein at least one of the item objective functions is a weighted uncertainty measure, an uncertainty measure being smallest and the first test item in a sequence of test items being most effective when all but one of the probability density values are near 0 after the hypothetical administration of the sequence of test items.

36. The method of claim 30 wherein at least one of the item objective functions is a weighted distance measure between the SPS after a hypothetical administration of a sequence of test items and the SPS prior to the hypothetical administration of the sequence of test items, the distance measure being a measure of the differences in the two SPSs.

37. The method of claim 30 wherein at least one of the item objective functions is a weighted discrepancy measure summed over pairs of states, a discrepancy measure for a test item given two states being a measure of the distance between the class conditional densities for the test item and the two states.

38. The method of claim 30 wherein at least one of the item objective functions is a two-valued function Φ

, the function Φ

being a function of (1) a test item and (2) a first state and a second state, Φ

having a first value if the test item separates the first and second states, Φ

having a second value if the test item does not separate the first and second states.

39. The method of claim 38 wherein Φ

has a first value for a plurality of the test items for a specified first state and a specified second state, the test item being selected in a random manner from the plurality of test items.

40. The method of claim 30 wherein at least one of the item objective functions is the sum of π

(j)π

(k)d_jk(i) over all states j and k in the domains for which an SPS is specified, π

(j) denoting the members of the SPS, d_jk(i) denoting a measure of the degree of discrimination between states j and k provided by test item i as measured by a discrepancy measure on the corresponding class conditional densities.

41. The method of claim 30 wherein at least one of the item objective functions is a weighted loss function for k=1, a loss function being a function of (1) a state in a domain, (2) a classification decision action that specifies a state, and (3) the number k of test items to be administered.

42. The method of claim 30 wherein at least one of the item objective functions is a loss function consisting of two additive components, the first component being a measure of the loss associated with the classification of the test subject after administering k test items, the loss associated with an incorrect classification being higher than the loss associated with a correct classification, the second component being the cost of administering the k test items.

43. The method of claim 42 wherein the first component of the loss function is (1) a constant A₁(s) if the test subject would be classified correctly after administering k additional test items and (2) a constant A₂(s) if the test subject would be classified incorrectly after administering k additional test items, the constants A₁(s) and A₂(s) having a possible dependence on the state s, the second component of the loss function being the sum of the individual costs of administering the k additional test items.

44. The method of claim 30 wherein at least one of the item objective functions is based on the Fisher information function.

45. The method of claim 30 wherein at least one of the item objective functions is a precision function.

46. The method of claim 30 wherein the domain objective function changes when one or more domain-objective-function criteria are satisfied.

47. The method of claim 46 wherein at least one of the domain-objective-function criteria is based on an uncertainty measure.

48. The method of claim 46 wherein at least one of the domain-objective-function criteria is based on one or more stopping rules.

49. The method of claim 22 wherein there is only one domain and the comparative evaluation utilizes a block objective function, the block objective function being a function of one or more item objective functions, a second function being a function of a first function includes the second function being identical to the first function, an item objective function providing a measure of effectiveness of a test item in classifying a test subject in a domain, a block objective function providing a measure of effectiveness of a test item block in classifying a test subject in a domain.

50. The method of claim 49 wherein at least one of the item objective functions is a weighted loss function given the hypothetical administration of a sequence of k test items, k being an integer, a loss function being a function of (1) a state in the domain, (2) a classification decision action that specifies a state, and (3) the number k of test items to be administered.

51. The method of claim 49 wherein at least one of the item objective functions is a function of a test item difficulty parameter and a state, the difficulty parameter being a measure of the difficulty that a test subject will have in providing the best response to a test item.

52. The method of claim 49 wherein at least one of the item objective functions is a weighted uncertainty measure, an uncertainty measure being a measure of the uncertainty as to which of the test item'"'"'s partitions that the test subject is in after the administration of a test item, an uncertainty measure being smallest and the test item being most effective when all but one of the partition probabilities are near 0, a partition probability being the probability of the test subject being in the partition.

53. The method of claim 49 wherein at least one of the item objective functions is a weighted uncertainty measure, an uncertainty measure being smallest and the test item being best when all but one of the SPS probability density values are near 0 after the hypothetical administration of the test item.

54. The method of claim 49 wherein at least one of the item objective functions is a weighted uncertainty measure, an uncertainty measure being smallest and the first test item in a sequence of test items being most effective when all but one of the SPS probability density values are near 0 after the hypothetical administration of the sequence of test items.

55. The method of claim 49 wherein at least one of the item objective functions is a weighted distance measure between the SPS after a hypothetical administration of a test item and the SPS prior to the hypothetical administration of the test item, the distance measure being a measure of the differences in the two SPSs.

56. The method of claim 49 wherein at least one of the item objective functions is a weighted distance measure between the SPS after a hypothetical administration of a sequence of test items and the SPS prior to the hypothetical administration of the sequence of test items, the distance measure being a measure of the differences in the two SPSs.

57. The method of claim 49 wherein at least one of the item objective functions is a weighted discrepancy measure summed over pairs of states, a discrepancy measure for a test item given two states being a measure of the distance between the class conditional densities for the test item and the two states.

58. The method of claim 49 wherein at least one of the item objective functions is a two-valued function Φ

, the function Φ

being a function of (1) a test item and (2) a first state and a second state, Φ

having a first value if the test item separates the first and second states, Φ

having a second value if the test item does not separate the first and second states.

59. The method of claim 58 wherein Φ

has a first value for a plurality of the test items for a specified first state and a specified second state, the test item being selected in a random manner from the plurality of test items.

60. The method of claim 49 wherein at least one of the item objective functions is the sum of π

(j)π

(k)d_jk(i) over all states j and k in the domains for which an SPS is specified, π

(j) denoting the members of the SPS, d_jk(i) denoting a measure of the degree of discrimination between states j and k provided by test item i as measured by a discrepancy measure on the corresponding class conditional densities.

61. The method of claim 49 wherein at least one of the item objective functions is a weighted loss function for k=1, a loss function being a function of (1) a state in a domain, (2) a classification decision action that specifies a state, and (3) the number k of test items to be administered.

62. The method of claim 49 wherein at least one of the item objective functions is a loss function consisting of two additive components, the first component being a measure of the loss associated with the classification of the test subject after administering k test items, the loss associated with an incorrect classification being higher than the loss associated with a correct classification, the second component being the cost of administering the k test items.

63. The method of claim 62 wherein the first component of the loss function is (1) a constant A₁(s) if the test subject would be classified correctly after administering k additional test items and (2) a constant A₂(s) if the test subject would be classified incorrectly after administering k additional test items, the constants A₁(s) and A₂(s) having a possible dependence on the state s, the second component of the loss function being the sum of the individual costs of administering the k additional test items.

64. The method of claim 49 wherein at least one of the item objective functions is based on the Fisher information function.

65. The method of claim 49 wherein at least one of the item objective functions is a precision function.

66. The method of claim 1 wherein in step (d) a test item block is tentatively selected using a predetermined selection rule, a random decision being made either to confirm the selection of the tentatively-selected test item block or to select another test item block.

67. The method of claim 66 wherein the test item blocks are ordered according to an effectiveness criterion associated with the predetermined selection rule, the tentatively-selected test item block being the most effective test item block, a plurality of the next-in-order test item blocks being denoted as the better test item blocks, one of the better test item blocks being selected for administration if the decision is made to select a test item block other than the tentatively-selected test item block.

68. The method of claim 67 wherein the selection of one of the better test item blocks is randomly made, the random selection being biased in accordance with the order of the better test item blocks.

69. The method of claim 1 wherein in step (d) each of a plurality of test item block selection rules produces a candidate test item block, the test item block selected for administration being a random selection from the plurality of candidate test item blocks.

70. The method of claim 1 wherein in step (d) the selected test item block is the test item block that maximizes a weighted relative ranking measure based on a plurality of test item block selection rules, a weighted relative ranking measure being a weighted function of the relative rankings of effectiveness for each test item block with respect to a plurality of item selection rules.

71. The method of claim 1 wherein step (d) comprises the steps:

(d1) selecting a test item block on the basis of specified rules of selection; and

(d2) rejecting the test item block with a probability based on an estimate of the exposure rate of the test item block, the exposure rate being a function of one or more state-specific exposure rates, a rejection of a test item block being followed by repeating steps (d1) and (d2);

otherwise, confirming the selection of the test item block for administration to a test subject.

72. The method of claim 1 wherein in step (d) a plurality of test-item-block sequences are generated, the test item blocks being selected from one of the plurality of test-item-block sequences based on a test-item-block sequence selection rule, the test-item-block sequence selection rule being based on a comparative evaluation of the test-item-block sequences based on one or more item objective functions.

73. The method of claim 1 wherein in step (d) the selection is made from one or more active domain pools, an active domain pool being associated with a domain for which one or more domain stopping rules have not been satisfied.

74. The method of claim 73 wherein a domain stopping rule is based on the SPS associated with one of a plurality of domains.

75. The method of claim 74 wherein the selection of the SPS is based on an uncertainty measure.

76. The method of claim 73 wherein at least one of the domain stopping rules is one of the group consisting of (1) that the marginal posterior value for a state in a domain is greater than a specified value, (2) that the posterior variance of an SPS is less than a specified value, (3) that a weighted uncertainty measure with respect to an SPS is less than a specified value, (4) that a weighted distance measure between an initial SPS and an SPS after administration of k test item blocks, k being an integer equal to or greater than one, exceeds a specified value, (5) that a weighted loss function is less than a specified value, (6) that the largest value of an SPS exceeds a specified value, (7) that responses to a predetermined number of test item blocks have been processed, (8) that responses to a predetermined number of test item blocks from a domain pool have been processed, (9) that given the hypothetical selection and administration of one or more sequences of k test item blocks, a weighted loss function is greater than a specified value, k being an integer equal to or greater than one, k for each sequence being the same or different from the k for any other sequence, (10) that given the hypothetical selection and administration of one or more sequences of k test item blocks, a weighted uncertainty measure decreases by less than a specified value, the specified value being expressed either in absolute terms or relative to the value of the weighted uncertainty measure prior to the hypothetical selection and administration of the one or more sequences of k test item blocks, k being an integer equal to or greater than one, k for each sequence being the same or different from the k for any other sequence, (11) that given the hypothetical selection and administration of one or more sequences of k test item blocks, a weighted distance measure increases by less than a specified value, the specified value being expressed either in absolute terms or relative to the value of the weighted distance measure prior to the hypothetical selection and administration of the one or more sequences of k test item blocks, k being an integer equal to or greater than one, k for each sequence being the same or different from the k for any other sequence, and (12) that the variance of an estimate of a value for the test subject is less than a specified value.

77. The method of claim 1 wherein a test subject'"'"'s relationship to a domain is represented by an SPS, the SPS being updated during each execution of step (e).

78. The method of claim 1 further comprising the steps:

(h) repeating steps (d), (e), and (z) for a plurality of test subjects;

(i) deleting superfluous states from the one or more domains; and

(j) adding missing states to the one or more domains.

79. The method of claim 1 further comprising the steps:

(h) repeating steps (d), (e), and (z) for a plurality of test subjects;

(i) determining the weighted frequency of administration for a test item block in a domain pool associated with a domain; and

(j) deleting the test item block from the domain pool if the weighted frequency of administration is less than a predetermined value.

80. The method of claim 1 further comprising the steps:

(h) repeating steps (d), (e), and (z) for a plurality of test subjects;

(i) determining the ideal response pattern for a test subject classified in each of one or more domain states for each administered sequence of test item blocks using the class conditional densities associated with each test item in each test item block, an ideal response pattern being a value or a set of values; and

(j) deleting a state from a domain if its corresponding ideal response pattern does not satisfy a specified criterion with respect to a specified number of test subject patterns, the specified criterion being expressed in terms of one or more distance measures, a distance measure being a measure of the differences between a test subject response pattern and an ideal response pattern.

81. The method of claim 1 further comprising the steps:

(h) repeating steps (d), (e), and (z) for a plurality of test subjects;

(i) determining the ideal response pattern for a test subject classified in each of one or more domain states for each administered sequence of test item blocks using the class conditional densities associated with each test item in each test item block, an ideal response pattern being a value or a set of values; and

(j) adding a state to a domain if a specified number of ideal response patterns do not satisfy a specified criterion with respect to one or more test subject response patterns, the specified criterion being expressed in terms of one or more distance measures, a distance measure being a measure of the differences between a test subject response pattern and an ideal response pattern.

82. The method of claim 1 further comprising the step:

(f) classifying the test subject in one or more domains in accordance with one or more decision rules if one or more stopping rules are satisfied, step (f) being performed after step (e) and before step (z).

83. The method of claim 82 wherein in step (f) a test subject is classified to a combination or value state, step (f) including the step:

(f1) transforming the one or more values associated with the state into one or more other values.

84. The method of claim 82 wherein a decision rule in step (f) is to classify to a state selected from the group consisting of (1) the state associated with the highest value in the SPS, (2) the state associated with the smallest value for a weighted loss function, (3) the state that has the greatest likelihood of being the true state of the test subject, (4) the state of a second domain that is equivalent to the state in which the test subject has been classified in a first domain, and (5) a state of a second domain based on a function of an SPS of a first domain.

85. The method of claim 82 wherein in step (f) a score value is based on a function of values corresponding to observed responses to test item blocks.

86. The method of claim 82 wherein a decision rule for a domain in step (f) is a function of the SPS corresponding to the domain.

87. The method of claim 82 wherein a state in a second domain is equivalent to a state in a first domain, the state in the second domain being expressed as a function of an ideal response pattern associated with the state in the first domain.

88. The method of claim 82 wherein classification to a state in a second domain is based on functions of ideal response patterns associated with one or more states in a first domain.

89. The method of claim 82 wherein an attribute is a subset of facts from one or more domains, the probability that a test subject possesses an attribute being called an attribute probability, an attribute probability being determined from one or more SPS'"'"'s.

90. The method of claim 89 wherein a determination as to whether or not an attribute is possessed by the test subject is based on the attribute probability.

91. The method of claim 89 wherein in step (d) the domain pool from which a test item block is to be selected is the domain pool associated with a domain chosen on the basis of one or more attribute probabilities.

92. The method of claim 82 further comprising the step:

(h) remediating the test subject, step (h) being performed after step (f) and before or after step (z).

93. The method of claim 92 being repeated one or more times.

94. The method of claim 93 wherein the test subject'"'"'s progress in remediation is expressed in terms of a change in classification.

95. The method of claim 92 wherein in step (h) a remediation program for a test subject classified to state X is a compilation of facts associated with one or more other states in the domain and a procedure for teaching the facts in the compilation to a test subject, the compilation not including facts associated with state X.

96. The method of claim 92 wherein in step (h) a criterion for selecting among domains on which to base remediation is that a dominant posterior probability value in a domain SPS exceeds a certain threshold level.

97. The method of claim 92 wherein in step (h) the specification of a remediation program for a state depends on an associated SPS.

98. The method of claim 92 wherein an attribute is a subset of facts from one or more domains, the probability that a test subject possesses an attribute being called an attribute probability, an attribute probability being calculated from one or more SPS'"'"'s, the specification of a remediation program in step (h) being based on one or more attribute probabilities of a test subject.

99. The method of claim 92 wherein step (h) comprises the steps:

(ha) compiling a collection of one or more topics, a topic being a set of facts, a set of values, or a combination of a set of facts and a set of values that characterize knowledge and/or functionality, the set of facts that characterize knowledge being any set of facts, the set of facts that characterize functionality being a set of facts relating to the functionality of a test subject;

(hb) compiling a collection of one or more treatments for each topic, a treatment comprising materials intended to teach a test subject;

(hc) specifying a plurality of question blocks for each of the one or more treatments of step (hb), a question block consisting of one or more questions, a response distribution being assigned to at least one of the questions in at least one of the question blocks;

(hd) selecting one or more topics from those in the collection of step (ha) for remediation;

(he) selecting one or more treatments from those specified in step (hb) for the topics selected in step (hd);

(hf) obtaining responses to one or more question blocks associated with the treatments selected in step (he) from a test subject after exposure to the one or more treatments or step (he); and

(hg) obtaining a measure of the effectiveness of the treatments of step (he) utilizing one or more of the response distributions assigned in step (hc).

100. The method of claim 99 wherein in step (hd) a topic is selected based on an SPS.

101. The method of claim 99 wherein the treatments specified in step (hb) can be classified as to treatment type, step (he) comprising the steps:

(he1) selecting one or more treatment types from a treatment-type pool for a topic selected if step (hd), the number of treatment types in the treatment-type pool being limited to one if a treatment-type selection process (TSP) stopping rule is satisfied; and

(he2) selecting one or more treatments from each treatment type selected in step (he1).

102. The method of claim 101 wherein in step (he1) the selection process is based on a weighted improvement measure, an improvement measure being a measure of the difference between a first and second knowledge representation associated respectively with a first and second state in a domain.

103. The method of claim 101 wherein in step (hg) the value of a treatment parameter is a measure of effectiveness, a probability distribution being associated with the treatment parameter, the selection process of step (he1) utilizing the probability distributions associated with one or more treatment parameters.

104. The method of claim 103 wherein the probability distribution associated with a treatment parameter is a function of the test subject'"'"'s SPS.

105. The method of claim 101 wherein step (hg) includes the step:

(hg1) estimating the value of a treatment parameter associated with a treatment type utilizing one or more responses to question blocks, a treatment parameter being a measure of effectiveness.

106. The method of claim 101 wherein in step (he1) the selection of a treatment type is based on one of a group consisting of (1) a weighted response function, (2) a weighted reward function, and (3) a weighted treatment loss function.

107. The method of claim 101 wherein in step (he1) selection of treatment type is based on one or more response distributions for questions, the response distributions being functions of one or more treatments or a treatment type.

108. The method of claim 101 wherein in step (he1) selection of treatment type is based on a weighted objective function, the weighting being done with respect to one or more response distributions for questions, a response distribution being a function of one or more treatments or a treatment type.

109. The method of claim 101 wherein in step (e1) a treatment type is selected by a process selected from the group consisting of (1) a random process and (2) a process selected randomly from plurality of treatment selection.

110. The method of claim 101 wherein step (e1) includes the steps:

(el-1) creating a plurality of remediation strategies, a remediation strategy being representable by one or more remediation strategy trees;

(el-2) selecting a best remediation strategy based on a comparative evaluation of the remediation strategies utilizing one or more objective functions; and

(el-3) selecting a treatment type from the best remediation strategy.

111. The method of claim 101 wherein in step (he1) selection of a treatment type is based on an SPS.

112. The method of claim 101 wherein in step (he1) selection of a treatment type is based on one or more item objective functions.

113. The method of claim of claim 99 further comprising the steps:

(h) repeating method from step (he) for one or more active topics, an active topic being a topic for which one or more treatment stopping rules have not been satisfied, a treatment stopping rule being one of the group consisting of (1) based on a function of one or more responses to question blocks, (2) that one of one or more predetermined sets of responses to question blocks have been obtained, (3) that a predetermined number of responses to question blocks have been obtained, (4) that a weighted treatment loss function value exceeds a predetermined value after hypothetical or actual administration of one or more treatments, (5) that a weighted treatment loss function value exceeds a predetermined value after hypothetical or actual administration of one or more questions, (6) that weighted treatment loss function value exceeds a predetermined value after hypothetic or actual administration of one or more topics, (7) the combination of one or more treatment stopping rules, (8) based on one or more responses, (9) based on one or more response function values, (10) that a predetermined number of treatment types have been administered, and (11) that a predetermined number of treatments have been administered;

otherwise;

(i) repeating method from step (d) unless a method termination rule is satisfied.

114. The method of claim 113 wherein a treatment loss function incorporates one or more of the group consisting of (1) a cost of administered treatment types, (2) a cost of administered treatments (3) a cost of administered questions, (4) a cost of administered topics, (5) response function values, and (6) a function of a state in a domain.

115. The method of claim 99 wherein there is a one-to-one correspondence between a plurality of test items and a plurality of questions, a response distribution for a test item being the same as a response distribution for a corresponding question.

116. The method of claim 99 further comprising the steps:

(h) repeating method from step (e) for one or more active topics, an active topic being a topic for which one or more treatment stopping rules have not been satisfied;

otherwise;

(i) repeating method from step (d) unless a method termination rule is satisfied.

117. The method of claim 1 wherein in step (e) an SPS is assigned to a first domain based on the SPS obtained for a second domain.