GENERALIZED ACTIVE LEARNING

• US 20100332423A1
• Filed: 06/24/2009
• Published: 12/30/2010
• Est. Priority Date: 06/24/2009
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Horvitz, Eric, Kapoor, Ashish
• Application Number:
  US12/490,449
• Publication Number:
  US 20100332423A1
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 5/045 Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence

Abstract

Active learning is extended to decisions on information acquisition of both missing labels and missing features within one or more cases. In one example, desired (e.g., optimal) information to acquire about a case at hand and about cases in a training library during diagnostic sessions can be computed concurrently. A joint distribution of variables, comprising observed and unobserved labels and features for one or more cases, is modeled and probability distributions are determined for unobserved variables. An unobserved variable is selected from the joint distribution that has a return on information (ROI) metric having a combination of a desired uncertainty metric for a value of the unobserved variable and a desired cost for observing the value of the unobserved variable. The value of the variable is observed, and the probability distributions for the respective unobserved variables in the joint distribution are updated using the value of the identified variable.

Claims

1. A method for active learning that includes decisions on information acquisition of both missing labels and missing features within one or more cases, executed via a processor on a computer comprising a memory whereon computer-executable instructions comprising the method are stored, the method comprising:

modeling a joint distribution of variables, comprising observed and unobserved labels and features, for one or more cases;

determining probability distributions for respective unobserved variables;

identifying an unobserved variable from the joint distribution of variables that has a return on information (ROI) metric corresponding to a combination of a desired uncertainty metric for a value of the unobserved variable and a desired cost for observing the value of the unobserved variable;

observing the value of the identified variable; and

updating the probability distributions for the respective unobserved variables in the joint distribution of variables utilizing the value of the identified variable.

2. The method of claim 1, where the modeling of a joint distribution of variables, comprising observed and unobserved labels and features, for one or more cases is represented with an undirected graphical model.

3. The method of claim 1, determining probability distributions for respective unobserved variables using the undirected graphical model of the joint distribution of variables to create a predictive model for unobserved features and labels.

4. The method of claim 2, identifying the unobserved variable in order to determine a desired label value to be observed for training the predictive model.

5. The method of claim 2, identifying the unobserved variable in order to determine a desired feature value to be observed for making a label prediction for a case using the predictive model.

6. The method of claim 1, identifying the unobserved variable comprising selecting the unobserved variable that has a desired ROI metric.

7. The method of claim 5, comprising determining a ROI metric comprising comparing the uncertainty metric for the unobserved variable to the cost for observing the value of the unobserved variable.

8. The method of claim 5, comprising determining the uncertainty metric for the unobserved variable comprising determining a probability of an unobserved variable from a case given a set of observed variables for the case in the joint distribution of variables.

9. The method of claim 5, comprising determining the uncertainty metric for the unobserved variable comprising identifying an unobserved variable for a case from a set of unobserved variables for the case that yields a desired expected information gain for the set of unobserved variables for the case.

10. The method of claim 8, comprising determining the expected information gain for the set related unobserved variables for the case comprising determining a reduction in uncertainty for the set of related unobserved variables for the case if the selected unobserved variable for the case is observed.

11. The method of claim 5, comprising determining the cost for observing the value of the unobserved variable comprising:

defining a set of cost related parameters;

determining a value for the respective cost related parameters for observing the value of the unobserved variable; and

combining the respective cost related parameters'"'"' values to determine the cost for observing the value of the unobserved variable.

12. The method of claim 1, observing the value of the identified variable comprising one of:

performing a test to determine the value of the identified variable; and

using an information source having a known value for the identified variable.

13. The method of claim 11, determining a value for the respective cost related parameters for observing the value of the unobserved variable comprising one of:

determining a value for the respective cost related parameters for performing a test to determine the value of the identified variable; and

determining a value for the respective cost related parameters for using an information source having a known value for the identified variable.

14. A system for active learning that includes decisions on information acquisition of both missing labels and missing features within one or more cases, comprising:

a variable modeling component configured to model a joint distribution of variables as an undirected graphical model, where the joint distribution of variables comprise observed and unobserved labels and features for one or more cases;

a probability distribution determination component configured to determine probability distributions for the respective unobserved variables in the joint distribution of variables;

a variable identification component configured to identify an unobserved variable from the joint distribution of variables that has a return on information (ROI) metric corresponding to a combination of a desired uncertainty metric for a value of the unobserved variable and a desired cost for observing the value of the unobserved variable;

a value observation component configured to observe the value of the identified variable; and

a probability distribution updating component configured to update the probability distributions for the respective unobserved variables in the joint distribution of variables utilizing the value of the identified variable.

15. The system of claim 14, comprising a predictive model created by combining the undirected graphical model of the joint distribution of variables with the probability distributions for the respective unobserved variables in the joint distribution of variables, and configured to provide for determination of probability values for unobserved features and labels.

16. The system of claim 14, comprising a ROI determination component configured to determine a ROI metric for unobserved variables, comprising a combination of the uncertainty metric for the unobserved variable with the cost for observing the value of the unobserved variable.

17. The system of claim 16, comprising an uncertainty determination component configured to determine the uncertainty metric for the unobserved variable comprising determining a probability of an unobserved variable from a case given a set of observed variables for the case in the joint distribution of variables.

18. The system of claim 14, the variable identification component configured to select the unobserved variable for a case from a set of unobserved variables for the case that yields:

a desired expected information gain for the set of unobserved variables for the case; and

a desired cost for observing the value of the unobserved variable for the case.

19. A method for using an expected value of information to compute a desired next piece of information to gather about one or more diagnostic cases, executed via a processor on a computer comprising a memory whereon computer-executable instructions comprising the method are stored, comprising:

comparing an expected value of acquiring information on extensions to a case library of training data and information known about one or more cases; and

determining a desired next piece of information for the one or more diagnostic cases based on the comparison.

20. A method for active learning that includes decisions on information acquisition of both missing labels and missing features within one or more cases, executed via a processor on a computer comprising a memory whereon computer-executable instructions comprising the method are stored, the method comprising:

modeling a joint distribution of variables, comprising observed and unobserved labels and features, for one or more cases as an undirected graphical model;

determining probability distributions for respective unobserved variables;

creating a predictive model for unobserved features and labels using the probability distributions for respective unobserved variables for the undirected graphical model of the joint distribution of variables;

identifying the unobserved variable comprising selecting the unobserved variable that has a desired return on information (ROI) metric, comprising;

determining an uncertainty metric for the unobserved variable comprising determining a probability of an unobserved variable from a case given a set of observed variables for the case in the joint distribution of variables;

determining the cost for observing the value of the unobserved variable comprising;

defining a set of cost related parameters;

determining a value for the respective cost related parameters for observing the value of the unobserved variable; and

combining the respective cost related parameters'"'"' values to determine the cost for observing the value of the unobserved variable; and

determining a ROI metric comprising comparing the uncertainty metric for the unobserved variable to the cost for observing the value of the unobserved variable;

observing the value of the identified variable, comprising;

performing a test to determine the value of the identified variable; and

using an information source having a known value for the identified variable; and

updating the probability distributions for the respective unobserved variables in the joint distribution of variables utilizing the value of the identified variable.