EXPLORING DATA USING MULTIPLE MACHINE-LEARNING MODELS

• US 20110307422A1
• Filed: 06/09/2010
• Published: 12/15/2011
• Est. Priority Date: 06/09/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Drucker, Steven Mark, Tan, Desney S., Kapoor, Ashish, Patel, Kayur Dushyant, Fogarty, James Anthony
• Granted Patent: US 8,595,153 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning

Abstract

A multiple model data exploration system and method for running multiple machine-learning models simultaneously to understand and explore data. Embodiments of the system and method allow a user to gain a greater understanding of the data and to gain new insights into their data. Embodiments of the system and method also allow a user to interactively explore the problem and to navigate different views of data. Many different classifier training and evaluation experiments are run simultaneously and results are obtained. The results are aggregated and visualized across each of the experiments to determine and understand how each example is classified for each different classifier. These results then are summarized in a variety of ways to allow users to obtain a greater understanding of the data both in terms of the individual examples themselves and features associated with the data.

Claims

1. A method for exploring data in a dataset, comprising:

inputting the dataset containing labeled data;

generating multiple models from the dataset so that each of the multiple models is based on a different configuration of the dataset; and

simultaneously running different classifier training and evaluation experiments using multiple models to explore and understand the data based on results of running different classifier training and evaluation experiments.

2. The method of claim 1, further comprising systematically varying the multiple models and associated parameters during generation of the multiple models in order to obtain an optimally differentiated set of results or a set of results that matches constraints set by a user.

3. The method of claim 1, further comprising:

generating a different set of predicted labels for each of the multiple models;

aggregating each set of predicted labels to obtain aggregated predicted labels;

computing summary statistics for the aggregated predicted labels; and

exploring and understanding the data in terms of what data are outliers, what data are consistently confused, what data are mislabeled, and an effect that different features of the data have on classifying the data.

4. The method of claim 3, further comprising:

visualizing the summary statistics; and

interacting with the visualized summary statistics to explore the data and iteratively query and refine understanding of the data.

5. The method of claim 3, further comprising:

input an initial set of tuples from the labeled data, where the tuples correspond to labels for the labeled data;

generating a set of trials from the initial set of tuples; and

executing each trial in the set of trials.

6. The method of claim 5, further comprising:

computing summary statistics for sets of examples, trials, and sets of trials;

computing a percentage of correctly-classified tuples to determine accuracies for each of the multiple models;

comparing accuracies across the multiple models; and

computing label entropy.

7. The method of claim 5, further comprising:

selecting a trial from the set of trials;

determining whether the selected trial provides high accuracy;

if not, then discarding the selected trial;

if so, then adding the selected trial to a set of filtered trials; and

computing the summary statistics from the set of filtered trials.

8. The method of claim 7, further comprising:

examining an incorrectness and entropy graph;

allowing a user to interact with the incorrectness and entropy graph; and

verifying an actual label on the labeled data based on the incorrectness and entropy graph to facilitate correcting label noise.

9. A method for aggregating data into a plurality of visualizations to allow a user to gain insight about the data, comprising:

generating multiple models from a dataset containing labeled data;

generating a set of trials from an initial set of tuples corresponding to labels for the label data;

executing each trial in the set of trials;

generating a new set of tuples from the executed trials corresponding to predicted labels;

aggregating sets of predicted labels to obtain aggregated predicted labels;

computing summary statistics for the aggregated predicted labels;

visualizing the summary statistics using incorrectness versus entropy graphs; and

examining the visualized summary statistics to understand and explore trends in the data.

10. The method of claim 9, further comprising:

computing a percentage of correctly classified tuples for the aggregated predicted labels and actual labels of the labeled data to determine an accuracy of each of the multiple models; and

comparing accuracies across the multiple models.

11. The method of claim 10, further comprising:

computing incorrectness from the accuracies across the multiple models;

computing label entropy; and

generating the incorrectness versus entropy graphs using the incorrectness and label entropy.

12. The method of claim 11, further comprising:

selecting a trial from the set of trials to obtain a selected trial;

determining that the selected trial provides high accuracy;

adding the selected trial to a set of filtered trials; and

computing the summary statistics from the set of filtered trials

13. The method of claim 12, further comprising having the user interact with the incorrectness versus entropy graphs to verify the accuracy of the actual labels and to facilitate correcting label noise.

14. The method of claim 13, further comprising:

selecting a first incorrectness versus entropy graph that is for before a feature set was added;

examining the first incorrectness versus entropy graph to obtain unsure examples;

selecting a second incorrectness versus entropy graph that is for after the feature set was added; and

examining the second incorrectness versus entropy graph to obtain canonical examples.

15. The method of claim 14, further comprising:

filtering a space of examples to include an intersection of the unsure examples and the canonical examples; and

inspecting the space of examples to form links between a user'"'"'s understanding of data represented by the incorrectness versus entropy graphs and a machine learning algorithm'"'"'s understanding of the data.

16. A method for exploring visualizations of labeled data to formulate an optimal model for a machine learning problem, comprising:

inputting an initial set of tuples from the labeled data such that the tuples correspond to labels for the labeled data;

generating a set of trials from the initial set of tuples;

simultaneously executing each trial in the set of trials to run different classifier training and evaluation experiments using the multiple models;

generating a new set of tuples from the executed trials that correspond to predicted labels such that a different set of predicted labels is generated for each of the multiple models;

aggregating each set of predicted labels to obtain an aggregated set of predicted labels;

computing summary statistics for each set of predicted labels and actual labels for the labeled data;

visualizing the summary statistics using incorrectness versus entropy graphs; and

allowing a user to interact with the incorrectness versus entropy graphs to formulate the optimal model for the machine learning problem.

17. The method of claim 16, further comprising:

examining unsure examples to understand which unsure examples the multiple models disagree on; and

generating a set of discriminative features that describe data from the unsure examples that the multiple models disagree on.

18. The method of claim 17, further comprising:

summing together confusion matrices for each of trial in the set of trials to generate an aggregate confusion matrix;

normalizing across each of the rows in the confusion matrices so that each row becomes a distribution of predicted labels for a class; and

adding each row to the aggregate confusion matrix.

19. The method of claim 18, further comprising:

using the aggregate confusion matrix to organize examples in an unsure region;

changing the aggregate confusion matrix whenever filters are added; and

color coding each distribution of predicted labels for a class to highlight areas of highest confusion.

20. The method of claim 19, further comprising:

allowing the user to hover over a cell in the aggregate confusion matrix; and

presenting to the user an example of an actual class and an example of a predicted class that is most similar to the actual class; and

updating the set of discriminative features by allowing the user to change the set of discriminative features based on the aggregate confusion matrix to arrive at the optimal model for the machine learning problem.