Method and apparatus for analysis and decomposition of classifier data anomalies

• US 20060047617A1
• Filed: 08/31/2004
• Published: 03/02/2006
• Est. Priority Date: 08/31/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chandrasekar, Raman, Wong, Leon Chih Wen, Leonard, Peter F., Sauntry, David Michael, Bacioiu, Ana Sultana, Boyle, James Scott
• Granted Patent: US 7,426,497 B2
• Time in Patent Office: Days
• US Class Current: 706/59
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A human assisted method of debugging training data used to train a machine learning classifier is provided. The method includes obtaining a classifier training data set. The training data set is then debugged using an integrated debugging tool configured to implement a debugging loop to obtain a debugged data set. The debugging tool can be configured to perform an estimation and simplification step to reduce data noise in the training data set prior to further analysis. The debugging tool also runs a panel of prediction-centric diagnostic metrics on the training data set, and provides the user prediction based listings of the results of the panel of prediction-centric diagnostic metrics.

Claims

1. A human assisted method of debugging training data used to train a machine learning classifier, the method comprising:

obtaining a classifier training data set; and

debugging the training data set using an integrated debugging tool configured to implement a debugging loop to obtain a debugged data set.

2. The method of claim 1, wherein debugging the training data set using the integrated debugging tool further comprises debugging the training data set using a computer implemented integrated debugging tool.

3. The method of claim 2, wherein debugging the training data set using the integrated debugging tool further comprises:

determining with the integrated debugging tool whether data noise in the training data set exceeds a threshold; and

performing an estimation and simplification step, with the integrated debugging tool, on the training data set if the data noise in the training data set exceeds the threshold to obtain a simplified training data set.

4. The method of claim 3, wherein the data noise is a distribution skewness type of data noise.

5. The method of claim 3, wherein the data noise is an ambiguity type of data noise.

6. The method of claim 3, wherein the step of performing the estimation and simplification step, with the integrated debugging tool, further comprises identifying hypothetical fixes to the training data set and estimating effects of the hypothetical fixes on training data errors.

7. The method of claim 6, wherein the step of identifying hypothetical fixes to the training data set and estimating effects of the hypothetical fixes on training data errors further comprises reducing data errors in the training data set caused by at least one of distribution skewness and ambiguity, thereby exposing other types of data errors in the training data set.

8. The method of claim 2, wherein debugging the training data set using the integrated debugging tool further comprises:

running a panel of prediction-centric diagnostic metrics on the training data set; and

providing to a user prediction based listings of the results of the panel of prediction-centric diagnostic metrics.

9. The method of claim 8, wherein the step of providing to the user the prediction based listings of the results of the panel of prediction-centric diagnostic metrics further comprises providing user configurable prediction based listings of the results.

10. The method of claim 9, wherein providing the user configurable prediction based listings of the results further comprises providing to the user sortable prediction based listings of the results.

11. The method of claim 9, wherein providing the user configurable prediction based listings of the results further comprises providing to the user filtered prediction based listings of the results.

12. The method of claim 9, wherein providing the user configurable prediction based listings of the results further comprises generating a graphical user interface which displays the prediction based listings of the results, and which is configured to receive user inputs and in response to configure the prediction based listings of the results.

13. The method of claim 12, wherein generating the graphical user interface further comprises highlighting statistical outliers in the prediction based listings of the results.

14. The method of claim 12, wherein generating the graphical user interface further comprises highlighting failed queries to associate the failed queries with failure causes.

15. The method of claim 14, wherein highlighting failed queries to associate the failed queries with failure causes further comprises color coding failed queries by failure cause.

16. The method of claim 15, wherein the graphical user interface is configured to display identified probable causes of the failure of failed queries.

17. The method of claim 12, wherein the graphical user interface is configured to receive a user input corresponding to a prediction cluster, and in response to zoom into the prediction cluster to display individual predictions included in the prediction cluster.

18. The method of claim 2, and further comprising training the machine learning classifier using the debugged data set.

19. A classifier analyzer which provides human assisted debugging of training data used to train a machine learning classifier, the classifier analyzer being configured to implement steps comprising:

obtaining a classifier training data set; and

debugging the training data set using a debugging loop to obtain a debugged data set.

20. The classifier analyzer of claim 19, wherein the step of debugging the training data set further comprises:

determining whether data noise in the training data set exceeds a threshold; and

performing an estimation and simplification step on the training data set if the data noise in the training data set exceeds the threshold to obtain a simplified training data set.

21. The classifier analyzer of claim 20, wherein performing the estimation and simplification step further comprises identifying hypothetical fixes to the training data set and estimating effects of the hypothetical fixes on training data errors.

22. The classifier analyzer of claim 21, wherein the step of identifying hypothetical fixes to the training data set and estimating effects of the hypothetical fixes on training data errors further comprises reducing data errors in the training data set caused by at least one of distribution skewness and ambiguity, thereby exposing other types of data errors in the training data set.

23. The classifier analyzer of claim 19, wherein debugging the training data set using the debugging loop further comprises:

running a panel of prediction-centric diagnostic metrics on the training data set; and

providing to a user prediction based listings of the results of the panel of prediction-centric diagnostic metrics.

24. The classifier analyzer of claim 23, wherein the step of providing to the user the prediction based listings of the results of the panel of prediction-centric diagnostic metrics further comprises providing user configurable prediction based listings of the results.

25. The classifier analyzer of claim 24, wherein providing the user configurable prediction based listings of the results further comprises providing to the user sortable prediction based listings of the results.

26. The classifier analyzer of claim 24, wherein providing the user configurable prediction based listings of the results further comprises providing to the user filtered prediction based listings of the results.

27. The classifier analyzer of claim 24, wherein providing the user configurable prediction based listings of the results further comprises generating a graphical user interface which displays the prediction based listings of the results, and which is configured to receive user inputs and in response to configure the prediction based listings of the results.

28. The classifier analyzer of claim 27, wherein generating the graphical user interface further comprises highlighting statistical outliers in the prediction based listings of the results.

29. The classifier analyzer of claim 27, wherein generating the graphical user interface further comprises highlighting failed queries to associate the failed queries with failure causes.

30. The classifier analyzer of claim 29, wherein highlighting failed queries to associate the failed queries with failure causes further comprises color coding failed queries by failure cause.

31. The classifier analyzer of claim 30, wherein the graphical user interface is configured to display identified probable causes of the failure of failed queries.

32. The classifier analyzer of claim 27, wherein the graphical user interface is configured to receive a user input corresponding to a prediction cluster, and in response to zoom into the prediction cluster to display individual predictions included in the prediction cluster.