Active featuring in computer-human interactive learning

• US 9,430,460 B2
• Filed: 11/08/2013
• Issued: 08/30/2016
• Est. Priority Date: 07/12/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Simard, Patrice Y., Chickering, David Max, Grangier, David G., Lakshmiratan, Aparna, Amershi, Saleema A.
• Primary Examiner(s): Hill, Stanley K
• Assistant Examiner(s): Bharadwaj, Kalpana
• Application Number:
  US14/075,708
• Publication Number:
  US 20150019463A1
• Time in Patent Office: 1,026 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 1/0072 Error control for data other than payload data, e.g. control data
• CPC Class Codes: H04L 1/0079 Formats for control data H04L1/16 takes precedence; training sequences H04L25/00 and H04L27/00

Abstract

A collection of data that is extremely large can be difficult to search and/or analyze. Relevance may be dramatically improved by automatically classifying queries and web pages in useful categories, and using these classification scores as relevance features. A thorough approach may require building a large number of classifiers, corresponding to the various types of information, activities, and products. Creation of classifiers and schematizers is provided on large data sets. Exercising the classifiers and schematizers on hundreds of millions of items may expose value that is inherent to the data by adding usable meta-data. Some aspects include active labeling exploration, automatic regularization and cold start, scaling with the number of items and the number of classifiers, active featuring, and segmentation and schematization.

Claims

1. A system for facilitating interactive feature selection for machine learning, the method comprising:

one or more memory devices; and

one or more processors configured to;

provide a first training set of data items, wherein one or more of the data items have been previously labeled with example labels of a particular class of data item;

provide a classifier to be trained to determine labels for data items;

utilize the classifier to determine predicted labels for one or more of the data items that have been previously labeled with example labels;

identify one or more data items from the first training set having a discrepancy between a respective example label and a predicted label that was determined by the classifier;

present via a user interface an indication of the one or more data items from the first training set having the discrepancy between the respective example label and the predicted label, wherein the user interface includes a feature-selection interface configured to receive a user selection of one or more features that are utilized as input features to train the classifier and improve the accuracy of the predicted labels;

receive, via the user interface, the user selection of one or more features; and

train the classifier with the one or more user-selected features as input features;

wherein the one or more processors are further configured to iteratively repeat the steps of utilize, identify, present, receive, and train, with the first training set until a user input is received which indicates that feature selection is complete.

2. The system of claim 1, the one or more processors further configured to:

store user-selectable features; and

present to the user an option to select one or more of the user-selectable features;

wherein the one or more user-selected features include one or more features selected from among the user-selectable features.

3. The system of claim 2, wherein the user-selectable features include one or more of a built-in feature, a user-generated feature, a trained classifier, a trained segment extractor, or a dictionary.

4. The system of claim 2, wherein the user-selectable features include features generated by a plurality of users.

5. The system of claim 4, wherein the user-selectable features are stored on a commonly accessible system shared by multiple users.

6. The system of claim 1, the one or more processors further configured to:

upon receiving the user input that indicates the feature selection is complete, designate the classifier as a feature that is usable as an input feature to train classifiers or schematizers, wherein the classifier is frozen such that the classifier is not allowed to be retrained.

7. One or more hardware computer-readable media having embodied thereon computer-usable instructions that, when executed, facilitate a method of interactive feature selection for machine learning, the method comprising:

providing a first set of data items, wherein one or more of the data items include tokens that have been previously labeled as portions of a particular schema;

for the one or more data items, utilizing a schematizer to determine predicted labels for one or more of the tokens that have been previously labeled;

identifying one or more tokens having a discrepancy between a previous label and a predicted label;

presenting via a user interface an indication of the one or more tokens having the discrepancy between the previous label and the predicted label, wherein the user interface includes a feature-selection interface configured to receive a user selection of one or more features that are usable as input features to train the schematizer;

receiving, via the user interface, the user selection of one or more features; and

training the schematizer with the one or more user-selected features as input features.

8. The media of claim 7, the method further comprising:

storing user-selectable features; and

presenting to the user an option to select one or more of the user-selectable features;

wherein the one or more user-selected features include one or more features selected from among the user-selectable features.

9. The media of claim 8, wherein the user-selectable features include one or more of a built-in feature, a user-generated feature, a trained classifier, a trained segment extractor, or a dictionary.

10. The media of claim 8, wherein the user-selectable features include features generated by a plurality of users.

11. The media of claim 10, wherein the user-selectable features are stored on a commonly accessible system shared by multiple users.

12. The media of claim 7, the method further comprising repeating the scoring, identifying, presenting, receiving, and training.

13. The media of claim 12, the method further comprising:

upon receiving a user input that indicates the feature selection is complete, designating the schematizer as a feature that is usable as an input feature to train classifiers or schematizers, wherein the schematizer is frozen such that the schematizer is not allowed to be retrained.

14. A method of interactive feature selection for machine learning, comprising:

providing a first set of data items, wherein one or more of the data items have been previously labeled as examples of a particular class of data item;

utilizing a classifier to determine predicted labels for one or more of the data items;

identifying one or more data items having a discrepancy between a previous label and a predicted label;

presenting via a user interface an indication of the one or more data items having the discrepancy between the previous label and the predicted label, wherein the user interface includes a feature-selection interface configured to receive a user selection of one or more features that are usable as input features to train the classifier;

receiving a search query via the search interface;

executing the search query on the first set of data items, wherein search results are generated;

presenting the search results to the user;

receiving a user input that selects the search query as a first feature for training the classifier;

receiving, via the user interface, a user input that selects a dictionary as a second feature for training the classifier, wherein the dictionary includes words that define a concept that corresponds to the second feature;

training the classifier with the search query and the dictionary as input features;

utilizing the classifier to determine new predicted labels for one or more of the data items;

identifying one or more data items having a discrepancy between a previous label and a new predicted label;

presenting via the user interface an indication of the one or more data items having the discrepancy between the previous label and the new predicted label;

receiving, via the user interface, a user selection of one or more features; and

training the classifier with the one or more user-selected features as input features.

15. The method of claim 14, wherein the dictionary is designated from among one or more dictionaries generated by the user.

16. The method of claim 14, wherein the interface includes an interface for selecting a dictionary from among a group of dictionaries generated by a plurality of users.

17. The method of claim 15, the method further comprising:

receiving a user input that designates a second dictionary generated by another user; and

utilizing the second dictionary as an input feature to train the classifier.

18. The method of claim 14, wherein determining predicted labels and training the classifier occur transparently in the background at a specified interval while receiving user inputs that designate features for training the classifier.

19. The method of claim 14, wherein the one or more features that are usable as input features include features generated by a plurality of users.

20. The method of claim 14, wherein the one or more features that are usable as input features are stored on a commonly accessible system shared by multiple users.