LABELING DATA SAMPLES USING OBJECTIVE QUESTIONS

• US 20100318539A1
• Filed: 06/15/2009
• Published: 12/16/2010
• Est. Priority Date: 06/15/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kenthapadi, Krishnaram, Kannan, Anitha, Fuxman, Ariel, Shafer, John C.
• Granted Patent: US 8,788,498 B2
• Time in Patent Office: Days
• US Class Current: 707/759
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

Described is a technology for obtaining labeled sample data. Labeling guidelines are converted into binary yes/no questions regarding data samples. The questions and data samples are provided to judges who then answer the questions for each sample. The answers are input to a label assignment algorithm that associates a label with each sample based upon the answers. If the guidelines are modified and previous answers to the binary questions are maintained, at least some of the previous answers may be used in re-labeling the samples in view of the modification.

Claims

1. In a computing environment, a method comprising, inputting answers to a set of objective questions regarding a data sample, determining a label for the data sample based upon the answers, and associating the label with the data sample.

2. The method of claim 1 wherein the questions comprise binary questions and wherein inputting the answers comprises receiving data corresponding to a yes or no answer for each question asked.

3. The method of claim 1 wherein the questions are arranged in a tree-like structure in which the answer to a question determines whether to ask a subsequent question, or determines which path to take to ask a subsequent question.

4. The method of claim 3 further comprising, producing a label assignment algorithm that determines the label by examining the paths of the questions posed and the answers, including traversing the tree-like structure in a depth first fashion, and associating each path ending in a leaf node to a label from a label set.

5. The method of claim 1 further comprising, producing the objective questions based upon a set of guidelines.

6. The method of claim 1 wherein a plurality of sets of answers are input for a data sample, and further comprising, combining a plurality of labels corresponding to the plurality of sets of answers to determine a single label for associating with the data sample.

7. The method of claim 1 further comprising, maintaining the answers to the questions as previous answers, receiving a modification to the set of questions, and reusing at least some of the previous answers for each sample to re-label the samples based upon the modification to the set of questions.

8. The method of claim 7 wherein receiving the modification to the set of questions comprises adding at least one question, changing at least one question, or deleting at least one question, or any combination of adding at least one question, changing at least one question, or deleting at least one question.

9. The method of claim 1 wherein the answers to the objective questions are received from a plurality of judges, and further comprising, identifying bad questions through inconsistencies across judges, or identifying bad judges through inconsistent answers, or both identifying bad questions through inconsistencies across judges and identifying bad judges through inconsistent answers.

10. The method of claim 1 further comprising maintaining the answers to the questions as previous answers, receiving a modification to a set of labels from which the label is selected, and reusing at least some of the previous answers for the sample to re-label the sample based upon the modification to the set of labels.

11. In a computing environment, a system comprising, a label assignment mechanism that assigns a label from among a set of labels to a data sample, the label assignment mechanism configured to determine which label to assign to the data sample based upon a set of answers to objective questions posed to at least one human judge.

12. The system of claim 11 wherein the sample corresponds to a search query, and wherein the objective questions result in the sample being labeled as having commercial intent or not having commercial intent.

13. The system of claim 11 wherein the sample corresponds to a search query and a document, and wherein the objective questions result in the sample being labeled with data indicating the document'"'"'s perceived relevance to the search query.

14. The system of claim 13 wherein the document corresponds to a commercial product.

15. The system of claim 11 wherein the objective questions comprise binary questions.

16. The system of claim 11 wherein the questions are based on a set of guidelines, and further comprising, means for changing at least one of the questions or changing at least one of the labels, or changing at least one of the questions and at least one of the labels, based upon a modification to the set of guidelines.

17. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising,

providing samples and a set of questions to a plurality of judges;

obtaining answers to the questions from each of the judges with respect to each sample; and

using the answers for each sample to determine which label of a finite set of labels to associate with that sample.

18. The one or more computer-readable media of claim 17 wherein using the answers for each sample to determine which label to associate with that sample comprises using the answers from the judges to determine votes in a voting scheme, and having further computer-executable instructions comprising, identifying bad questions through inconsistencies across judges, or identifying bad judges through inconsistent answers, or both identifying bad questions through inconsistencies across judges and identifying bad judges through inconsistent answers.

19. The one or more computer-readable media of claim 17 having further computer-executable instructions comprising, maintaining the answers to the questions as previous answers, receiving a modification to the set of questions, and reusing at least some of the previous answers for each sample to re-label the samples based upon the modification to the set of questions.

20. The one or more computer-readable media of claim 17 having further computer-executable instructions comprising, maintaining the answers to the questions as previous answers, receiving a modification to the set of labels, and reusing at least some of the previous answers for each sample to re-label the samples based upon the modification to the set of labels.