GENERATING GOLD QUESTIONS FOR CROWDSOURCING

• US 20150235160A1
• Filed: 02/20/2014
• Published: 08/20/2015
• Est. Priority Date: 02/20/2014
• Status: Abandoned Application

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Larlus-Larrondo, Diane, Perronnin, Florent C., Mishra, Vivek Kumar, Kompalli, Pramod Sankar
• Application Number:
  US14/184,936
• Publication Number:
  US 20150235160A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/06398 Performance of employee with respect to a job function

Abstract

A system and method for generating gold questions for labeling tasks are disclosed. The method includes sampling a positive class from a predefined set of classes to be used in labeling documents, based on a computed measure of class popularity. A set of negative classes is identified from the set of classes based on a distance measure between the positive class and other classes in the set of classes. A gold question is generated which includes a document representative of the positive class and a set of candidate answers. The candidate answers include a label for the positive class and a label for each of the negative classes in the identified set of negative classes. A task may be generated which includes the gold question and a plurality of standard questions which each include a document to be labeled. A computer processor may implement all or part of the method.

Claims

1. A method for generating a gold question for a labeling task comprising:

sampling a positive class from a predefined set of classes to be used in labeling documents, based on a computed measure of class popularity;

for the positive class, identifying a set of negative classes from the set of classes based on a distance measure between the positive class and other classes in the set of classes;

generating a gold question which includes a document representative of the positive class and a set of candidate answers, the candidate answers including a label for the positive class and a label for each of the negative classes in the identified set of negative classes; and

outputting the gold question,

wherein at least one of the sampling, identifying, and generating is performed with a computer processor.

2. The method of claim 1, further comprising, for each of the classes in the predefined set of classes, computing the measure of class popularity.

3. The method of claim 1, wherein the sampling of the positive class comprises identifying a set of positive classes from the predetermined set of classes based on a computed measure of class popularity for each of at least some of the classes in the predetermined set of classes and the sampling includes sampling a class from the set of positive classes.

4. The method of claim 1, wherein the sampling of the positive class includes sampling from at least a subset of the classes with a probability that is an increasing function of a computed measure of class popularity for the at least a subset of the classes.

5. The method of claim 1, wherein the measure of class popularity is derived from public resources.

6. The method of claim 1, wherein the measure of class popularity is based on at least one of:

a quantity of hits returned by a search engine when queried with the class label;

a quantity of hits returned by a search engine when queried with the class label for documents of a same type as the documents to be labeled;

a quantity of groups on a document-sharing website that are linked to the class; and

a quantity of documents of the type to be labeled which are submitted to groups on a document-sharing website that are linked to the class.

7. The method of claim 1, wherein the identifying of the set of negative classes comprises at least one of:

identifying a pool of negative classes, the set of negative classes being sampled from the pool, and

sampling negative classes from at least a subset of the set of classes with a probability which is an increasing function of a distance between the sampled positive class and the sampled negative classes.

8. The method of claim 1, further comprising, computing the distance measure between the sampled positive class and other classes in the set of classes.

9. The method of claim 8, wherein the distance measure is computed based on a distance between the positive class and the other classes in an embedding space.

10. The method of claim 1, wherein the method includes, for each of at least some of the classes in the set of classes, computing a feature vector, the distance measure being computed as a function of a distance between the feature vectors.

11. The method of claim 9, wherein the feature vectors include values for a set of features, the features being based on at least one of class attributes and an ontology of classes.

12. The method of claim 1, further comprising generating a labeling task by combining the gold question with a set of standard questions, each of the standard questions including a document to be labeled and a set of candidate answers, the candidate answers including labels for at least a subset of classes from the set of classes.

13. The method of claim 11, wherein the subset of classes for the document to be labeled is identified by classifying the document to be labeled with a classifier.

14. The method of claim 11, further comprising submitting the task to a crowdsourcing marketplace for crowdworkers to perform the task.

15. The method of claim 14, further comprising receiving answers to the gold question and standard questions from a crowdworker and determining a reliability of the crowdworker by comparing an answer to the gold question with the label of the for the positive class.

16. The method of claim 1, wherein the documents to be labeled comprise photographic images.

17. A computer program product comprising a non-transitory recording medium storing instructions, which when executed on a computer causes the computer to perform the method of claim 1.

18. A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

19. A system for generating a gold question for a labeling task comprising:

a positive class selector for sampling a positive class from a predefined set of classes to be used in labeling documents, the sampling being based on a computed measure of class popularity;

a negative class selector for identifying a set of negative classes from the predefined set of classes based on a distance measure between the positive class and other classes in the set of classes;

a gold question generator which generates a gold question that includes a document representative of the positive class and a set of candidate answers, the candidate answers including a label for the positive class and a label for each of the negative classes in the identified set of negative classes;

a task outsource component which outputs a task including the gold question; and

a computer processor which implements the positive class selector, negative class selector, and gold question generator.

20. The system of claim 19, wherein the system further comprises a task generator which generates the task by combining the gold question with a set of standard questions, without distinguishing between the gold question and the standard questions in the task, each of the standard questions including a document to be labeled and a set of candidate answers, the candidate answers including labels for at least a subset of classes from the set of classes.

21. The system of claim 19, further comprising a classification component which identifies a set of class labels for each of the standard questions based on the respective document to be labeled.

22. A method for generating a human intelligence task comprising:

computing a measure of popularity for each of a set of classes to be used in labeling documents;

sampling a positive class from the set of classes based on the computed measure of popularity;

identifying a set of negative classes from the set of classes based on a distance measure between the positive class and other classes in the set of classes;

generating a gold question which includes a document representative of the positive class and a set of candidate answers, the candidate answers including a label for the positive class and a label for each of the negative classes in the identified set of negative classes; and

generating a human intelligence task comprising combining the gold question with a set of standard questions, each of the standard questions including a document to be labeled and a set of candidate answers, the candidate answers including labels for at least a subset of classes from the set of classes; and

outputting the human intelligence task,

wherein at least one of the computing, sampling, identifying, generating the gold question, and generating the task is performed with a computer processor.