Combining human and machine intelligence to solve tasks with crowd sourcing

• US 9,305,263 B2
• Filed: 06/30/2010
• Issued: 04/05/2016
• Est. Priority Date: 06/30/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Horvitz, Eric, Koch, Paul B., Hacker, Severin
• Primary Examiner(s): Wong, Lut
• Application Number:
  US12/827,307
• Publication Number:
  US 20120005131A1
• Time in Patent Office: 2,106 Days
• Field of Search: 706/11, 706/45
• US Class Current: 1/1
• CPC Class Codes: G06N 5/043 Distributed expert systems; Blackboards
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services

Abstract

Methods are described for ideally joining human and machine computing resources to solve tasks, based on the construction of predictive models from case libraries of data about the abilities of people and machines and their collaboration. Predictive models include methods for folding together human contributions, such as voting, with machine computation, such as automated visual analyses, as well as the routing of tasks to people based on prior performance and interests. An optimal distribution of tasks to selected participants of the plurality of participants is determined according to a model that considers the demonstrated competencies of people based on a value of information analysis that considers the value of human computation and the ideal people for providing a contribution.

Claims

1. A computer-implemented method, comprising:

receiving computer-based guidance for solving a task from a computer-based resource;

receiving historical human resource experience information associated with a plurality of participants, the historical human resource experience information indicating an experience level for individuals of the plurality of participants with respect to solving previous tasks;

receiving historical human resource activity information associated with the plurality of participants, the historical human resource activity information indicating an availablility and an activity level for the individuals of the plurality of participants with respect to solving the previous tasks or other previous tasks;

generating a distribution of the task to selected participants in the plurality of participants based on the computer-based guidance, the historical human resource experience information, and the historical human resource activity information; and

generating a global solution to the task based on the computer-based guidance and human-based contributions from the selected participants.

2. The computer-implemented method of claim 1, further compromising:

routing the task to the selected participants in the plurality of participants according to the distribution; and

receiving the human-based contributions from the selected participants in the plurality of participants.

3. The computer-implemented method of claim 1, wherein the historical human resource experience information is associated with backgrounds of the individuals.

4. The computer-implemented method of claim 3, wherein the historical human resource activity information comprises an activity pattern indicating human resource productivity for the individuals of the plurality of participants with respect to solving the previous tasks or the other previous tasks.

5. The computer-implemented method of claim 1, further comprising generating a model for combining the computer-based guidance and human-based contributions for solving the task from the selected participants in the plurality of participants, wherein the global solution is generated according to the model.

6. The computer-implemented method of claim 1, wherein the task comprises classifying an image with a selected image identifier from a plurality of image identifiers.

7. The computer-implemented method of claim 6, wherein the receiving the computer-based guidance comprises receiving predictive output from a probabilistic model based on features of the image identified through computer vision techniques.

8. The computer-implemented method of claim 6, wherein the receiving the human-based contributions comprise human votes, wherein individual human votes classify the image with one of the plurality of image identifiers.

9. The computer-implemented method of claim 6, wherein the generating the global solution to the task is based on a probability distribution, wherein the probability distribution is generated based on whether individual image identifiers of the plurality of image identifiers correctly classify the image.

10. The computer-implemented method of claim 1, wherein the generating the distribution of the task further comprises:

determining an expected value of information of the human-based contributions; and

generating the distribution of the task based on the expected value of information of the human-based contributions.

11. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a computer, cause the computer to:

generate a model for routing a task;

generate a distribution of the task to selected participants in a plurality of participants according to the model, wherein the distribution is based on historical human resource experience and activity information indicating an experience level and an activity level for individuals of the plurality of participants with respect to solving previous tasks, the historical human resource exrerience and activity information beinci received prior to beginning processing of the task;

route the task to the selected participants according to the distribution; and

receive human-based contributions for solving the task from the selected participants.

12. The computer-readable storage medium of claim 11, wherein the previous tasks include different types of tasks than the task, and wherein the computer-executable instructions further cause the computer to train the model with prior performance of the individuals of the plurality of participants with respect to solving the previous tasks including the different types of tasks.

13. The computer-readable storage medium of claim 11, wherein the computer-executable instructions further cause the computer to train the model with interests of the individuals of the plurality of participants.

14. The computer-readable storage medium of claim 11, wherein the computer-executable instructions further cause the computer to:

receive computer-based guidance for solving the task, the task being from a computer-based resource;

generate a second model for combining the computer-based guidance and the human-based contributions; and

generate a global solution to the task by combining the computer-based guidance and the human-based contributions according to the second model.

15. The computer-readable storage medium of claim 11, wherein the model uses output from a probabilistic model to generate the distribution of the task.

16. The computer-readable storage medium of claim 11, wherein the model comprises a decision-theoretic model that is used to route the task based on a value of information analysis.

17. The computer-readable storage medium of claim 11, wherein the computer-executable instructions further cause the computer to:

determine an expected value of information for the individuals of the plurality of participants; and

generate the distribution of the task based on the expected value of information.

18. A computer system, comprising:

a processor;

a memory communicatively coupled to the processor; and

a task distribution module which executes in the processor from the memory and which, when executed by the processor, causes the computer system to route and solve tasks in a crowd sourcing application by;

generating a model based on machine learning techniques,

generating a distribution of an individual task to selected participants in a plurality of participants according to the model, the distribution based on historical human resource experience and activity information that was collected during completion of a previous crowd sourcing application and prior to generation the model,

routing the individual task to the selected participants according to the distribution,

receiving human-based contributions for solving the individual task from the selected participants,

receiving computer-based guidance for solving the individual task from a computer-based resource, and

generating a global solution to the individual task by combining the computer-based guidance and the human-based contributions according to the model.

19. The computer system of claim 18, wherein the individual task comprises classifying an image with a selected image identifier from a plurality of image identifiers.

20. The computer system of claim 19, wherein:

receiving the computer-based guidance comprises receiving predictive output from a probabilistic model based on features of the image identified through computer vision techniques;

receiving the human-based contributions comprises receiving human votes, individual human votes classifying the image with one of the plurality of image identifiers; and

generating the global solution to the individual task is based on a probability distribution, wherein the probability distribution is generated based on whether individual image identifiers of the plurality of image identifiers correctly classify the image.