Combining Human and Machine Intelligence to Solve Tasks With Crowd Sourcing

• US 20120005131A1
• Filed: 06/30/2010
• Published: 01/05/2012
• Est. Priority Date: 06/30/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Horvitz, Eric, Koch, Paul B., Hacker, Severin
• Granted Patent: US 9,305,263 B2
• Time in Patent Office: Days
• US Class Current: 706/11
• CPC Class Codes: G06N 5/043 Distributed expert systems; Blackboards
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services

Abstract

Methods are described for ideally joining human and machine computing resources to solve tasks, based on the construction of predictive models from case libraries of data about the abilities of people and machines and their collaboration. Predictive models include methods for folding together human contributions, such as voting, with machine computation, such as automated visual analyses, as well as the routing of tasks to people based on prior performance and interests. An optimal distribution of tasks to selected participants of the plurality of participants is determined according to a model that considers the demonstrated competencies of people based on a value of information analysis that considers the value of human computation and the ideal people for providing a contribution.

Claims

1. A computer-implemented method for routing and solving tasks in a crowd sourcing application, the method comprising computer-implemented operations for:

receiving computer-based guidance for solving a task from a computer-based resource;

receiving human-based contributions for solving the task;

generating a model for combining the computer-based guidance and the human-based contributions; and

generating a global solution to the task by combining the computer-based guidance and the human-based contributions according to the model.

2. The computer-implemented method of claim 1, wherein receiving human-based contributions for solving the task comprises:

generating an optimal distribution of the task to selected participants in a plurality of participants;

routing the task to the selected participants according to the optimal distribution; and

receiving the human-based contributions for solving the task from the selected participants.

3. The computer-implemented method of claim 2, wherein generating an optimal distribution of the task to selected participants in a plurality of participants comprises:

generating a second model for distributing the task; and

generating the optimal distribution of the task to the selected participants in the plurality of participants according to the second model.

4. The computer-implemented method of claim 3, wherein generating the optimal distribution of the task to the selected participants in the plurality of participants according to a second model comprises training the second model with prior performance of the plurality of participants with respect to solving the task.

5. The computer-implemented method of claim 3, wherein generating the optimal distribution of the task to the selected participants in the plurality of participants according to a second model comprises training the second model with interests of the plurality of participants.

6. The computer-implemented method of claim 1, wherein the task comprises classifying an image with a selected image identifier from a plurality of image identifiers.

7. The computer-implemented method of claim 6, wherein receiving computer-based guidance comprises receiving predictive output from a probabilistic model based on features of the image identified through computer vision techniques.

8. The computer-implemented method of claim 6, wherein receiving human-based contributions for solving the task comprises receiving human votes, each of the human votes classifying the image with one of the plurality of image identifiers.

9. The computer-implemented method of claim 6, wherein generating a global solution to the task by combining the computer-based guidance and the human-based contributions according to the model comprises assigning a probability distribution that each of the plurality of image identifiers correctly classifies the image.

10. The computer-implemented method of claim 1, wherein generating a global solution to the task by combining the computer-based guidance and the human-based contributions according to the model comprises:

determining an expected value of information of the human-based contributions; and

generating the global solution to the task based on the expected value of information of the human-based contributions.

11. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a computer, cause the computer to:

generate a model for routing a task;

generate an optimal distribution of a task to selected participants in a plurality of participants according to the model;

route the task to the selected participants according to the optimal distribution; and

receive human-based contributions for solving the task from the selected participants.

12. The computer-readable storage medium of claim 11, wherein to generate the optimal distribution of the task to the selected participants in the plurality of participants according to a second model, the computer-readable storage medium having computer-executable instructions stored thereon which, when executed by the computer, cause the computer to train the second model with prior performance of the plurality of participants with respect to solving the task.

13. The computer-readable storage medium of claim 11, wherein to generate the optimal distribution of the task to the selected participants in the plurality of participants according to a second model, the computer-readable storage medium having computer-executable instructions stored thereon which, when executed by the computer, cause the computer to train the second model with interests of the plurality of participants.

14. The computer-readable storage medium of claim 11 having computer-executable instructions stored thereon which, when executed by the computer, further cause the computer to:

receive computer-based guidance for solving a task from a computer-based resource;

generate a second model for combining the computer-based guidance and the human-based contributions; and

generate a global solution to the task by combining the computer-based guidance and the human-based contributions according to the second model.

15. The computer-readable storage medium of claim 11, wherein the model comprises a probabilistic model.

16. The computer-readable storage medium of claim 11, wherein the model comprises a decision-theoretic model that is used to route the task based on a value of information analysis.

17. The computer-readable storage medium of claim 11, wherein to generate a global solution to the task by combining the computer-based guidance and the human-based contributions according to the model, the computer-readable storage medium having computer-executable instructions stored thereon which, when executed by the computer, cause the computer to:

determine an expected value of information of the human-based contributions; and

generate the global solution to the task based on the expected value of information of the human-based contributions.

18. A computer system for routing and solving tasks in a crowd sourcing application, comprising:

a processor;

a memory communicatively coupled to the processor; and

an task distribution module which executes in the processor from the memory and which, when executed by the processor, causes the computer system to route and solve tasks in the crowd sourcing application by

generating a model based on machine learning techniques,

generating an optimal distribution of the task to selected participants in a plurality of participants according to the model,

routing the task to the selected participants according to the optimal distribution,

receiving the human-based contributions for solving the task from the selected participants,

receiving computer-based guidance for solving a task from a computer-based resource, and

generating a global solution to the task by combining the computer-based guidance and the human-based contributions according to the model.

19. The computer system of claim 18, wherein the task comprises classifying an image with a selected image identifier from a plurality of image identifiers.

20. The computer system of claim 18, wherein receiving computer-based guidance comprises receiving predictive output from a probabilistic model based on features of the image identified through computer vision techniques;

wherein receiving human-based contributions for solving the task comprises receiving human votes, each of the human votes classifying the image with one of the plurality of image identifiers; and

wherein generating a global solution to the task by combining the computer-based guidance and the human-based contributions according to the model comprises assigning a probability distribution that each of the plurality of image identifiers correctly classifies the image.