SKILL ESTIMATION METHOD IN MACHINE-HUMAN HYBRID CROWDSOURCING

• US 20160132815A1
• Filed: 09/29/2015
• Published: 05/12/2016
• Est. Priority Date: 11/07/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Itoko, Toshinari, Kobayashi, Masatomo, Saito, Shin, Watanabe, Toshihiro
• Granted Patent: US 9,721,221 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/0639 Performance analysis of employees; Performance analysis of enterprise or organisation operations
• CPC Class Codes: G06Q 10/06398 Performance of employee with respect to a job function
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

In one embodiment, a computer-implemented method for estimating an ability of a worker in a process for integrating work results of multiple workers for the same task includes acquiring, for each of one or more tasks, a work result of a preceding-stage worker and a work result of a succeeding-stage worker that works based on the work result of the preceding-stage worker. The method also includes estimating multiple parameters of a probability model in which an ability of the succeeding-stage worker conditioned by a quality of the work result of the preceding-stage worker is introduced as a conditioned ability parameter, based on the multiple work results obtained for each of the one or more tasks.

Claims

1. A computer-implemented method for estimating an ability of a worker in a process for integrating work results of multiple workers for the same task, the method comprising:

acquiring, for each of one or more tasks, a work result of a preceding-stage worker and a work result of a succeeding-stage worker that works based on the work result of the preceding-stage worker; and

estimating multiple parameters of a probability model in which an ability of the succeeding-stage worker conditioned by a quality of the work result of the preceding-stage worker is introduced as a conditioned ability parameter, based on the multiple work results obtained for each of the one or more tasks.

2. The estimation method according to claim 1, wherein the probability model is a generative model that includes, for each of the one or more tasks, a work result of each of one or more preceding-stage workers as a first observed variable, a work result of each of one or more succeeding-stage workers as a second observed variable, and a work result to be obtained as a latent variable.

3. The estimation method according to claim 2, further comprising:

calculating, for each of the one or more tasks, a posterior probability of each of one or more given unique work results being the work result to be obtained, based on an estimated value of each of the multiple parameters of the probability model; and

outputting, for each of the one or more tasks, a work result that maximizes the posterior probability from among the one or more given unique work results as the work result to be obtained.

4. The estimation method according to claim 2, wherein the conditioned ability of the succeeding-stage worker includes, for each condition, an ability of the workers as a whole, an ability of each worker, or an ability of each worker for each task.

5. The estimation method according to claim 2, wherein the conditioned ability of the succeeding-stage worker includes, for each condition, the ability of each worker.

6. The estimation method according to claim 2,

wherein the work result of a worker is an answer of the worker to a task,

wherein the work result to be obtained is a true correct answer of the task, and

wherein the conditioned ability of the succeeding-stage worker includes a probability of the succeeding-stage worker giving the correct answer under a condition that an answer of the preceding-stage worker is correct, and a probability of the succeeding-stage worker giving the correct answer under a condition that the answer of the preceding-stage worker is incorrect.

7. The estimation method according to claim 2, wherein the conditioned ability of the succeeding-stage worker is further conditioned by a kind and individual of the preceding-stage worker or either the kind or the individual.

8. The estimation method according to claim 2, wherein the multiple parameters of the probability model further include an ability of the preceding-stage worker, and, for each of the one or more tasks, a prior probability of each of one or more given unique work results being the work result to be obtained.

9. The estimation method according to claim 2, wherein the multiple parameters of the probability model further include a degree of task difficulty.

10. The estimation method according to claim 2, wherein estimating the multiple parameters includes estimating a value of each of the multiple parameters that maximizes a posterior probability of the multiple parameters on condition that the work results of the preceding-stage and succeeding-stage workers for each of the one or more tasks are given.

11. The estimation method according to claim 2, wherein estimating the multiple parameters comprises:

calculating, for each of the one or more tasks, the work result to be obtained, based on a current value of each of the multiple parameters;

calculating a new value of each of the multiple parameters based on the calculated work result to be obtained for each of the one or more tasks; and

determining whether or not a convergence condition is satisfied.

12. The estimation method according to claim 11, wherein calculating the work result to be obtained comprises performing, for each of the tasks, weighted evaluation of each of given one or more unique work results by the ability of the preceding-stage and succeeding-stage workers as a whole based on a given work result of each of the preceding-stage and succeeding-stage workers.

13. The estimation method according to claim 11, wherein calculating the new value of each of the multiple parameters comprises explicitly determining, for at least the multiple parameters indicating the conditioned ability of the succeeding-stage worker, a new value from the work result to be obtained, which is calculated in a form of posterior distribution in calculating the work result to be obtained.

14. The estimation method according to claim 1, wherein the preceding-stage worker is a machine or a person, and the succeeding-stage worker is a person.

15. An estimation system for estimating an ability of a worker in a process for integrating work results of multiple workers for the same task, the system comprising:

an acquisition section configured to acquire, for each of one or more tasks, a work result of a preceding-stage worker and a work result of a succeeding-stage worker that works based on the work result of the preceding-stage worker; and

an estimation section configured to estimate multiple parameters of a probability model in which an ability of the succeeding-stage worker conditioned by a quality of the work result of the preceding-stage worker is introduced as a parameter, based on the multiple work results obtained for each of the one or more tasks.

16. The estimation system according to claim 15,

wherein the probability model is a generative model that includes, for each of the one or more tasks, a work result of each of the preceding-stage workers as a first observed variable, a work result of each of one or more succeeding-stage workers as a second observed variable, and a work result to be obtained, as a latent variable;

and

wherein the estimation system further comprises an output section configured to output, for each of the one or more tasks, a work result that maximizes a posterior probability calculated based on an estimated value of each of the multiple parameters, from among one or more given unique work results as the work result to be obtained.

17. A computer system for estimating an ability of a worker in a process for integrating work results of multiple workers for the same task, the computer system comprising a processor and a memory communicating with the processor, wherein the processor is configured to:

acquire, for each of one or more tasks, a work result of a preceding-stage worker and a work result of a succeeding-stage worker that works based on the work result of the preceding-stage worker; and

estimate multiple parameters of a probability model in which an ability of the succeeding-stage worker conditioned by a quality of the work result of the preceding-stage worker is introduced as a parameter, based on the multiple work results obtained for each of the one or more tasks.

18. A computer-readable program product for estimating an ability of a worker in a process for integrating work results of multiple workers for the same task, the program product causing a computer system to function as:

an acquisition section configured to acquire, for each of one or more tasks, a work result of a preceding-stage worker and a work result of a succeeding-stage worker that works based on the work result of the preceding-stage worker; and

an estimation section configured to estimate multiple parameters of a probability model in which an ability of the succeeding-stage worker conditioned by a quality of the work result of the preceding-stage worker is introduced as a parameter, based on the multiple work results obtained for each of the one or more tasks.

19. A computer-readable program product for integrating work results of multiple workers for the same task, the program product causing a computer system to function as:

an acceptance section configured to present a work result of a preceding-stage worker to one or more succeeding-stage workers and accept a work result from each of the one or more succeeding-stage workers, for a task; and

a results integration section configured to estimate a work result to be obtained for the task as a result of integrating the work results of the multiple workers based on a probability model in which an ability of each succeeding-stage worker conditioned by a quality of the work result of the preceding-stage worker is introduced as a parameter.

20. The program product according to claim 19, wherein the results integration section comprises:

an acquisition section configured to acquire, for each of one or more tasks, a work result of the preceding-stage worker and a work result of the succeeding-stage worker; and

an estimation section configured to estimate multiple parameters of the probability model based on the multiple work results obtained for each of the one or more tasks.