Predicting future performance of multiple workers on crowdsourcing tasks and selecting repeated crowdsourcing workers

• US 8,626,545 B2
• Filed: 10/17/2011
• Issued: 01/07/2014
• Est. Priority Date: 10/17/2011
• Status: Active Grant

• Current Assignee: CrowdFlower, Inc. (Appen Ltd.)
• Original Assignee: CrowdFlower, Inc. (Appen Ltd.)
• Inventors: Van Pelt, Christopher R., Cox, Ryan, Sorokin, Alex, Juster, Michael
• Primary Examiner(s): Fleischer, Mark
• Assistant Examiner(s): Rotaru, Octavian
• Application Number:
  US13/275,284
• Publication Number:
  US 20130096968A1
• Time in Patent Office: 813 Days
• Field of Search: None
• US Class Current: 705/7.13
• CPC Class Codes: G06Q 10/063112 Skill-based matching of a person or a group to a task
• CPC Class Codes: G06Q 10/06398 Performance of employee with respect to a job function

Abstract

Systems and methods of a job distribution platform for aggregating performance data in a worker profile for workers in performing crowd sourced tasks are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, of generating and storing performance data when a worker performs one or more tasks distributed online to their respective computing devices by a job distribution platform which crowd sources tasks over a network to remote workers. The tasks can span current jobs and a history of previous jobs distributed to the worker and the job performance data for the worker is collected for current and previous jobs. New jobs can be assigned to a worker selected based on performance data of the worker.

Claims

1. A method, performed by a task distribution server, for predicting future performance of multiple workers on crowdsourcing tasks and for selecting repeated crowdsourcing workers, the method comprising:

monitoring, by the task distribution server, results on crowdsourcing tasks performed by multiple workers;

generating, by the task distribution server, a test task with known responses;

comparing, by the task distribution server, the monitored results generated by the multiple workers to the test task;

monitoring, by the task distribution server, a speed rate at which the multiple workers complete the crowdsourcing tasks;

computing, by the task distribution server, an average speed rate at with which the multiple workers can satisfactorily complete a task;

comparing, by the task distribution server, the monitored speed rate at which the multiple workers complete the crowdsourcing tasks to the average speed rate at with which the multiple workers can satisfactorily complete a task;

determining, by the task distribution server, work accuracy of each of the multiple workers based on comparing the monitored results generated by the multiple workers to the test task, and based on comparing the monitored speed rate at which the multiple workers complete the crowdsourcing tasks to the average speed rate at with which the multiple workers can satisfactorily complete a task;

selecting, by the task distribution server, repeat workers to perform a new task based on the determined work accuracy of each of the by multiple workers;

temporarily or permanently suspending, by the task distribution server, the repeat workers from performing the new task when their work accuracy falls below a threshold;

re-routing, by the task distribution server, the new task from the repeat workers to other workers;

aggregating, by the task distribution server, the performance data for the multiple workers across several worker pools;

predicting, by the task distribution server, future performance of the multiple workers based on the aggregated performance data.

2. The method of claim 1, wherein, the aggregated job performance data is stored, by the task distribution server, for the multiple workers for future use in selecting the repeat worker to perform the a new task.

3. The method of claim 1, wherein the work accuracy of each of the multiple workers is indicated by quantitative metrics.

4. A task distribution server for predicting future performance of multiple workers on crowdsourcing tasks and for selecting repeated crowdsourcing workers, the task distribution server comprising a memory storing instructions that when executed, by one or more computer processors, causes the task distribution server to:

monitor results on crowdsourcing tasks performed by multiple workers;

generate a test task with known responses;

compare the monitored results generated by the multiple workers to the test task;

monitor, a speed rate at which the multiple workers complete the crowdsourcing tasks;

compute, an average speed rate at with which the multiple workers can satisfactorily complete a task;

compare, the monitored speed rate at which the multiple workers complete the crowdsourcing tasks to the average speed rate at with which the multiple workers can satisfactorily complete a task;

determine, work accuracy of each of the multiple workers based on comparing the monitored results generated by the multiple workers to the test task, and based on comparing the monitored speed rate at which the multiple workers complete the crowdsourcing tasks to the average speed rate at with which the multiple workers can satisfactorily complete a task;

select, repeat workers to perform a new task based on the determined work accuracy of each of the by multiple workers;

temporarily or permanently suspend, the repeat workers from performing the new task when their work accuracy falls below a threshold;

re-route the new task from the repeat workers to other workers;

aggregate the performance data for the multiple workers across several worker pools;

predict, future performance of the multiple workers based on the aggregated performance data.