EVALUATING A WORKER IN PERFORMING CROWD SOURCED TASKS AND PROVIDING IN-TASK TRAINING THROUGH PROGRAMMATICALLY GENERATED TEST TASKS
First Claim
1. A method of worker quality control in crowd sourcing tasks, the method, comprising:
- selecting a job distributed through a job distribution platform for workers to work on to generate a test task, the job being associated with a known correct result;
associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task;
presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker;
wherein, the job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.
3 Assignments
0 Petitions
Accused Products
Abstract
Systems and methods for evaluating a worker in performing crowd sourced tasks and providing in-task training are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, for selecting a job distributed through a job distribution platform for workers to work on, for use to generate a test task, the job being associated with a known correct result, associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task, and/or presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker. The job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.
36 Citations
29 Claims
-
1. A method of worker quality control in crowd sourcing tasks, the method, comprising:
-
selecting a job distributed through a job distribution platform for workers to work on to generate a test task, the job being associated with a known correct result; associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task; presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker; wherein, the job distribution platform crowd sources tasks online to workers to work on via their respective computing devices. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A method of worker quality control in crowd sourcing tasks, the method, comprising:
-
using a job distributed through a job distribution platform for workers to work on to generate a test task; presenting the test task to a worker for evaluation of work quality of the worker based on a result provided by the worker in response to the test task; wherein, the job distribution platform crowd sources tasks online to workers to work on via their respective computing devices. - View Dependent Claims (9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A machine-readable storage medium having stored thereon a set of instructions which when executed causes a processor to perform a method of providing in-task training of a worker in a crowd sourced environment, the method, comprising:
-
presenting a job with a manipulated result as a test task to a worker for evaluation of work quality of the worker, the manipulated result being an inaccurate result; evaluating the work quality of the worker by determining whether the worker has identified the manipulated result as being an inaccurate result for the job; automatically generating a feedback message from the incorrect result; presenting the feedback message to the worker via a respective computing device to provide in-task training of the worker. - View Dependent Claims (20, 21)
-
-
22. A system for training and evaluating a worker in performing crowd sourcing tasks, the system, comprising:
-
means for, detecting a new worker working on jobs distributed through a crowd sourcing service; means for, initiating a training session for the new worker; means for, presenting a test task to the worker for evaluation of work quality of the worker based on a result provided by the worker in response to the test task, after the training session has been completed; means for, computing performance statistics based on the worker'"'"'s performance with respect to one or more test tasks. - View Dependent Claims (23, 24, 25, 26)
-
-
27. A system for worker quality control in crowd sourcing tasks, the system, comprising:
-
means for, using a job distributed by a job distribution service for workers to work on to generate a test task; means for, presenting the test task to a worker for evaluation of work quality of the worker based on a result provided by the worker in response to the test task; means for, computing performance statistics based on the worker'"'"'s performance with respect to one or more test tasks. - View Dependent Claims (28, 29)
-
Specification