EVALUATING A WORKER IN PERFORMING CROWD SOURCED TASKS AND PROVIDING IN-TASK TRAINING THROUGH PROGRAMMATICALLY GENERATED TEST TASKS
First Claim
1. A method of worker quality control in crowd sourcing tasks, the method, comprising:
- selecting a job distributed through a job distribution platform for workers to work on to generate a test task, the job being associated with a known correct result;
associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task;
presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker;
wherein, the job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.
1 Assignment
0 Petitions
Accused Products
Abstract
Systems and methods for evaluating a worker in performing crowd sourced tasks and providing in-task training are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, for selecting a job distributed through a job distribution platform for workers to work on, for use to generate a test task, the job being associated with a known correct result, associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task, and/or presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker. The job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.
-
Citations
29 Claims
-
1. A method of worker quality control in crowd sourcing tasks, the method, comprising:
-
selecting a job distributed through a job distribution platform for workers to work on to generate a test task, the job being associated with a known correct result; associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task; presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker; wherein, the job distribution platform crowd sources tasks online to workers to work on via their respective computing devices. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A method of worker quality control in crowd sourcing tasks, the method, comprising:
-
using a job distributed through a job distribution platform for workers to work on to generate a test task; presenting the test task to a worker for evaluation of work quality of the worker based on a result provided by the worker in response to the test task; wherein, the job distribution platform crowd sources tasks online to workers to work on via their respective computing devices. - View Dependent Claims (9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A machine-readable storage medium having stored thereon a set of instructions which when executed causes a processor to perform a method of providing in-task training of a worker in a crowd sourced environment, the method, comprising:
-
presenting a job with a manipulated result as a test task to a worker for evaluation of work quality of the worker, the manipulated result being an inaccurate result; evaluating the work quality of the worker by determining whether the worker has identified the manipulated result as being an inaccurate result for the job; automatically generating a feedback message from the incorrect result; presenting the feedback message to the worker via a respective computing device to provide in-task training of the worker. - View Dependent Claims (20, 21)
-
-
22. A system for training and evaluating a worker in performing crowd sourcing tasks, the system, comprising:
-
means for, detecting a new worker working on jobs distributed through a crowd sourcing service; means for, initiating a training session for the new worker; means for, presenting a test task to the worker for evaluation of work quality of the worker based on a result provided by the worker in response to the test task, after the training session has been completed; means for, computing performance statistics based on the worker'"'"'s performance with respect to one or more test tasks. - View Dependent Claims (23, 24, 25, 26)
-
-
27. A system for worker quality control in crowd sourcing tasks, the system, comprising:
-
means for, using a job distributed by a job distribution service for workers to work on to generate a test task; means for, presenting the test task to a worker for evaluation of work quality of the worker based on a result provided by the worker in response to the test task; means for, computing performance statistics based on the worker'"'"'s performance with respect to one or more test tasks. - View Dependent Claims (28, 29)
-
Specification