EVALUATING A WORKER IN PERFORMING CROWD SOURCED TASKS AND PROVIDING IN-TASK TRAINING THROUGH PROGRAMMATICALLY GENERATED TEST TASKS

• US 20140039985A1
• Filed: 10/08/2013
• Published: 02/06/2014
• Est. Priority Date: 06/29/2011
• Status: Abandoned Application

• Current Assignee: Figure Eight Technologies, Inc. (Appen Ltd.)
• Original Assignee: CrowdFlower, Inc. (Appen Ltd.)
• Inventors: Oleson, David, Sorokin, Alexander, Laughlin, Greg, Hester, Vaughn, Le, John, Van Pelt, Christopher R., Biewald, Lukas A.
• Application Number:
  US14/048,342
• Publication Number:
  US 20140039985A1
• Time in Patent Office: Days
• US Class Current: 705/7.41
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 10/06395 Quality analysis or management
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

Systems and methods for evaluating a worker in performing crowd sourced tasks and providing in-task training are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, for selecting a job distributed through a job distribution platform for workers to work on, for use to generate a test task, the job being associated with a known correct result, associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task, and/or presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker. The job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.

Claims

1. A method of worker quality control in crowd sourcing tasks, the method, comprising:

selecting a job distributed through a job distribution platform for workers to work on to generate a test task, the job being associated with a known correct result;

associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task;

presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker;

wherein, the job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.

2. The method of claim 1, wherein, the test task is presented among other jobs or tasks such that the worker is unaware of which of the jobs or tasks are being used to evaluate the work quality.

3. The method of claim 1, further comprising, evaluating the work quality of the worker by determining whether the worker has identified the manipulated result as being an inaccurate result for the job.

4. The method of claim 1, further comprising, generating a feedback message from the incorrect result.

5. The method of claim 4, wherein, the feedback message is presented to the worker evaluating the test task if the worker fails to identify the manipulated result as being inaccurate for the job.

6. The method of claim 1, wherein, the manipulated result is determined from frequently received worker errors in response to working on the job.

7. The method of claim 1, wherein, the known result is determined from agreement among results provided from different workers.

8. A method of worker quality control in crowd sourcing tasks, the method, comprising:

using a job distributed through a job distribution platform for workers to work on to generate a test task;

presenting the test task to a worker for evaluation of work quality of the worker based on a result provided by the worker in response to the test task;

wherein, the job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.

9. The method of claim 8, further comprising, computing performance statistics based on the worker'"'"'s performance with respect to one or more test tasks;

wherein, the performance statistics are used in generating a worker profile for the worker including an aggregate indication of job performance and competency.

10. The method of claim 9, wherein, the performance statistics are used to determine compensation for the worker.

11. The method of claim 8, wherein, the job used to generate the test task is selected from jobs with known results.

12. The method of claim 8, wherein, the job used to generate the test task is selected from jobs with results known to meet or exceed a threshold level of confidence.

13. The method of claim 12, wherein, the threshold level of confidence is represented by a number of matching results provided by workers.

14. The method of claim 12, wherein, the test task is generated from the job by associating a manipulated result, known to be an incorrect result for the job, with the job.

15. The method of claim 14, wherein, the manipulated result is selected from actual worker errors.

16. The method of claim 8, wherein, the job includes verification of a business listing.

17. The method of claim 8, wherein, the job includes verification of a URL.

18. The method of claim 8, wherein, the job includes verification of a search result generated from a search engine.

19. A machine-readable storage medium having stored thereon a set of instructions which when executed causes a processor to perform a method of providing in-task training of a worker in a crowd sourced environment, the method, comprising:

presenting a job with a manipulated result as a test task to a worker for evaluation of work quality of the worker, the manipulated result being an inaccurate result;

evaluating the work quality of the worker by determining whether the worker has identified the manipulated result as being an inaccurate result for the job;

automatically generating a feedback message from the incorrect result;

presenting the feedback message to the worker via a respective computing device to provide in-task training of the worker.

20. The method of claim 19, wherein, the presentation of the test task is disguised among actual jobs that are not test tasks.

21. The method of claim 19, wherein, the worker is presented with the test task after participating in a training session.

22. A system for training and evaluating a worker in performing crowd sourcing tasks, the system, comprising:

means for, detecting a new worker working on jobs distributed through a crowd sourcing service;

means for, initiating a training session for the new worker;

means for, presenting a test task to the worker for evaluation of work quality of the worker based on a result provided by the worker in response to the test task, after the training session has been completed;

means for, computing performance statistics based on the worker'"'"'s performance with respect to one or more test tasks.

23. The system of claim 22, further comprising, means for, using a job distributed by the crowd sourcing service for workers to work on to generate a test task.

24. The system of claim 22, further comprising, means for, using the performance statistics to determine whether to use the new worker for jobs distributed through the crowd sourcing service.

25. The system of claim 22, further comprising, means for, using the performance statistics to determine whether the new worker should be blocked from performing work on jobs distributed through the crowd sourcing service.

26. The system of claim 22, further comprising, means for, using the performance statistics to determine whether a wage for the new worker.

27. A system for worker quality control in crowd sourcing tasks, the system, comprising:

means for, using a job distributed by a job distribution service for workers to work on to generate a test task;

means for, presenting the test task to a worker for evaluation of work quality of the worker based on a result provided by the worker in response to the test task;

means for, computing performance statistics based on the worker'"'"'s performance with respect to one or more test tasks.

28. The system of claim 27, wherein, the job distribution service crowd sources tasks online to workers to work on via their respective computing devices.

29. The system of claim 27, further comprising:

means for, aggregating performance statistics for the worker over a period of time;

means for, computing performance statistics for multiple workers that receive jobs through the job distribution service.