Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks

• US 8,554,605 B2
• Filed: 10/17/2011
• Issued: 10/08/2013
• Est. Priority Date: 06/29/2011
• Status: Active Grant

• Current Assignee: CrowdFlower, Inc. (Appen Ltd.)
• Original Assignee: CrowdFlower, Inc. (Appen Ltd.)
• Inventors: Oleson, David, Sorokin, Alexander, Laughlin, Greg, Hester, Vaughn, Le, John, Van Pelt, Christopher R., Biewald, Lukas A.
• Primary Examiner(s): Fleischer, Mark A
• Assistant Examiner(s): Rotaru, Octavian
• Application Number:
  US13/275,290
• Publication Number:
  US 20130006717A1
• Time in Patent Office: 722 Days
• Field of Search: None
• US Class Current: 705/7.41
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 10/06395 Quality analysis or management
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

Systems and methods for evaluating a worker in performing crowd sourced tasks and providing in-task training are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, for selecting a job distributed through a job distribution platform for workers to work on, for use to generate a test task, the job being associated with a known correct result, associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task, and/or presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker. The job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.

Claims

1. A method for testing and controlling work quality in crowdsourcing jobs, the method comprising:

selecting a crowdsourcing job, distributed through a processor based job distribution platform, with a known correct result determined by agreement of different workers;

generating a test crowdsourcing job by manipulating the known correct result into an incorrect result based on frequently received worker errors;

presenting the test crowdsourcing job for evaluation to a first worker who is unaware of being evaluated;

evaluating, by the processor based job distribution platform, work quality of the first worker by determining whether the first worker has identified the test crowdsourcing job as having the inaccurate result that was manipulated from the known correct result;

controlling, by the processor based job distribution platform, work quality of the first worker, by minimizing repeat views of test crowdsourcing jobs, by the first worker, to a value equal to a ratio of maximum allowed job completeness for the first worker and test adjusted tasks.

2. The method of claim 1, further providing a feedback mechanism to allow the first worker to contest a wrong result when working on the test crowdsourcing job.

3. The method of claim 1, further adjusting a frequency with which the test crowdsourcing jobs are presented to the first worker among other crowdsourcing jobs based on the work quality of the first worker.

4. The method of claim 1, further generating a feedback message.

5. The method of claim 4, wherein, the feedback message is presented to the first worker when the first worker fails to identify the manipulated result as being inaccurate for the test crowdsourcing job.

6. The method of claim 1, further comprising:

determining the known correct result from similarities among results provided from the different workers.

7. The method of claim 1, wherein, the test crowdsourcing job includes verification of a URL or verification of a search result generated from a search engine.

8. A system for testing and controlling work quality in crowdsourcing jobs, the system comprising a memory storing a set of computer instructions, which when executed by one or more processors causes the system to:

select a crowdsourcing job, with a known correct result determined by agreement of different workers;

generate a test crowdsourcing job by manipulating the known correct result into an incorrect result based on frequently received worker errors;

present the test crowdsourcing job for evaluation to a first worker who is unaware of being evaluated;

evaluate, work quality of the first worker by determining whether the first worker has identified the test crowdsourcing job as having the inaccurate result that was manipulated from the known correct result;

control, work quality of the first worker, by minimizing repeat views of test crowdsourcing jobs, by the first worker, to a value equal to a ratio of maximum allowed job completeness for the first worker and test adjusted tasks.

9. The system of claim 8, further comprising a contested answer manager, which when executed by the one or more processors, allows the first worker to contest a wrong result when working on the test crowdsourcing job.

10. The system of claim 8, further comprising a test data generator which, when executed by the one or more processors, adjusts a frequency with which test crowdsourcing jobs are presented to the first worker among other crowdsourcing jobs based on the work quality of the first worker.

11. The system of claim 8, further comprising a feedback generator, which when executed by the one or more processor, generates a feedback message.

12. The system of claim 11, wherein, the feedback generator presents the feedback message to the first worker when the first worker fails to identify the manipulated result as being inaccurate for the test crowdsourcing job.

13. The system of claim 8, further comprising a data optimization module, which when executed by the one or more processors, determines the known correct result from similarities among results provided from the different workers.

14. The system of claim 8, further comprising an error detector which, when executed by the one or more processors, verifies a URL or a search result generated from a search engine that were provided by the first worker in response to the test crowdsourcing job.