Result aggregation
First Claim
1. A method for result aggregation in a crowdsourcing environment, the method comprising:
- providing a task with a plurality of solution options and a range of confidence values corresponding to the plurality of solution options to multiple workers;
receiving responses for the task from the multiple workers, wherein each of the responses comprises at least a worker'"'"'s selection of one of the plurality of solution options and a worker response confidence value selected from the range of confidence values corresponding to the selected one of the plurality of solution options, and wherein the worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct;
estimating a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and
computing a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.
8 Assignments
0 Petitions
Accused Products
Abstract
Methods and systems for result aggregation in crowdsourcing are described. In one implementation, a task with a plurality of solution options is provided for workers to respond. Responses for the task are received from multiple workers, where each of the responses includes a worker response confidence value corresponding to one of the plurality of solution options. The worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct. A task answer for the task is estimated based on at least the worker response confidence value received from each of the multiple workers. A performance bonus for each of the multiple workers is computed based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.
8 Citations
20 Claims
-
1. A method for result aggregation in a crowdsourcing environment, the method comprising:
-
providing a task with a plurality of solution options and a range of confidence values corresponding to the plurality of solution options to multiple workers; receiving responses for the task from the multiple workers, wherein each of the responses comprises at least a worker'"'"'s selection of one of the plurality of solution options and a worker response confidence value selected from the range of confidence values corresponding to the selected one of the plurality of solution options, and wherein the worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct; estimating a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and computing a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A result aggregation system comprising:
-
a processor; and a memory storing instructions that when executed by the processor cause the processor to; provide a task with a plurality of solution options to multiple workers; receive responses for the task from the multiple workers, wherein each of the responses comprises at least one worker response confidence value corresponding to at least one of the plurality of solution options, wherein the worker response confidence value is indicative of confidence of a worker for the at least one of the plurality of solution options to be correct and the worker response confidence value is selected from a range of confidence values; estimate a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and compute a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule. - View Dependent Claims (9, 10, 11, 12, 20)
-
-
13. A non-transitory computer-readable medium comprising instructions for result aggregation, wherein the instructions are executable by a processor to cause the processor to:
-
provide a task with a plurality of solution options to multiple workers via a Graphical User Interface (GUI) that includes an input mechanism that receives worker response confidence values corresponding to the plurality of solution options; receive responses for the task from the multiple workers, wherein each of the responses comprises an input for the worker response confidence value corresponding to a selected one of the plurality of solution options, and wherein the worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct; estimate a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and compute a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule. - View Dependent Claims (14, 15, 16, 17, 18, 19)
-
Specification