Optimized Judge Assignment under Constraints
First Claim
1. In a computing environment, a method performed at least in part on at least one processor comprising, providing probing tasks to judges to obtain accuracy-related data with respect to labeling tasks, and using the accuracy-related data to compute an assignment model for assigning other labeling tasks to at least some of the judges, in which the assignment model distributes the other labeling tasks based at least in part upon the accuracy-related data.
2 Assignments
0 Petitions
Accused Products
Abstract
Described is a technology by which an assignment model is computed to distribute labeling tasks among judging entities (judges). The assignment model is optimized by obtaining accuracy-related data of the judges, e.g., by probing the judges with labeling tasks having a gold standard label and evaluating the judges'"'"' labels against the gold standard labels, and optimizing for accuracy. Optimization may be based upon on or more other constraints, such as per-judge cost and/or quota.
-
Citations
20 Claims
- 1. In a computing environment, a method performed at least in part on at least one processor comprising, providing probing tasks to judges to obtain accuracy-related data with respect to labeling tasks, and using the accuracy-related data to compute an assignment model for assigning other labeling tasks to at least some of the judges, in which the assignment model distributes the other labeling tasks based at least in part upon the accuracy-related data.
- 9. A system comprising, an assignment engine configured to provide regular tasks and probing tasks to a plurality of judges for labeling, and an optimization mechanism configured to evaluate probed labels corresponding to the probing tasks against gold standard labels to obtain accuracy-related data and optimize an assignment model based at least in part upon the accuracy-related data, in which the assignment model is useable to distribute other labeling tasks based at least in part upon the accuracy-related data to at least some of the judges.
-
17. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:
-
receiving a task set comprising labeling tasks for assigning to judges; and distributing the labeling tasks from the task set according to an assignment model, in which the assignment model is optimized at least in part according to per-judge accuracy-related data obtained by probing of the judges. - View Dependent Claims (18, 19, 20)
-
Specification