System and method for detecting human judgment drift and variation control
First Claim
1. A method for rescaling human judgment data for one or more items of content, the method comprising:
- generating one or more test sets comprising one or more items of content the one or more items of content including one or more web pages and one or more queries;
generating one or more benchmark sets comprising one or more items of content common to each of the one or more test sets;
receiving judgment data for the one or more items of content comprising the one or more test sets from one or more human editors;
identifying a variation correction factor for each of the one or more human editors on the basis of the judgment data received from the one or more human editors for the one or more items of content comprising the benchmark sets;
identifying a drift correction factor for each of the one or more human editors on the basis of historical judgment data associated with the one or more items of content comprising the benchmark sets; and
applying the variation correction factor and drift correction factor associated with each respective human editor to the one or more items of content comprising the one or more test sets for which each human editor provided judgment data.
9 Assignments
0 Petitions
Accused Products
Abstract
The present invention relates to methods, systems, and computer readable media comprising instructions for rescaling human judgment data for one or more items of content. The method of the present invention comprises generating one or more test sets comprising one or more items of content and generating one or more benchmark sets comprising one or more items of content common to each of the test sets. Judgment data for the one or more items of content comprising the one or more test sets from one or more human editors is received. A variation correction factor and a drift correction factor are identified for each of the one or more human editors. The variation correction factor and drift correction factor associated with each respective human editor are thereafter applied to the one or more items of content comprising the test set for which each human editor provided judgment data.
-
Citations
15 Claims
-
1. A method for rescaling human judgment data for one or more items of content, the method comprising:
-
generating one or more test sets comprising one or more items of content the one or more items of content including one or more web pages and one or more queries; generating one or more benchmark sets comprising one or more items of content common to each of the one or more test sets; receiving judgment data for the one or more items of content comprising the one or more test sets from one or more human editors; identifying a variation correction factor for each of the one or more human editors on the basis of the judgment data received from the one or more human editors for the one or more items of content comprising the benchmark sets; identifying a drift correction factor for each of the one or more human editors on the basis of historical judgment data associated with the one or more items of content comprising the benchmark sets; and applying the variation correction factor and drift correction factor associated with each respective human editor to the one or more items of content comprising the one or more test sets for which each human editor provided judgment data. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A system for rescaling human judgment data for one or more items of content, the system comprising:
-
a benchmark component operative to; generate one or more test sets comprising one or more items of content, the one or more items of content including one or more web pages and one or more queries; and generate one or more benchmark sets comprising one or more items of content common to each of the one or more test sets; a human editor interface operative to receive judgment data for the one or more items of content comprising the one or more test sets from one or more human editors; a variation component operative to identify a variation correction factor for each of the one or more human editors on the basis of the judgment data received from the one or more human editors via the human editor interface for the one or more items of content comprising the benchmark sets; a drift component operative to identify a drift correction factor for each of the one or more human editors on the basis of historical judgment data associated with the one or more items of content comprising the benchmark sets; and a correction factor component operative to apply the variation correction factor and drift correction factor associated with each respective human editor to the one or more items of content comprising the one or more test sets for which each human editor provided judgment data. - View Dependent Claims (7, 8, 9, 10)
-
-
11. A computer readable media comprising program code that when executed instructs a processor to perform a method for rescaling human judgment data for one or more items of content, the method comprising:
-
instructions for generating one or more test sets comprising one or more items of content, the one or more items of content including one or more web pages and one or more queries; instructions for generating one or more benchmark sets comprising one or more items of content common to each of the one or more test sets; instructions for receiving judgment data for the one or more items of content comprising the one or more test sets from one or more human editors; instructions for identifying a variation correction factor for each of the one or more human editors on the basis of the judgment data received from the one or more human editors for the one or more items of content comprising the benchmark sets; instructions for identifying a drift correction factor for each of the one or more human editors on the basis of historical judgment data associated with the one or more items of content comprising the benchmark sets; and instructions for applying the variation correction factor and drift correction factor associated with each respective human editor to the one or more items of content comprising the one or more test sets for which each human editor provided judgment data. - View Dependent Claims (12, 13, 14, 15)
-
Specification