SYSTEM AND METHOD FOR DETECTING HUMAN JUDGMENT DRIFT AND VARIATION CONTROL

• US 20090240643A1
• Filed: 03/18/2008
• Published: 09/24/2009
• Est. Priority Date: 03/18/2008
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Wai, Lawrence, Bridgewater, Jesse
• Granted Patent: US 8,005,775 B2
• Time in Patent Office: Days
• US Class Current: 706/45
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Abstract

The present invention relates to methods, systems, and computer readable media comprising instructions for rescaling human judgment data for one or more items of content. The method of the present invention comprises generating one or more test sets comprising one or more items of content and generating one or more benchmark sets comprising one or more items of content common to each of the test sets. Judgment data for the one or more items of content comprising the one or more test sets from one or more human editors is received. A variation correction factor and a drift correction factor are identified for each of the one or more human editors. The variation correction factor and drift correction factor associated with each respective human editor are thereafter applied to the one or more items of content comprising the test set for which each human editor provided judgment data.

Claims

1. A method for rescaling human judgment data for one or more items of content, the method comprising:

generating one or more test sets comprising one or more items of content;

generating one or more benchmark sets comprising one or more items of content common to each of the one or more test sets;

receiving judgment data for the one or more items of content comprising the one or more test sets from one or more human editors;

identifying a variation correction factor for each of the one or more human editors on the basis of the judgment data received from the one or more human editors for the one or more items of content comprising the benchmark sets;

identifying a drift correction factor for each of the one or more human editors on the basis of historical judgment data associated with the one or more items of content comprising the benchmark sets; and

applying the variation correction factor and drift correction factor associated with each respective human editor to the one or more items of content comprising the test set for which each human editor provided judgment data.

2. The method of claim 1 wherein generating one or more test sets comprises generating one or more tests comprising one or web pages and one or more queries.

3. The method of claim 2 wherein receiving judgment data for an item of content comprises receiving judgment data indicating a degree to which a given web page is relevant with respect to a given query.

4. The method of claim 1 wherein identifying a variation correction factor comprises identifying a degree to which judgment data provided by a given human editor for a given item of content from a benchmark set differs from the judgment data provided by one or more other human editors for the item of content from the benchmark set.

5. The method of claim 1 wherein identifying a drift correction factor comprises identifying a degree to which judgment data provided by a given human editor for a given item of content from a benchmark set differs from prior judgment data provided by one or more human editors for the item of content from the benchmark set.

6. The method of claim 1 wherein applying the variation correction factor and drift correction factor comprises:

applying the variation correction factor and drift correction factor associated with a given human editor to the one or more items of content comprising the test set for which the human editor provided judgment data; and

rescaling the judgment data through use of Monte Carlo techniques.

7. A system for rescaling human judgment data for one or more items of content, the system comprising:

a benchmark component operative to;

generate one or more test sets comprising one or more items of content; and

generate one or more benchmark sets comprising one or more items of content common to each of the one or more test sets;

a human editor interface operative to receive judgment data for the one or more items of content comprising the one or more test sets from one or more human editors;

a variation component operative to identify a variation correction factor for each of the one or more human editors on the basis of the judgment data received from the one or more human editors via the human editor interface for the one or more items of content comprising the benchmark sets;

a drift component operative to identify a drift correction factor for each of the one or more human editors on the basis of historical judgment data associated with the one or more items of content comprising the benchmark sets; and

a correction factor component operative to apply the variation correction factor and drift correction factor associated with each respective human editor to the one or more items of content comprising the test set for which each human editor provided judgment data.

8. The system of claim 7 wherein the benchmark component is operative to generate one or more test sets comprising one or more web pages and one or more queries.

9. The system of claim 8 wherein the human editor interface is operative to receive judgment data from one or more human editors indicating a degree to which a given web page is relevant with respect to a given query.

10. The system of claim 7 wherein the variation component is operative to identify a degree to which judgment data provided by a given human editor via the human editor interface for a given item of content from a benchmark set differs from the judgment data provided by one or more other human editors for the item of content from the benchmark set.

11. The system of claim 7 wherein the drift component is operative to identify a degree to which judgment data provided by a given human editor via the human editor interface for a given item of content from a benchmark set differs from prior judgment data provided by one or more human editors for the item of content from the benchmark set.

12. The system of claim 7 wherein the correction factor component is operative to:

apply the variation correction factor and drift correction factors associated with a given human editor to the one or more items of content comprising the test set for which the human editor provided judgment data; and

rescale the judgment data through use of Monte Carlo techniques.

13. A computer readable media comprising program code that when executed instructs a processor to perform a method for resealing human judgment data for one or more items of content, the method comprising:

instructions for generating one or more test sets comprising one or more items of content;

instructions for generating one or more benchmark sets comprising one or more items of content common to each of the one or more test sets;

instructions for receiving judgment data for the one or more items of content comprising the one or more test sets from one or more human editors;

instructions for identifying a variation correction factor for each of the one or more human editors on the basis of the judgment data received from the one or more human editors for the one or more items of content comprising the benchmark sets;

instructions for identifying a drift correction factor for each of the one or more human editors on the basis of historical judgment data associated with the one or more items of content comprising the benchmark sets; and

instructions for applying the variation correction factor and drift correction factor associated with each respective human editor to the one or more items of content comprising the test set for which each human editor provided judgment data.

14. The computer readable media of claim 13 wherein the instructions for generating one or more test sets comprises instructions for generating one or more tests comprising one or web pages and one or more queries.

15. The computer readable media of claim 14 wherein the instructions for receiving judgment data for an item of content comprises instructions for receiving judgment data indicating a degree to which a given web page is relevant with respect to a given query.

16. The computer readable media of claim 13 wherein the instructions for identifying a variation correction factor comprises instructions for identifying a degree to which judgment data provided by a given human editor for a given item of content from a benchmark set differs from the judgment data provided by one or more other human editors for the item of content from the benchmark set.

17. The computer readable media of claim 13 wherein the instructions for identifying a drift correction factor comprises instructions for identifying a degree to which judgment data provided by a given human editor for a given item of content from a benchmark set differs from prior judgment data provided by one or more human editors for the item of content from the benchmark set.

18. The computer readable media of claim 13 wherein the instructions for applying the variation correction factor and drift correction factor comprises:

instructions for applying the variation correction factor and drift correction factor associated with a given human editor to the one or more items of content comprising the test set for which the human editor provided judgment data; and

instructions for rescaling the judgment data through use of Monte Carlo techniques.