System and method of information retrieval engine evaluation using human judgment input

• US 20080027913A1
• Filed: 07/25/2006
• Published: 01/31/2008
• Est. Priority Date: 07/25/2006
• Status: Abandoned Application

• Current Assignee: Oath Inc. (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Chen, Jian, Chang, Chi-Chao
• Application Number:
  US11/493,268
• Publication Number:
  US 20080027913A1
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06Q 10/00 Administration; Management

Abstract

An information retrieval engine evaluation system and method is disclosed, which uses judgment input, or feedback, received from one or more individuals, or judges. Judgment input is provided by the one or judges, each of whom review at least one aspect of performance of a software application, and provide judgment input in the form of responses to questions. The judgment input is received and analyzed, and can be used to generate the one or more metrics. The metrics can be examined to evaluate at least one indicator in order to determine performance of the software application.

Claims

1. An information retrieval engine evaluation method, comprising:

identifying a query benchmark comprising a plurality of queries, the queries having corresponding search results;

obtaining judgment input from one or more judges, the judgment input corresponding to the set of search results;

determining, using the judgment input obtained from the one or more judges, at least one metric corresponding to an indicator of performance, a first value of the at least one metric corresponding to a first information retrieval engine and a second value of the at least one metric corresponding to a second information retrieval engine; and

comparing the first and second values of the at least one metric to evaluate the performance indicator so as to evaluate performance of the first information retrieval engine relative to the second information retrieval engine based on the query benchmark.

2. The method of claim 1, wherein the first and information retrieval engines comprise search engine instances and the performance indicator comprises an indicator of relevance of search results generated by the search engine instances.

3. The method of claim 1, further comprising

determining at least one other metric corresponding to at least one other indicator of performance, a first value of the at least one other metric corresponding to the first information retrieval engine and a second value of the at least one other metric corresponding to the second information retrieval engine;

and

comparing the first and second values of the at least one other metric to evaluate the at least one other performance indicator so as to evaluate performance of the first information retrieval engine relative to the second information retrieval engine.

4. The method of claim 3, wherein the first and second information retrieval engines comprise search engine instances and the at least one other performance indicator comprises an indicator of differences between search results generated by the search engine instances.

5. The method of claim 3, wherein the first and second information retrieval engine comprise search engine instances and the at least one other performance indicator comprises an indicator of coverage of search results generated by the search engine instances.

6. The method of claim 1, wherein the first and second information retrieval engines comprise different configurations of a search engine.

7. The method of claim 1, wherein the first and second information retrieval engine comprise different search engines.

8. The method of claim 1, further comprising:

examining the judgment input to identify inconsistencies in the judgment input.

9. The method of claim 8, wherein examining the judgment input further comprises:

examining the judgment input to identify an inconsistency in a given judge'"'"'s judgment input.

10. The method of claim 8, wherein examining the judgment input further comprises:

examining the judgment input to identify an inconsistency in one judge'"'"'s judgment input relative to at least one other judge'"'"'s judgment input.

11. A method of measuring search engine performance using a set of stored queries, results and judgments, the method comprising:

generating a query benchmark from the set of stored queries, the query benchmark includes one or more queries;

obtaining query results using the one or more benchmark queries;

retrieving one or more stored results associated with the plurality of queries;

retrieving a judgment associated with at least one stored result;

predicting a judgment associated with one or more of the obtained results based on the at least one stored result; and

determining a performance measure using the retrieved and predicted judgments.

12. The method of claim 11, wherein a judgment associated with a stored result identifies relevancy of the stored result to the query.

13. The method of claim 11, further comprising:

storing each of the plurality of queries in a query log;

for each query stored in the query log;

performing the query to obtain a set of results, the set of results including at least one result item; and

obtaining a judgment corresponding to the at least one result item.

14. The method of claim 13, further comprising:

examining an obtained judgment to determine whether an inconsistency exists in the obtained judgment.

15. The method of claim 14, wherein a judgment is based on user input, and wherein examining an obtained judgment to determine whether an inconsistency exists in the obtained judgment further comprises:

comparing a user'"'"'s judgment input associated with a result item to predetermined judgment input for the associated result item.

16. An information retrieval engine evaluation system, comprising:

program memory for storing process steps executable to;

identify a query benchmark comprising a plurality of queries, the queries having corresponding search results, obtain judgment input from one or more judges, the judgment input corresponding to the set of search results determine, using the judgment input obtained from the one or more judges, at least one metric corresponding to an indicator of performance, a first value of the at least one metric corresponding to a first information retrieval engine and a second value of the at least one metric corresponding to a second information retrieval engine; and

compare the first and second values of the at least one metric to evaluate the performance indicator so as to evaluate performance of the first information retrieval engine relative to the second information retrieval engine; and

at least one processor for executing the process steps stored in said program memory.

17. A system for measuring search engine performance using a set of stored queries, results and judgments, the system comprising:

program memory for storing process steps executable to;

generate a query benchmark from the set of stored queries, the query benchmark includes one or more queries;

obtain query results using the one or more benchmark queries;

retrieve one or more stored results associated with the plurality of queries;

retrieve a judgment associated with at least one stored result;

predict a judgment associated with one or more of the obtained results based on the at least one stored result; and

determine a performance measure using the retrieved and predicted judgments; and

at least one processor for executing the process steps stored in said program memory.