System and method for generating normalized relevance measure for analysis of search results

• US 20060004891A1
• Filed: 06/30/2004
• Published: 01/05/2006
• Est. Priority Date: 06/30/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hurst-Hiller, Oliver, Brewer, Brett D., Chavet, Laurent
• Granted Patent: US 7,725,463 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3325 Reformulation based on results of preceding query
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/3346 using probabilistic model
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A system and related techniques permit a search service operator to access a variety of disparate relevance measures, and integrate those measures into idealized or unified data sets. A search service operator may employ self-learning networks to generate relevance rankings of Web site hits in response to user queries or searches, such as Boolean text or other searches. To improve the accuracy and quality of the rankings of results, the service provider may accept as inputs relevance measures created from query logs, from human-annotated search records, from independent commercial or other search sites, or from other sources and feed those measures to a normalization engine. That engine may normalize those relevance ratings to a common scale, such as quintiles, percentages or other scales or levels. The provider may then use that idealized or normalized combined measure to, for example, train the search algorithms or heuristics to arrive at better or more accurate results.

Claims

1. A system for generating a normalized relevance rating for use in search result analysis, comprising:

an input interface to receive a set of independently scaled relevance metrics; and

a normalization engine, the normalization engine communicating with the input interface to receive the set of independently scaled relevance metrics to generate a normalize relevance output.

2. A system according to claim 1, wherein the set of independently scaled relevance metrics comprises at least one of relevance data derived from query logs, human-annotated relevance data and relevance ratings derived from an independent search service.

3. A system according to claim 2, wherein the human-annotated relevance data comprises at least one of human live site feedback, human search analyst relevance data, human explicit feedback data from a search service, and user group feedback data.

4. A system according to claim 2, wherein the independently scaled relevance metrics comprise at least relevance data derived from query logs, and the relevance data derived from query logs comprises relevance data derived from user selection data.

5. A system according to claim 4, wherein the relevance data derived from user selection data comprises user click-through data selecting from a search results list.

6. A system according to claim 4, wherein the relevance data derived from user selection data comprises variance data indicating variance of user selection from an ordinal relevance ranking.

7. A system according to claim 1, wherein the normalization engine scales the set of independently scaled relevance metrics to generate the normalized relevance output to a common scale.

8. A system according to claim 7, wherein the scaling of the set of independently scaled relevance metrics comprises adjusting the contribution of each of the independently scaled relevance metrics to the normalized relevance output based on at least one of relevance metric type, a confidence measure associated with each of the relevance metrics, and a frequency of each of the relevance metrics.

9. A system according to claim 7, wherein the common scale comprises a set of qualitatively labeled relevance levels.

10. A system according to claim 1, wherein the scaling of the set of independently scaled relevance metrics comprises at least one of applying a scaling weight and applying a scaling function.

11. A method for generating a normalized relevance rating for use in search result analysis, comprising:

receiving a set of independently scaled relevance metrics; and

generating a normalized relevance output from the set of independently scaled relevance metrics.

12. A method according to claim 11, wherein the set of independently scaled relevance metrics comprises at least one of relevance data derived from query logs, human-annotated relevance data and relevance ratings derived from an independent search service.

13. A method according to claim 12, wherein the human-annotated relevance data comprises at least one of human live site feedback, human search analyst relevance data, human explicit feedback data from a search service, and user group feedback data.

14. A method according to claim 12, wherein the independently scaled relevance metrics comprise at least relevance data derived from query logs, and the relevance data derived from query logs comprises relevance data derived from user selection data.

15. A method according to claim 14, wherein the relevance data derived from user selection data comprises user click-through data selecting from a search results list.

16. A method according to claim 14, wherein the relevance data derived from user selection data comprises variance data indicating variance of user selection from an ordinal relevance ranking.

17. A method according to claim 11, wherein the generating comprises scaling the set of independently scaled relevance metrics to generate the normalized relevance output to a common scale.

18. A method according to claim 17, wherein the scaling of the set of independently scaled relevance metrics comprises adjusting the contribution of each of the independently scaled relevance metrics to the normalized relevance output based on at least one of relevance metric type, a confidence measure associated with each of the relevance metrics, and a frequency of each of the relevance metrics.

19. A method according to claim 17, wherein the common scale comprises a set of qualitatively labeled relevance levels.

20. A method according to claim 11, wherein the scaling of the set of independently scaled relevance metrics comprises at least one of applying a scaling weight and applying a scaling function.

21. A normalized relevance rating for use in search result analysis, the normalized relevance rating being generated according to a method comprising:

receiving a set of independently scaled relevance metrics; and

generating a normalized relevance output from the set of independently scaled relevance metrics.

22. A normalized relevance rating according to claim 21, wherein the set of independently scaled relevance metrics comprises at least one of relevance data derived from query logs, human-annotated relevance data and relevance ratings derived from an independent search service.

23. A normalized relevance rating according to claim 22, wherein the human-annotated relevance data comprises at least one of human live site feedback, human search analyst relevance data, human explicit feedback data from a search service, and user group feedback data.

24. A normalized relevance rating according to claim 22, wherein the independently scaled relevance metrics comprise at least relevance data derived from query logs, and the relevance data derived from query logs comprises relevance data derived from user selection data.

25. A normalized relevance rating according to claim 24, wherein the relevance data derived from user selection data comprises user click-through data selecting from a search results list.

26. A normalized relevance rating according to claim 24, wherein the relevance data derived from user selection data comprises variance data indicating variance of user selection from an ordinal relevance ranking.

27. A normalized relevance rating according to claim 21, wherein the generating comprises scaling the set of independently scaled relevance metrics to generate the normalized relevance output to a common scale.

28. A normalized relevance rating according to claim 27, wherein the scaling of the set of independently scaled relevance metrics comprises adjusting the contribution of each of the independently scaled relevance metrics to the normalized relevance output based on at least one of relevance metric type, a confidence measure associated with each of the relevance metrics, and a frequency of each of the relevance metrics.

29. A normalized relevance rating according to claim 27, wherein the common scale comprises a set of qualitatively labeled relevance levels.

30. A normalized relevance rating according to claim 21, wherein the scaling of the set of independently scaled relevance metrics comprises at least one of applying a scaling weight and applying a scaling function.

31. A relevance predictor model for analyzing the relevance of search results, the relevance predictor model being trained according to a method of:

receiving a set of independently scaled relevance metrics;

generating a normalized relevance output from the set of independently scaled relevance metrics; and

training the relevance predictor model using the normalized relevance output.

32. A relevance predictor model according to claim 31, wherein the set of independently scaled relevance metrics comprises at least one of relevance data derived from query logs, human-annotated relevance data and relevance ratings derived from an independent search service.

33. A relevance predictor model according to claim 32, wherein the human-annotated relevance data comprises at least one of human live site feedback, human search analyst relevance data, human explicit feedback data from a search service, and user group feedback data.

34. A relevance predictor model according to claim 32, wherein the independently scaled relevance metrics comprise at least relevance data derived from query logs, and the relevance data derived from query logs comprises relevance data derived from user selection data.

35. A relevance predictor model according to claim 34, wherein the relevance data derived from user selection data comprises user click-through data selecting from a search results list.

36. A relevance predictor model according to claim 34, wherein the relevance data derived from user selection data comprises variance data indicating variance of user selection from an ordinal relevance ranking.

37. A relevance predictor model according to claim 31, wherein the generating comprises scaling the set of independently scaled relevance metrics to generate the normalized relevance output to a common scale.

38. A relevance predictor model according to claim 37, wherein the scaling of the set of independently scaled relevance metrics comprises adjusting the contribution of each of the independently scaled relevance metrics to the normalized relevance output based on at least one of relevance metric type, a confidence measure associated with each of the relevance metrics, and a frequency of each of the relevance metrics.

39. A relevance predictor model according to claim 37, wherein the common scale comprises a set of qualitatively labeled relevance levels.

40. A relevance predictor model according to claim 31, wherein the scaling of the set of independently scaled relevance metrics comprises at least one of applying a scaling weight and applying a scaling function.

41. A relevance predictor model according to claim 31, wherein the relevance predictor model comprise a self-learning network.

42. A relevance predictor model according to claim 41, wherein the self-learning network comprises at least one of a neural network and a support vector model.

43. A set of search results, the set of search results being generated via a relevance predictor model generated according to claim 31.