System and method for automated optimization of search result relevance

• US 20050154716A1
• Filed: 03/22/2004
• Published: 07/14/2005
• Est. Priority Date: 01/09/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Watson, Eric B., Hurst-Hiller, Oliver
• Granted Patent: US 7,444,327 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/835 Query processing
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

A method, system, and computer-accessible medium are provided for automating the optimization of search result relevance in a search engine. The system and method continually collect data that represent various aspects of how a search result is performing and compare that performance data to the expected performance for the search result. The system and method further diagnose the possible causes of under performing results and automatically adjust the search engine operation to optimize the search result relevance.

Claims

1. A method for automating the optimization of search results displayed in a search Web page, the method comprising:

collecting data that represents a performance of a search result, the data originating from at least one of a plurality of sources of performance data;

normalizing the collected data in accordance with a relative importance of the source of the data;

comparing the normalized performance data to an expected performance data for the search result;

diagnosing at least one possible cause for an under performing search result; and

adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance.

2. The method of claim 1, wherein the at least one of the plurality of sources of performance data includes one of an implicit performance data, an explicit performance data, a human-judged performance data, a relevance verification data, and a sample test data.

3. The method of claim 1, wherein normalizing the collected data in accordance with a relative importance of the source of the data includes giving greater weight to the data from the more important sources and combining the data to reflect the relative importance of the source from which the data originated.

4. The method of claim 2, wherein comparing the normalized performance data to the expected performance data includes determining that the search result is under performing when the normalized performance is below or substantially below the expected performance.

5. The method of claim 2, wherein the most important source of data is implicit performance data, and normalizing the collected data includes giving implicit performance data substantially greater weight when combining the data.

6. The method of claim 2, wherein implicit performance data is automatically captured when a user interacts with the search result.

7. The method of claim 6, wherein implicit performance data includes at least one of whether the user clicked on the result, a location of the result when the user clicked the result, and a length of time that the user browsed the result.

8. The method of claim 7, wherein implicit performance data further identifies an operation that the user performed on the result, including at least one of editing, e-mailing, printing, bookmarking, and copying.

9. The method of claim 6, wherein implicit performance data includes captured data that has been aggregated across multiple interactions with the search result.

10. The method of claim 6, wherein implicit performance data includes captured data that has been aggregated across multiple users interacting with the search result.

11. The method of claim 2, wherein explicit performance data includes data obtained from user response to a search engine operator-generated inquiry about the search result, wherein the inquiry includes one of an on-line inquiry and a telephone inquiry.

12. The method of claim 2, wherein human-judged performance data includes data obtained from a human evaluation of the search result.

13. The method of claim 2, wherein relevance verification data includes data obtained from a verification test of the search result'"'"'s relevance to verify whether a search result having a known relevance is still included in the search result generated by the search engine.

14. The method of claim 2, wherein sample test data includes data obtained from a test of the search result'"'"'s relevance performed on a sample of a subset of users.

15. The method of claim 2, wherein diagnosing at least one possible cause for an under performing search result includes considering at least one of whether the search result is no longer valid, whether the search result appears in a poor location, whether a search term that generated the search result is easily misspelled, whether the search term is too broad to generate a meaningful result, and whether a search for the search term should be constrained to a specific resource.

16. The method of claim 2, wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes modifying the search engine'"'"'s search schema to change the search result generated for the search term, including at least one of reranking, removing, and replacing the search results.

17. The method of claim 2, wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes modifying the search engine'"'"'s search schema to augment a presentation of the search results generated for the search term, including at least one of highlighting, animating, enlarging, and repositioning an appearance of the search result on a search results web page.

18. The method of claim 2, wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes increasing the search engine'"'"'s spellchecker tolerance.

19. The method of claim 2, wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes prompting the user to one of clarify or narrow the search term with an additional user input.

20. The method of claim 2, wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes temporarily adjusting the operation of the search engine, and determining whether the adjustments have actually improved the search result performance before permanently adjusting the operation of the search engine.

21. The method of claim 2, wherein adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance, includes adjusting the operation of the search engine in real time.

22. The method of claim 21, wherein adjusting the operation of the search engine in real time includes intercepting the search result generated by the search engine and modifying the search result before the search engine displays the search result to the user.

23. An automated search result optimization system comprising:

an input data representing a performance of a search result generated by a search engine for a search term, the input data originating from a plurality of sources;

a data collection process to collect the input performance data from the plurality of sources;

a diagnostic process to compare a performance of the search result, as represented by the collected performance data, to an expected performance of the search result; and

an adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine whenever the search result performance compares unfavorably to the expected performance, wherein the adjusted operation of the search engine is to improve the performance of the search result.

24. The system of claim 23, wherein the at least one of the plurality of sources of input performance data includes one of an implicit performance data, an explicit performance data, a human-judged performance data, a relevance verification data, and a sample test data.

25. The system of claim 23, wherein the data collection process includes a normalizing process to normalize the collected data in accordance with a relative importance of the source of the data, and further wherein normalizing includes giving greater weight to the data from the more important sources and combining the data to reflect the relative importance of the source from which the data originated.

26. The system of claim 25, wherein the diagnostic process compares the performance of the search result as represented by the normalized collected performance data to an expected performance of the search result;

and further, wherein the comparison is unfavorable when the performance is lower than or substantially lower than the expected performance.

27. The system of claim 25, wherein the most important source of data is implicit performance data, and normalizing the collected data includes giving implicit performance data substantially greater weight when combining the data.

28. The system of claim 27, wherein implicit performance data is automatically captured when a user interacts with the search result.

29. The system of claim 28, wherein implicit performance data includes at least one of whether the user clicked on the result, a location of the result when the user clicked the result, and a length of time that the user browsed the result.

30. The system of claim 29, wherein implicit performance data further includes data representing an identification of an operation that the user performed on the result, including at least one of editing, e-mailing, printing, bookmarking, and copying.

31. The system of claim 28, wherein implicit performance data includes captured data that has been aggregated across multiple interactions with the search result.

32. The system of claim 28, wherein implicit performance data includes captured data that has been aggregated across multiple users interacting with the search result.

33. The system of claim 24, wherein explicit performance data includes data obtained from user response to a search engine operator-generated user satisfaction survey about the search result.

34. The system of claim 24, wherein human-judged performance data includes data obtained from a human evaluation of the search result.

35. The system of claim 24, wherein relevance verification data includes data obtained from a verification test of the search result'"'"'s relevance to verify whether a search result having a known relevance is still included in the search result generated by the search engine.

36. The system of claim 24, wherein sample test data includes data obtained from a test of the search result'"'"'s relevance performed on a sample of a subset of users.

37. The system of claim 24, wherein the diagnostic process further includes diagnosing at least one possible reason why the performance of the search result compares unfavorably to the expected performance, including considering at least one of whether the search result is no longer valid, whether the search result appears in a poor location, whether a search term that generated the search result is easily misspelled, whether the search term is too broad to generate a meaningful result, and whether a search for the search term should be constrained to a specific resource.

38. The system of claim 23, wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to modify the search engine'"'"'s search schema, wherein the modified search schema changes the way the search engine generates a search result for a search term, including at least one of reranking, removing, and replacing the search result.

39. The system of claim 23, wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to modify the search engine'"'"'s search schema, wherein the modified search schema augments the way the search engine presents the search result for a search term, including at least one of highlighting, animating, enlarging, and repositioning an appearance of the search result on a search result Web page.

40. The system of claim 23, wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to increase the search engine'"'"'s spellchecker tolerance.

41. The system of claim 23, wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to prompt the user to one of clarify or narrow the search term with an additional user input.

42. The system of claim 23, wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to temporarily adjust the operation of the search engine, and to further determine whether the temporary adjustment has actually improved the search result performance before generating an action to permanently adjust the operation of the search engine.

43. The system of claim 23, wherein the adjustment process to generate an output data representing an action to automatically adjust an operation of the search engine, includes generating output data that represents an action to adjust the operation of the search engine in real time.

44. The system of claim 42, wherein the action to adjust the operation of the search engine in real time includes an action to intercept the search result generated by the search engine and to further modify the search result before the search engine displays the search result to the user.

45. One or more computer-accessible media having instructions for facilitating the automated optimization of a search result in a search result user interface, the instructions comprising:

collecting data that represents a performance of a search result from at least one of a plurality of sources of performance data;

normalizing the collected data in accordance with a relative importance of the source of the data;

comparing the normalized performance data to an expected performance data for the search result;

diagnosing at least one possible reason why the search result performance compares unfavorably to the expected performance; and

adjusting an operation of a search engine that produced the search result in accordance with the diagnosis to improve the search result performance.

46. The computer-accessible media of claim 45, wherein the at least one of the plurality of sources of performance data includes one of an implicit performance data, an explicit performance data, a human-judged performance data, a relevance verification data, and a sample test data.

47. The computer-accessible media of claim 45, wherein the instruction to normalize the collected data in accordance with a relative importance of the source of the data includes instructions to give greater weight to the data from the more important sources and to combine the data to reflect the relative importance of the source from which the data originated.

48. The computer-accessible media of claim 45, wherein the instruction to compare the normalized performance data to the expected performance data includes an instruction to determine that the search result is under performing when the normalized performance is below or substantially below the expected performance.

49. The computer-accessible media of claim 47, wherein the most important source of data is implicit performance data, and the instruction to normalize the collected data includes an instruction to give implicit performance data substantially greater weight when combining the data.

50. The computer-accessible media of claim 46, wherein implicit performance data is data that is automatically captured by a search engine when a user interacts with a search result.

51. The computer-accessible media of claim 50, wherein implicit performance data includes data that represents at least one of whether the user clicked on the result, a location of the result when the user clicked the result, and a length of time that the user browsed the result.

52. The computer-accessible media of claim 50, wherein implicit performance data further identifies an operation that the user performed on the result, including at least one of editing, e-mailing, printing, bookmarking, and copying.

53. The computer-accessible media of claim 50, wherein implicit performance data includes captured data that has been aggregated across multiple interactions with the search result.

54. The computer-accessible media of claim 50, wherein implicit performance data includes captured data that has been aggregated across multiple users interacting with the search result.

55. The computer-accessible media of claim 46, wherein explicit performance data includes data obtained from user response to a search engine operator-generated inquiry about the search result, wherein the inquiry includes one of an on-line inquiry and a telephone inquiry.

56. The computer-accessible media of claim 46, wherein human-judged performance data includes data obtained from a human evaluation of the search result.

57. The computer-accessible media of claim 46, wherein relevance verification data includes data obtained from a verification test of the search result'"'"'s relevance to verify whether a search result having a known relevance is still included in the search result generated by the search engine.

58. The computer-accessible media of claim 46, wherein sample test data includes data obtained from a test of the search result'"'"'s relevance performed on a sample of a subset of users.

59. The computer-accessible media of claim 45, wherein the instruction to diagnose at least one possible reason why the search result performance compares unfavorably to the expected performance includes an instruction to determine at least one of whether the search result is no longer valid, whether the search result appears in a poor location, whether a search term that generated the search result is easily misspelled, whether the search term is too broad to generate a meaningful result, and whether a search for the search term should be constrained to a specific resource.

60. The computer-accessible media of claim 45, wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to modify the search engine'"'"'s search schema to change the search result generated for the search term, wherein to change the search result includes at least one of to rerank, remove, and replace the search result.

61. The computer-accessible media of claim 45, wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to modify the search engine'"'"'s search schema to augment a presentation of the search result generated for the search term, wherein to augment the presentation includes at least one of to highlight, animate, enlarge, and reposition a display of the search result on a search result Web page.

62. The computer-accessible media of claim 45, wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to increase the search engine'"'"'s spellchecker tolerance.

63. The computer-accessible media of claim 45, wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to prompt the user to one of clarify or narrow the search term with an additional user input.

64. The computer-accessible media of claim 45, wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to temporarily adjust the operation of the search engine, and an instruction to further determine whether the adjustments have actually improved the search result performance before performing an instruction to permanently adjust the operation of the search engine.

65. The computer-accessible media of claim 45, wherein the instruction to adjust the operation of the search engine that produced the search result in accordance with the diagnosis, includes an instruction to adjust the operation of the search engine in real time.

66. The computer-accessible media of claim 65, wherein the instruction to adjust the operation of the search engine in real time includes an instruction to intercept the search result generated by the search engine and a further instruction to modify the search result before the search engine displays the search result to the user.