System and method for measuring and improving search result relevance based on user satisfaction

• US 7,526,470 B1
• Filed: 05/28/2003
• Issued: 04/28/2009
• Est. Priority Date: 05/28/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Taylor, Thomas A., Fox, Steven C., Karnawat, Kuldeep, White, Thomas D., Mydland, Mark B.
• Primary Examiner(s): Mofiz; Apu M
• Assistant Examiner(s): Pyo; Monica M
• Application Number:
  US10/446,537
• Time in Patent Office: 2,162 Days
• Field of Search: 707/3, 707/5, 707/10, 707/2, 706/45
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2457 with adaptation to user needs
• CPC Class Codes: G06F 16/903 Querying for retrieval from the web G06F16/953
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

Systems and methods for measuring and improving search result relevance based on user satisfaction are disclosed. An event log is preferably scanned to determine the start and the end of a session. User behaviors within the session are preferably identified. User satisfaction is preferably estimated based on the user behaviors. User satisfaction is preferably a measure of the user'"'"'s perception of the extent to which a result correlates to his intent based on his behaviors. User satisfaction is preferably estimated to be one of three satisfaction levels: satisfied, partially-satisfied, and dissatisfied. Symptoms, which are indications of a need for improvement within a search system, are preferably observed based on the user behaviors and the user satisfaction. Diagnostics, which are conditions that prevent a search system from performing optimally, are preferably derived based on the symptoms. A performance report for the search system based, at least in part, on the user satisfaction and the diagnostics may be generated. Such a report may be analyzed to determine methods for improving the relevance of results generated by the search system.

Claims

1. A method for measuring the relevance of results in a search session, the search session including a set of actions, the method comprising:

executing instructions stored in memory, the instructions comprising;

logging a plurality of user actions in an event log;

determining a start of the search session by scanning the event log to identify an initial query;

determining an end of the search session by identifying an expiration of a timeout before an action is committed by a user;

for each result in the search session, identifying at least one user behavior in response to the result, the user behavior identified by determining whether the user has performed each of ignoring the result, rejecting the result, exploring the result, and accepting the result;

estimating a level of satisfaction with respect to the results in the search session based on the user behavior, whereby the level of satisfaction is estimated using three satisfaction tiers comprising dissatisfaction, satisfaction, and partial satisfaction, and whereby ignoring the result is estimated as dissatisfaction with the result, rejecting the result is estimated as dissatisfaction with the result, accepting the result is estimated as satisfaction with the result, and exploring the result is estimated as partial satisfaction with the result;

using the estimated level of satisfaction with respect to the results in the search session to determine an electronic search system'"'"'s ability to provide relevant search results that correlate to intents of user queries;

observing symptoms based on the at least one user behavior and the estimated level of satisfaction, the symptoms indicating a need for improvements within the electronic search system;

deriving diagnostics based on the observed symptoms, the diagnostics being conditions that prevent the electronic search system from performing optimally, wherein the diagnostics comprise at least one member of a group comprising content quantity, poor content quality, intent determination, result ranking, result precision, and result description, wherein deriving diagnostics comprises;

deriving the diagnostic content quantity when symptoms re-query present, no result, no accept, and re-query successful are observed; and

deriving the diagnostic poor content quality when symptom lower partial-satisfaction is observed; and

generating a performance report for the electronic search system based on the diagnostics and the estimated level of satisfaction.

2. The method of claim 1, wherein determining the end of the search session comprises scanning the event log to determine the end of the search session.

3. The method of claim 1, wherein determining the start and the end of the session comprises determining the start and the end of the session based on at least one member of a group comprising a user identifier and a client identifier.

4. The method of claim 1, wherein the user behavior is identified by further determining whether the user has performed each of invoking the result and skipping the result.

5. The method of claim 1, wherein observing symptoms comprises observing at least one member of a group comprising lower satisfaction, lower invocation, skip present, re-query present, re-query successful, no accept, lower partial-satisfaction, no result, and high explore behavior.

6. The method of claim 5, wherein deriving diagnostics further comprises:

deriving the diagnostic intent determination when the symptoms re-query present, re-query successful, no accept, and no result are observed;

deriving the diagnostic result ranking when the symptom lower satisfaction is observed;

deriving the diagnostic result precision when the symptoms skip present and high explore behavior are observed; and

deriving the diagnostic result description when the symptom lower invocation is observed and the symptom lower satisfaction is not observed.

7. The method of claim 1, wherein the instructions further comprise analyzing the report to improve the relevance of results generated by the search system.

8. A method for measuring the relevance of a search result, the method comprising:

executing instructions stored in memory, the instructions comprising;

identifying at least one user behavior in response to the search result, the user behavior identified by determining whether the user has performed each of ignoring the search result, rejecting the search result, exploring the search result, and accepting the search result; and

estimating a level of satisfaction with respect to the search result based on the at least one user behavior, whereby the level of satisfaction is estimated using three satisfaction tiers comprising dissatisfaction, satisfaction, and partial satisfaction, and whereby ignoring the search result is estimated as dissatisfaction with the search result, rejecting the search result is estimated as dissatisfaction with the search result, accepting the search result is estimated as satisfaction with the search result, and exploring the search result is estimated as partial satisfaction with the search result; and

using the estimated level of satisfaction with respect to the search result to improve the relevance of results generated by the search system by identifying at least one diagnostic selected from a group comprising content quantity, poor content quality, intent determination, result ranking, result precision, and result description, wherein identifying at least one diagnostic comprises;

identifying the diagnostic content quantity when re-query present, no result, no accept, and re-query successful are observed; and

identifying the diagnostic poor content quality when lower partial-satisfaction is observed.

9. The method of claim 8, wherein the user behavior is identified by further determining whether the user has performed each of invoking the search result and skipping the search result.

10. The method of claim 8, wherein the instructions further comprise observing at least one member of a group comprising lower satisfaction, lower invocation, skip present, re-query present, re-query successful, no accept, lower partial-satisfaction, no result, and high explore behavior.

11. The method of claim 10, wherein identifying the at least one diagnostic further comprises:

identifying the diagnostic intent determination when re-query present, re-query successful, no accept, and no result are observed;

identifying the diagnostic result ranking when lower satisfaction is observed;

identifying the diagnostic result precision when skip present and high explore behavior are observed; and

identifying the diagnostic result description when lower invocation is observed and lower satisfaction is not observed.

12. The method of claim 8, wherein the instructions further comprise generating a performance report for the search system based on the problem and the level of satisfaction.

13. The method of claim 12, wherein the instructions further comprise analyzing the report to improve the relevance of search results generated by the search system.

14. A system for measuring the relevance of an electronic search result, the system comprising:

a database for storing a log of actions performed by a user during a search;

a computing device in communication with said database, the computing device comprising;

a processor operative to execute computer executable instructions; and

a memory having stored therein computer executable instructions for performing the following steps;

scanning the log to identify a user behavior in response to the electronic search result, the user behavior identified by determining whether the user has performed each of ignoring the electronic search result, rejecting the electronic search result, exploring the electronic search result, and accepting the electronic search result;

estimating a level of satisfaction with respect to the electronic search result based on the user behavior, whereby the level of satisfaction is estimated using three satisfaction tiers comprising dissatisfaction, satisfaction, and partial satisfaction, and whereby ignoring the electronic search result is estimated as dissatisfaction with the electronic search result, rejecting the electronic search result is estimated as dissatisfaction with the electronic search result, accepting the electronic search result is estimated as satisfaction with the electronic search result, and exploring the electronic search result is estimated as partial satisfaction with the electronic search result; and

using the estimated level of satisfaction with respect to the electronic search result to determine an electronic search system'"'"'s ability to provide relevant search results that correlate to intents of user queries by identifying at least one diagnostic selected from a group comprising content quantity, poor content quality, intent determination, result ranking, result precision, and result description, wherein identifying at least one diagnostic comprises;

identifying the diagnostic content quantity when symptoms re-query present, no result, no accept, and re-query successful are observed;

identifying the diagnostic poor content quality when symptom lower partial-satisfaction is observed.

15. The system of claim 14, wherein the user behavior is identified by further determining whether the user has performed each of invoking the electronic search result and skipping the electronic search result.

16. The system of claim 14, wherein the memory has stored therein further computer executable instructions for observing symptoms based on the behavior and the level of satisfaction, the symptoms indicating needs for improvements within the electronic search system.

17. The system of claim 16, wherein observing symptoms comprises observing at least one member of a group comprising lower satisfaction, lower invocation, skip present, re-query present, re-query successful, no accept, lower partial-satisfaction, no result, and high explore behavior.

18. The system of claim 17, wherein identifying diagnostics further comprises:

identifying the diagnostic intent determination when the symptoms re-query present, re-query successful, no accept, and no result are observed;

identifying the diagnostic result ranking when the symptom lower satisfaction is observed;

identifying the diagnostic result precision when the symptoms skip present and high explore behavior are observed; and

identifying the diagnostic result description when the symptom lower invocation is observed and the symptom lower satisfaction is not observed.

19. The system of claim 14, wherein the memory has stored therein further computer executable instructions for generating a performance report for the electronic search system based on the at least one diagnostic and the estimated level of satisfaction.

20. The system of claim 19, wherein the database is further configured for storing the performance report for further processing.

21. A method for improving performance of a search system, the method comprising:

executing instructions stored in memory, the instructions comprising;

observing symptoms based on user behaviors with respect to one or more search results and user satisfaction with respect to the one or more search results, the symptoms enabling the search system'"'"'s performance to be evaluated by indicating needs for improvements within the search system; and

deriving diagnostics based on the symptoms, the diagnostics being conditions that prevent the search system from performing optimally, wherein the diagnostics comprise at least one member of a group comprising content quantity, poor content quality, intent determination, result ranking, result precision, and result description, wherein deriving diagnostics comprises;

deriving the diagnostic content quantity when symptoms re-query present, no result, no accept, and re-query successful are observed;

deriving the diagnostic poor content quality when symptom lower partial-satisfaction is observed; and

using the diagnostics to improve the search system'"'"'s ability to provide more relevant results that correlate to intents of user queries.

22. The method of claim 21, wherein the user satisfaction with respect to the one or more search results is estimated based on the user behaviors.

23. The method of claim 21, wherein observing symptoms comprises observing at least one member of a group comprising lower satisfaction, lower invocation, skip present, re-query present, re-query successful, no accept, lower partial-satisfaction, no result, and high explore behavior.

24. The method of claim 23, wherein deriving diagnostics further comprises:

deriving the diagnostic intent determination when the symptoms re-query present, re-query successful, no accept, and no result are observed;

deriving the diagnostic result ranking when the symptom lower satisfaction is observed;

deriving the diagnostic result precision when the symptoms skip present and high explore behavior are observed; and

deriving the diagnostic result description when the symptom lower invocation is observed and the symptom lower satisfaction is not observed.

25. The method of claim 21, wherein the instructions further comprise generating a performance report for the search system based on the diagnostics and the user satisfaction with respect to the one or more search results.