Automated satisfaction measurement for web search

• US 7,937,340 B2
• Filed: 03/22/2004
• Issued: 05/03/2011
• Est. Priority Date: 12/03/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Watson, Eric, Dumais, Susan T., Hurst-Hiller, Oliver
• Primary Examiner(s): Vincent; David R
• Assistant Examiner(s): Fernandez Rivas; Omar F
• Application Number:
  US10/806,271
• Publication Number:
  US 20050125390A1
• Time in Patent Office: 2,598 Days
• Field of Search: 706/1, 706 10- 21, 709/201, 709/227, 709/203, 715/700, 715/701, 715/745, 715/789, 715/780, 715/811, 707/3, 707/5, 707/6, 707/1, 707/10
• US Class Current: 706/14
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 7/00 Methods or arrangements for processing data by operating upon the order or content of the data handled logic circuits H03K19/00

Abstract

Context-based user behavior data is collected from a search mechanism. This data includes, for a given query, user feedback (implicit and explicit) on the query and context information on the query. A predictive pattern is applied to the context-based user behavior data in order to produce predicted user satisfaction data. Data mining techniques may be used to create and improve one or more predictive patterns. Predicted user satisfaction data can be used to monitor or improve search mechanism performance, via a display reporting the performance or identification of any queries with a shared characteristic and sub-par user satisfaction. A dynamically-improving search mechanism uses the predicted user satisfaction data to improve the performance of the search mechanism.

Claims

1. A method for obtaining predicted user satisfaction data regarding the performance of a search mechanism for a computer which provides search results in response to user queries, the method comprising:

storing interaction of users with the search results obtained in response to a query;

determining at least one predictive pattern model for predicting user-satisfaction with the search results from the stored interaction of multiple users with the search results to improve the quality of search results, wherein determining at least one predictive pattern model comprises;

running a pre judged set of results through the search mechanism;

applying a candidate predictive pattern to context-based user behavior data for the pre-judged set of results;

comparing resulting predicted user satisfaction data with judged satisfaction for the pre judged set of results; and

if the candidate predictive pattern is successful at determining a predicted user satisfaction corresponding to the pre-judged user satisfaction, selecting the candidate predictive pattern as one of the at least one predictive pattern model;

storing the at least one predictive pattern model; and

applying said predictive pattern model to context-based user behavior data to determine a satisfaction value for the users with queries that indicate the satisfaction of the users with the search results, wherein the context-based user behavior data comprises user feedback data and context data associated with users feedback data, the context-based user behavior data acquired after receipt by users of the search results of said application of said predictive pattern model further comprises isolating a set of said performed queries which are unsatisfactory and which share a common characteristic in order to identify problems which appear for multiple users or queries.

2. The method of claim 1, where said storing at least one predictive pattern model comprises utilizing data mining techniques to determine at least one predictive pattern model for user satisfaction.

3. The method of claim 1, where said context-based user behavior data comprises explicit user feedback data.

4. The method of claim 1, where said context-based user behavior data comprises implicit user feedback data.

5. The method of claim 4, where said context-based user behavior data is selected from the group comprising:

user navigation to a new page using a hyperlink;

user navigation to a new page using a history list;

user navigation to a new page using an address bar;

user navigation to a new page using a favorites list;

user scrolling behavior;

user document printing behavior;

user adding a document to said favorites list;

user switching focus to a different application;

user switching focus back from a different application;

user closing a window;

user dwell time behavior;

user initiation of a new query;

sequences of user behaviors; and

user inactivity without switching focus from a window relating to said performed query.

6. The method of claim 1, where said application of said predictive pattern model yields predicted user satisfaction data regarding said search mechanism, and where said method further comprises:

displaying said predicted user satisfaction data.

7. The method of claim 1, where said context-based user behavior data comprises a testing set of context-based user behavior data.

8. The method of claim 1, further comprising isolating problematic queries based on the predicted user satisfaction data.

9. The method of claim 1, further comprising generating a summary of measured satisfaction based on the predicted user satisfaction data.

10. The method of claim 1, further comprising monitoring a search mechanism responsive to the predicted user satisfaction data.

11. A system for obtaining predicted user satisfaction data regarding the performance of a search mechanism for a computer which provides search results in response to user'"'"'s queries, the system comprising:

storage for storing interaction of users with the search results obtained in response to a query;

a processor for determining at least one predictive pattern model for predicting user satisfaction with the search results from the stored interaction of multiple users with the search results to improve search quality, wherein the at least one predictive pattern model is determined by;

running a pre-judged set of results through the search mechanism;

applying a candidate predictive pattern to context-based user behavior data for the pre judged set of results;

comparing resulting predicted user satisfaction data with judged satisfaction for the pre judged set of results; and

if the candidate predictive pattern is successful at determining a predicted user satisfaction corresponding to the pre judged user satisfaction, selecting the candidate predictive pattern as one of the at least one predictive pattern model;

storage for storing at least one predictive pattern model; and

data mining apparatus for applying said predictive pattern model to context-based user behavior data, said context-based user behavior data comprising user feedback data and context data associated with the user feedback data, the context-based user behavior data acquired after receipt by users of the search results of said application of said predictive pattern model further comprises isolating a set of said performed queries which are unsatisfactory and which share a common characteristic in order to identify problems which appear for multiple users or queries.

12. The system of claim 11, where said predictive pattern model is derived from the use of data mining techniques to determine at least one predictive pattern model for user satisfaction.

13. The system of claim 11, where said context-based user behavior data comprises explicit user feedback data.

14. The system of claim 11, where said context-based user behavior data comprises implicit user feedback data.

15. The system of claim 14, where said context-based user behavior data is selected from the group comprising:

user navigation to a new page using a hyperlink;

user navigation to a new page using a history list;

user navigation to a new page using an address bar;

user navigation to a new page using a favorites list;

user scrolling behavior;

user document printing behavior;

user adding a document to said favorites list;

user switching focus to a different application;

user switching focus back from a different application;

user closing a window;

user dwell time behavior;

user initiating a new query;

sequences of user behaviors; and

user inactivity without switching focus from a window relating to said performed query.

16. The system of claim 11, where said data mining apparatus produces predicted user satisfaction data regarding said search mechanism, and where said method further comprises:

displaying said predicted user satisfaction data.

17. The system of claim 11, where said data mining apparatus further isolates a set of said performed queries which are unsatisfactory and which share a common characteristic.

18. The system of claim 11, where said context-based user behavior data comprises a testing set of context-based user behavior data.

19. A system for real-time optimization of a search mechanism for a computer which provides search results in response to user queries, comprising:

a processor, comprising;

means for storing satisfaction of users with search results that is determined by an evaluation of the search results by the user;

means for storing interaction of users with the search results obtained in response to a query;

means for determining at least one predictive pattern model for predicting user satisfaction with the search results based on the satisfaction of the user stored in the means for storing and the user interaction stored in the means for storing multiple users interaction to improve search quality, wherein the at least one predictive pattern model is determined by;

running a pre-judged set of results through the search mechanism;

applying a candidate predictive pattern to context-based user behavior data for the pre-judged set of results;

comparing resulting predicted user satisfaction data with judged satisfaction for the pre-judged set of results; and

if the candidate predictive pattern is successful at determining a predicted user satisfaction corresponding to the pre-judged user satisfaction, selecting the candidate predictive pattern as one of the at least one predictive pattern model;

means for storing at least one predictive pattern model;

means for applying said predictive pattern model to context-based user behavior data comprising user feedback data and context data associated with user'"'"'s feedback data, the context-based user behavior data acquired after receipt by a user of the search results of said application of said predictive pattern model further comprises isolating a set of said performed queries which are unsatisfactory and which share a common characteristic in order to identify problems which appear for multiple users or queries; and

means for outputting predicted user satisfaction data which is indicative of a level of satisfaction experienced by users of search results returned by the search mechanism, the outputted predicted user satisfaction data based on the application of said predictive pattern model.