Search diagnostics based upon query sets

• US 7,904,440 B2
• Filed: 04/26/2007
• Issued: 03/08/2011
• Est. Priority Date: 04/26/2007
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Anderson, Blake E., Jacobs, Nancy E., Mehta, Reshma Girish
• Primary Examiner(s): Abel-Jalil; Neveen
• Assistant Examiner(s): YOUNG, KEVIN L.
• Application Number:
  US11/740,484
• Publication Number:
  US 20080270356A1
• Time in Patent Office: 1,412 Days
• Field of Search: None
• US Class Current: 707/706
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The subject disclosure pertains to systems and methods that facilitate monitoring of data collections and related data retrieval systems as well as generating recommendations for enhancing performance and utility of such systems. In aspects, the systems and methods described herein evaluate data retrieval performance by identifying unsuccessful search queries and mapping such queries to appropriate content in a data collection. To facilitate such evaluation, similar queries can be grouped and treated as equivalent. Similarity of queries can be determined based upon search terms. For example, queries whose search terms only vary syntactically can be combined in a group. Such groups can include search information related to multiple users, increasing reliability of recommendations based upon such groups. In further aspects, recommendations for adding content can be generated in conjunction with any mapping recommendations.

Claims

1. A diagnostic system that facilitates evaluation of performance of a search system, comprising:

a memory for storing a plurality of components comprising executable program code; and

a processor functionally coupled to the memory, the processor being responsive to computer-executable instructions contained in the program code and operative;

to identify a set of substantially similar search queries issued by a plurality of search system users, wherein the set of substantially similar search queries are grouped for analysis, the grouped set of search queries being issued by disparate search system users;

to identify a mapping recommendation that relates the grouped set of search queries issued by the disparate search system users to an accepted asset within a data collection when at least one query in the set of search queries appears in a session that leads to the accepted asset, wherein the at least one query is within a session comprising a plurality of queries, the at least one query comprising at least one previous query made prior to a subsequent query, the subsequent query resulting in the retrieval of an accepted asset in the session, wherein the at least one previous query does nor result in the retrieval of the accepted asset, wherein the mapping indicates a relationship between the grouped set of search queries and the accepted asset where the grouped set of search queries would not otherwise be associated with the accepted asset;

to evaluate search system performance as a function of the set of search queries; and

to generate a confidence value based upon the amount of support for the mapping within the search data that reflects probable user satisfaction associated with the mapping recommendation, wherein the mapping recommendation is categorized based upon the confidence value, wherein the mapping recommendation is implemented automatically by the search system when the mapping recommendation has a high confidence value, wherein the mapping recommendation is included in recommendations to content providers by the search system when the mapping recommendation has a mid-range confidence value and wherein the mapping recommendation is identified for further evaluation by the search system when the mapping recommendation has a relatively low confidence value.

2. The system of claim 1, wherein the processor is further operative to obtain search data that includes search history for the search system and the set of search queries is identified from the search data.

3. The system of claim 1, wherein the processor is further operative to adjust the confidence value as a function of at least one of number of assets to which the set of queries is mapped, number of query sets to which the asset is mapped, or a combination thereof.

4. The system of claim 1, wherein the processor is further operative to adjust the confidence value as a function of combinations of the set of queries and the asset within the search data.

5. The system of claim 1, further comprising a user interface that alerts a user to the mapping recommendation.

6. The system of claim 1, wherein the processor is further operative to analyze at least one of implicit user feedback, explicit user feedback, or a combination thereof to determine user satisfaction with the asset, wherein the mapping recommendation is based at least in part upon user satisfaction with the asset.

7. The system of claim 1, wherein the processor is further operative to identify potential content related to the set of queries for addition to a data collection associated with the search system.

8. The system of claim 7, wherein the processor is further operative to determine if there is a mapping recommendation that relates the set of search queries to an asset within the data collection prior to recommending addition of the potential content to the data collection.

9. A methodology that facilitates evaluation of search performance, comprising:

obtaining a search history associated with a search system that searches a data set;

analyzing at least one group of similar queries within the search history, the at least one group of similar queries being issued by disparate search system users;

identifying a mapping recommendation that relates the at least one group of similar queries issued by the disparate search system users to an accepted asset within a data collection when at least one query in the set of search queries appears in a session that leads to the accepted asset, wherein the at least one query is within a session comprising a plurality of queries, the at least one query comprising at least one previous query made prior to a subsequent query, the subsequent query resulting in the retrieval of an accepted asset in the session, wherein the at least one previous query does nor result in the retrieval of the accepted asset, wherein the mapping indicates a relationship between the at least one group of similar queries and the accepted asset where the at least one group of similar queries would not otherwise be associated with the accepted asset;

determining a potential modification designed to enhance user satisfaction with the search system as a function of the analysis of the group of similar queries; and

generating a confidence value based upon the amount of support for the mapping within the search data that reflects probable user satisfaction associated with the mapping recommendation, wherein the mapping recommendation is categorized based upon the confidence value, wherein the mapping recommendation is implemented automatically by the search system when the mapping recommendation has a high confidence value, wherein the mapping recommendation is included in recommendations to content providers by the search system when the mapping recommendation has a mid-range confidence value and wherein the mapping recommendation is identified for further evaluation by the search system when the mapping recommendation has a relatively low confidence value.

10. The methodology of claim 9, further comprising:

selecting a candidate query from the search history;

identifying a group from the at least one group of queries, wherein the candidate query is a member of the identified group; and

determining a score that reflects the utility of a mapping of the candidate query to an asset within the data set, wherein the score is based at least in part upon support related to the identified group of queries.

11. The methodology of claim 10, further comprising applying a penalty to the score as a function of at least one of broad nature of the asset, broad nature of the candidate query, or a combination thereof.

12. The methodology of claim 9, further comprising generating a recommendation to add content to the data set, wherein the recommendation is a function of the analysis of the at least one group of similar queries.

13. The methodology of claim 12, further comprising:

selecting a group from the at least one group of similar queries; and

determining a percentage of unsuccessful queries within the selected group, wherein the recommendation is based at least in part upon comparison of the percentage to a configurable threshold.

14. The methodology of claim 13, further comprising determining whether a mapping recommendation exists for the selected group of queries, wherein the recommendation is based at least in part upon the determination.

15. The methodology of claim 9, wherein queries within the at least one group of queries are identified as similar based upon a search term contained within the queries.

16. The methodology of claim 15, wherein syntactic variations among search terms are ignored during identification of the similar queries.

17. A diagnostic apparatus that facilitates evaluation of performance of a search system, comprising:

a memory means for storing a plurality of components comprising executable program code; and

a processing means functionally coupled to the memory means, the processor being responsive to computer-executable instructions contained in the program code and operative;

to identify a set of substantially similar search queries issued by a plurality of search system users, wherein the set of substantially similar search queries are grouped for analysis, the grouped set of search queries being issued by disparate search system users;

to identify a mapping recommendation that relates the grouped set of search queries issued by the disparate search system users to an accepted asset within a data collection when at least one query in the set of search queries appears in a session that leads to the accepted asset, wherein the at least one query is within a session comprising a plurality of queries, the at least one query comprising at least one previous query made prior to a subsequent query, the subsequent query resulting in the retrieval of an accepted asset in the session, wherein the at least one previous query does nor result in the retrieval of the accepted asset, wherein the mapping indicates a relationship between the grouped set of search queries and the accepted asset where the grouped set of search queries would not otherwise be associated with the accepted asset;

to evaluate search system performance as a function of the set of search queries; and

to generate a confidence value based upon the amount of support for the mapping within the search data that reflects probable user satisfaction associated with the mapping recommendation, wherein the mapping recommendation is categorized based upon the confidence value, wherein the mapping recommendation is implemented automatically by the search system when the mapping recommendation has a high confidence value, wherein the mapping recommendation is included in recommendations to content providers by the search system when the mapping recommendation has a mid-range confidence value and wherein the mapping recommendation is identified for further evaluation by the search system when the mapping recommendation has a relatively low confidence value.