Ranking search results using click-based data

• US 8,370,337 B2
• Filed: 04/19/2010
• Issued: 02/05/2013
• Est. Priority Date: 04/19/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kanungo, Tapas, Agrawal, Nitin, McDonald, Kieran Richard, Meyers, Christopher Avery, Malhotra, Nipoon, Pattabiraman, Kumaresh
• Primary Examiner(s): HICKS, MICHAEL J
• Application Number:
  US12/762,929
• Publication Number:
  US 20110258149A1
• Time in Patent Office: 1,023 Days
• Field of Search: None
• US Class Current: 707/723
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9558 Details of hyperlinks; Management of linked annotations
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Methods and computer-storage media having computer-executable instructions embodied thereon that facilitate generating a machine-learned model for ranking search results using click-based data are provided. Data is referenced from user queries, which may include search results generated by general search engines and vertical search engines. A training set is generated from the search results and click-based judgments are associated with the search results in the training set. Based on click-based judgments, identifiable features are determined from the search results in a training set. Based on determining identifiable features in a training set, a rule set is generated for ranking subsequent search results.

Claims

1. One or more computer-readable media storing computer-useable instructions that, when used by one or more computing devices, causes the one or more computing devices to perform a method of generating a machine-learned model for ranking search results using click-based data comprising:

referencing data from one or more user queries, wherein the data is referenced from one or more of a general search engine and a vertical search engine;

generating a training set of data, wherein the training set comprises one or more search results extracted from the data;

associating one or more click-based judgments with each of the one or more search results in the training set, wherein associating one or more click-based judgments with each of the one or more search results in the training set comprises;

(1) determining that a plurality of queries are tail queries, wherein a tail query satisfies a threshold for the data referenced from one or more queries;

(2) aggregating the search results of the plurality of tail queries into one or more classes of tail queries, such that the training set comprises at least one class of tail queries having a plurality of tail queries; and

(3) associating one or more click-based judgments with each of the one or more classes of tail queries in the training set;

based on associating one or more click-based judgments with each of the one or more search results in the training set, determining one or more identifiable features from the training set; and

based on determining one or more identifiable features, generating a rule set for ranking subsequent search results of one or more user queries.

2. The one or more computer-readable media of claim 1, wherein the one or more search results extracted from the data comprises one or more of general search engine results and vertical search engine results.

3. The one or more computer-readable media of claim 1, wherein associating one or more click-based judgments with each of the one or more search results in the training set comprises:

determining that one or more user queries is a head query, wherein a head query satisfies a threshold for the data referenced from one or more user queries; and

associating one or more click-based judgments directly with each of the one or more search results of the one or more head queries in the training set.

4. The one or more computer-readable media of claim 1, wherein associating one or more click-based judgments with each of the one or more search results in the training set comprises:

receiving click-based judgments from one or more human judges, wherein the click-based judgments correspond to user preferences regarding one or more of the plurality of search results in the training set.

5. The one or more computer-readable media of claim 4, wherein receiving click-based judgments from one or more human judges comprises:

presenting one or more human judges with the training set; and

generating a corresponding set of click-based judgments based on feedback from one or more human judges regarding each of the one or more search results in the training set.

6. The one or more computer-readable media of claim 1, wherein the one or more identifiable features comprise at least one of clicks on only textual search results, clicks on vertical search results, clicks on captions and URLs of query-URL pairs, query classification confidence, vertical query confidence, overall clicks on the entire webpage, clicks on advertisements, and clicks on links to see the next page of results.

7. The one or more computer-readable media of claim 1, wherein the method further comprises:

associating a particular click-based judgment with each of the determined one or more identifiable features.

8. The one or more computer-readable media of claim 1, wherein the method further comprises:

appending an absolute numerical value to each of the one or more identifiable features.

9. The one or more computer-readable media of claim 1, wherein the method further comprises:

appending an absolute numerical value to each of the one or more click-based judgments, wherein the absolute numerical value is based on a scale of numerical values associated with particular click-based judgments.

10. A method performed by one or more server devices for ranking search results, the method comprising:

referencing data from one or more user queries, wherein the data comprises one or more of general search engine results and vertical search engine results;

generating a training set, wherein the training set comprises one or more search results extracted from the data;

associating one or more click-based judgments with each of the one or more search results in the training set, wherein associating one or more click-based judgments with each of the one or more search results in the training set comprises;

(1) determining that a plurality user queries are tail queries, wherein a tail query satisfies a threshold for the data referenced from one or more user queries;

(2) aggregating the search results of plurality of tail queries into one or more classes of tail queries, such that the training set comprises at least one class of tail queries having a plurality of tail queries; and

(3) associating one or more click-based judgments with each of the one or more classes of tail queries in the training set;

determining one or more identifiable features from the training set based on the associated one or more click-based judgments; and

based on determining one or more identifiable features from the training set, generating a rule set for ranking search results.

11. The method of claim 10, wherein associating one or more click-based judgments with each of the one or more search results in the training set comprises:

determining that one or more user queries is a head query, wherein a head query satisfies a threshold for the data referenced from one or more user queries; and

associating one or more click-based judgments directly with each of the one or more search results of the one or more head queries in the training set.

12. The method of claim 10, wherein associating one or more click-based judgments with each of the one or more search results in the training set comprises:

presenting one or more human judges with the training set; and

generating a corresponding set of click-based judgments based on feedback from one or more human judges, wherein the click-based judgments correspond to user preferences regarding one or more of the plurality of search results in the training set.

13. The method of claim 10, wherein the one or more identifiable features comprise at least one of clicks on only textual search results, clicks on vertical search results, clicks on captions and URLs of query-URL pairs, query classification confidence, vertical query confidence, overall clicks on the entire webpage, clicks on advertisements, and clicks on links to see the next page of results.

14. The method of claim 10, further comprising:

appending an absolute numerical value to each of the one or more identifiable features.

15. The method of claim 10, further comprising:

appending an absolute numerical value to each of the one or more click-based judgments, wherein the absolute numerical value is based on a scale of numerical values associated with particular click-based judgments.

16. The method of claim 10 further comprising:

associating one or more human-based judgments with at least one of the one or more search results in the training set;

determining one or more identifiable features from the training set based on the associated one or more human-based judgments;

based on determining one or more identifiable features, generating a rule set.

17. The method of claim 16 further comprising:

adjusting a proportion of click-based judgments associated with one or more search results and a proportion of human-based judgments associated with one or more search results for a training set using an equation as follows;

TOTAL=(α

)X+(1−

α

)Y

wherein the value of a is adjusted between 0 and 1;

X represents human-based judgments associated with one or more search results in the training set; and

Y represents click-based judgments associated with one or more search results in the training set.

18. One or more computer-readable media storing computer-useable instructions that, when used by one or more computing devices, causes the one or more computing devices to perform a method for ranking search results using click-based data comprising:

referencing data from one or more user queries, wherein the data is referenced from one or more of general search engines and vertical search engines;

generating a training set of data, wherein the training set comprises a plurality of search results from the data, further wherein the plurality of search results comprises one or more of a general search engine result and a vertical search engine result;

associating one or more click-based judgments with the plurality of search results in the training set based on;

(1) determining whether one or more user queries is a head query, wherein a head query satisfies a threshold for the data referenced from one or more user queries;

(2) associating one or more click-based judgments directly with each of the plurality of search results for the one or more head queries in the training set;

(3) determining whether a plurality of user queries are tail queries, wherein a tail query satisfies a threshold for the data referenced from one or more user queries;

(4) aggregating the plurality of search results of one or more tail queries into one or more classes of tail queries, such that the training set comprises at least one class of tail queries having a plurality of tail queries; and

(5) associating one or more click-based judgments with each of the classes of tail queries in the training set;

based on associating the one or more click-based judgments with each of the plurality of search results in the training set, determining one or more identifiable features from the training set, wherein the one or more identifiable features comprise at least one of;

clicks on only textual search results;

clicks on vertical search results;

clicks on captions and URLs of query-URL pairs;

query classification confidence;

vertical query confidence;

overall clicks on the entire webpage;

clicks on advertisements; and

clicks on links to see the next page of results

appending an absolute numerical value to each of the one or more identifiable features;

generating a rule set based on the one or more identifiable features of the one or more user queries; and

utilizing the rule set to rank subsequent search results.