Modifying search result ranking based on implicit user feedback and a model of presentation bias

• US 8,938,463 B1
• Filed: 03/12/2007
• Issued: 01/20/2015
• Est. Priority Date: 03/12/2007
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kim, Hyung-Jin, Corduneanu, Adrian D.
• Primary Examiner(s): Bhatia, Ajay
• Assistant Examiner(s): MUELLER, KURT A
• Application Number:
  US11/685,095
• Time in Patent Office: 2,871 Days
• Field of Search: 707/751
• US Class Current: 707/751
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

The present disclosure includes systems and techniques relating to ranking search results of a search query. In general, the subject matter described in this specification can be embodied in a computer-implemented method that includes: receiving multiple features, including a first feature indicative of presentation bias that affects document result selection for search results presented in a user interface of a document search service; obtaining, based on the multiple features, information regarding document result selections for searches performed using the document search service; generating a prior model using the information, the prior model representing a background probability of document result selection given values of the multiple features; and outputting the prior model to a ranking engine for ranking of search results to reduce influence of the presentation bias.

Claims

1. A computer-implemented method comprising:

obtaining information regarding selections of search results provided in response to a plurality of search queries, the obtained information for one or more of the selected search results comprising one or more presentation bias features of a presentation of the search result and one or more relevancy features of the search result, wherein at least one of the presentation bias features is a rank of the search result in the search results;

training a model using the obtained information, wherein the model is trained to predict a click through rate based on input comprising the one or more presentation bias features and the one or more relevancy features; and

providing the model for use with a search engine, wherein the search engine is configured to provide presentation bias and relevancy features of given search results as input to the model and to use predictive outputs of the model to reduce presentation bias in a presentation of the given search results by determining a quality score for each of the given search results and factoring out independent effects of presentation bias from the quality scores using the predictive outputs of the model, wherein the predictive outputs used to reduce the presentation bias in the presentation of the given search results include a predicted click through rate predicted based on the presentation bias and relevancy features of the given search results and the model.

2. The method of claim 1 wherein the model is a linear model or a regression model.

3. The method of claim 1 wherein the one or more presentation bias features include a position of another search result in the graphical user interface, a title of the search result, a length of the title, a snippet for the search result, bolded text within the snippet, or whether one or more advertisements were presented with the search result.

4. The method of claim 1 wherein the one or more relevancy features include an information retrieval score of the search result, an information retrieval score of another search result returned along with the search result, a language of the query, or a count of words in the query.

5. The method of claim 1 wherein training the model comprises:

training a first model using the obtained presentation bias features to predict a click through rate based on input comprising the one or more presentation bias features; and

training a second model using the obtained relevancy features to predict a click through rate based on input comprising the one or more relevancy features.

6. The method of claim 5 wherein providing the model for use with the search engine comprises providing the first and second models for use with the search engine, wherein the search engine is configured to use predictive outputs of the first and second models to adjust respective ranking scores of the given search results obtained as the search results for the given query to reduce presentation bias in a presentation of the given search results, and wherein the search engine is configured to provide the presentation bias and relevance features of the given search results as input to the first and second models.

7. The method of claim 5 wherein the predicted click through rate for the particular second search result is a ratio of the respective predictive outputs of the first and second models.

8. The method of claim 1 wherein the search engine is further configured to use comparisons of the predictive outputs of the model with an implicit user feedback model to adjust respective ranking scores of the given search results.

9. The method of claim 8 wherein the implicit user feedback model is a click fraction.

10. A system comprising:

data processing apparatus programmed to perform operations comprising;

obtaining information regarding selections of search results provided in response to a plurality of search queries, the obtained information for one or more of the selected search results comprising one or more presentation bias features of a presentation of the search result and one or more relevancy features of the search result, wherein at least one of the presentation bias features is a rank of the search result in the search results;

training a model using the obtained information, wherein the model is trained to predict a click through rate based on input comprising the one or more presentation bias features and the one or more relevancy features; and

providing the model for use with a search engine, wherein the search engine is configured to provide presentation bias and relevancy features of given search results as input to the model and to use predictive outputs of the model to reduce presentation bias in a presentation of the given search results by determining a quality score for each of the given search results and factoring out independent effects of presentation bias from the quality scores using the predictive outputs of the model, wherein the predictive outputs used to reduce the presentation bias in the presentation of the given search results include a predicted click through rate predicted based on the presentation bias and relevancy features of the given search results and the model.

11. The system of claim 10 wherein the model is a linear model or a regression model.

12. The system of claim 10 wherein the one or more presentation bias features include a position of another search result in the graphical user interface, a title of the search result, a length of the title, a snippet for the search result, bolded text within the snippet, or whether one or more advertisements were presented with the search result.

13. The system of claim 10 wherein the one or more relevancy features include an information retrieval score of the search result, an information retrieval score of another search result returned along with the search result, a language of the query, or a count of words in the query.

14. The system of claim 10 wherein training the model comprises:

training a first model using the obtained presentation bias features to predict a click through rate based on input comprising the one or more presentation bias features; and

training a second model using the obtained relevancy features to predict a click through rate based on input comprising the one or more relevancy features.

15. The system of claim 14 wherein providing the model for use with the search engine comprises providing the first and second models for use with the search engine, wherein the search engine is configured to use predictive outputs of the first and second models to adjust respective ranking scores of the given search results obtained as the search results for the given query to reduce presentation bias in a presentation of the given search results, and wherein the search engine is configured to provide the presentation bias and relevance features of the given search results as input to the first and second models.

16. The system of claim 15 wherein the search engine is further configured to use ratios of the predictive outputs of the first and second models to adjust the respective ranking scores of the given search results.

17. The system of claim 10 wherein the search engine is further configured to use comparisons of the predictive outputs of the model with an implicit user feedback model to adjust respective ranking scores of the given search results.

18. The system of claim 17 wherein the implicit user feedback model is a click fraction, and the search engine is configured to use a comparison of the predicted click through rate and the click fraction to adjust the respective ranking scores of the given search results, the comparison including at least one of a ratio of the predicted click through rate and the click fraction or a value difference between the predicted click through rate and the click fraction.

19. A machine-readable storage device having instructions stored thereon that, when executed by data processing apparatus, cause the data processing apparatus to perform operations comprising:

obtaining information regarding selections of search results provided in response to a plurality of search queries, the obtained information for one or more of the selected search results comprising one or more presentation bias features of a presentation of the search result and one or more relevancy features of the search result, wherein at least one of the presentation bias features is a rank of the search result in the search results;

training a model using the obtained information, wherein the model is trained to predict a click through rate based on input comprising the one or more presentation bias features and the one or more relevancy features; and

providing the model for use with a search engine, wherein the search engine is configured to provide presentation bias and relevancy features of given search results as input to the model and to use predictive outputs of the model to reduce presentation bias in a presentation of the given search results by determining a quality score for each of the given search results and factoring out independent effects of presentation bias from the quality scores using the predictive outputs of the model, wherein the predictive outputs used to reduce the presentation bias in the presentation of the given search results include a predicted click through rate predicted based on the presentation bias and relevancy features of the given search results and the model.

20. The machine-readable storage of claim 19 wherein the model is a linear model or a regression model.

21. The machine-readable storage of claim 19 wherein the one or more presentation bias features include a position of another search result in the graphical user interface, a title of the search result, a length of the title, a snippet for the search result, bolded text within the snippet, or whether one or more advertisements were presented with the search result.

22. The machine-readable storage of claim 19 wherein the one or more relevancy features include an information retrieval score of the search result, an information retrieval score of another search result returned along with the search result, a language of the query, or a count of words in the query.

23. The machine-readable storage of claim 19 wherein training the model comprises:

training a first model using the obtained presentation bias features to predict a click through rate based on input comprising the one or more presentation bias features; and

training a second model using the obtained relevancy features to predict a click through rate based on input comprising the one or more relevancy features.

24. The machine-readable storage of claim 23 wherein providing the model for use with the search engine comprises providing the first and second models for use with the search engine, wherein the search engine is configured to use predictive outputs of the first and second models to adjust respective ranking scores of the given search results obtained as the search results for the given query to reduce presentation bias in a presentation of the given search results, and wherein the search engine is configured to provide the presentation bias and relevance features of the given search results as input to the first and second models.

25. The machine-readable storage of claim 24 wherein the search engine is further configured to use ratios of the predictive outputs of the first and second models to adjust the respective ranking scores of the given search results.

26. The machine-readable storage of claim 19 wherein the search engine is further configured to use comparisons of the predictive outputs of the model with an implicit user feedback model to adjust respective ranking scores of the given search results.

27. The machine-readable storage of claim 26 wherein the implicit user feedback model is a click fraction, and the search engine is configured to use a comparison of the predicted click through rate and the click fraction to adjust the respective ranking scores of the given search results, the comparison including at least one of a ratio of the predicted click through rate and the click fraction or a value difference between the predicted click through rate and the click fraction.