Modifying search result ranking based on implicit user feedback

• US 9,811,566 B1
• Filed: 01/11/2016
• Issued: 11/07/2017
• Est. Priority Date: 11/02/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kim, Hyung-Jin, Tong, Simon, Shazeer, Noam M., Diligenti, Michelangelo
• Primary Examiner(s): Lu, Kuen
• Application Number:
  US14/992,959
• Time in Patent Office: 666 Days
• Field of Search: 707723
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

The present disclosure includes systems and techniques relating to ranking search results of a search query. In general, the subject matter described in this specification can be embodied in a computer-implemented method that includes determining a measure of relevance for a document result within a context of a search query for which the document result is returned, the determining being based on a first number in relation to a second number, the first number corresponding to longer views of the document result, and the second number corresponding to at least shorter views of the document result; and outputting the measure of relevance to a ranking engine for ranking of search results, including the document result, for a new search corresponding to the search query. The subject matter described in this specification can also be embodied in various corresponding computer program products, apparatus and systems.

Claims

1. A system comprising:

a client system implemented on one or more computers for use by a plurality of users;

wherein the client system comprises;

result selection logs stored on memory devices of the client system; and

a tracking component that is operable to store in the result selection logs information about search result selections made by users of the client system;

wherein the client system is programmed with instructions that are operable, when executed on the client system, to cause the client system to perform operations to;

submit a search query to a search engine that receives search queries from the client system and from other sources of queries;

receive search results responsive to the search query, the search results including either an initial ranking of the search results from the search engine or actual information retrieval scores for the search results from the search engine;

compute, for each of the search results, a respective measure of relevance based on the information stored by the tracking component of the client system about search result selections made by multiple users of the client system and not by other users of the search engine; and

re-rank the search results returned by the search engine based on the respective measures of relevance and provide the re-ranked search results in a response to the search query.

2. The system of claim 1, wherein the client system is made up of computers of an enterprise and a data communication network of the enterprise.

3. The system of claim 2, wherein the computers of the enterprise include personal computers that have a web browser plug-in that incorporates instructions to re-rank locally the search results returned by the search engine.

4. The system of claim 2, wherein the tracking component is a proxy system through which user selections of search results are routed.

5. The system of claim 1, wherein:

the tracking component is operable to store click data on search results provided to users by the client system in the result selections logs stored on the client system.

6. The system of claim 5, wherein the tracking component is operable to store click data that comprises, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (i) the search query corresponding to the user click, and (ii) data identifying the search result corresponding to the user click.

7. The system of claim 6, wherein the tracking component is further operable to store click data that comprises, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (iii) data representing how much time a respective user spent on a corresponding document result.

8. The system of claim 6, wherein the tracking component is further operable to store click data that comprises, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (iv) data representing where in a user interface the respective search result was presented.

9. The system of claim 6, wherein the tracking component is further operable to store click data that comprises, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (v) information retrieval scores of all search results presented in a user interface before the search result corresponding to the user click is presented in the user interface.

10. The system of claim 6, wherein the tracking component is further operable to store click data that comprises, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (vi) titles and snippets of search results presented in a user interface before the search result corresponding to the user click is presented in the user interface.

11. The system of claim 5, wherein the tracking component is operable to store click data that comprises information about user interactions with presentations of search results, including information identifying document results that were presented in the presentations but not clicked.

12. The system of claim 1, wherein the client system is implemented on a single computer.

13. A non-transitory computer-readable medium storing a computer program product that, when executed by a client system for use by a plurality of users, causes the client system to perform operations comprising:

storing in result selection logs of the client system, by a tracking component, information about search result selections made by multiple users of the client system;

submitting a search query to a search engine that receives search queries from the client system and from other sources of queries;

receiving search results responsive to the search query, the search results including either an initial ranking of the search results from the search engine or actual information retrieval scores for the search results from the search engine;

computing, for each of the search results received from the search engine, a respective measure of relevance based on the information stored by the tracking component of the client system about search result selections made by multiple users of the client system and not by other users of the search engine; and

re-ranking the search results returned by the search engine based on the respective measure of relevance from the search result selections stored in the result selection logs of the client system and providing the re-ranked search results in response to the search query.

14. The non-transitory computer-readable medium of claim 13, wherein the operations comprise:

storing in the result selection logs of the client system, by the tracking component, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (i) the search query corresponding to the user click, and (ii) data identifying the search result corresponding to the user click.

15. The non-transitory computer-readable medium of claim 14, wherein the operations comprise:

storing in the result selection logs of the client system, by the tracking component, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (iii) data representing how much time a respective user spent on a corresponding document result.

16. The non-transitory computer-readable medium of claim 14, wherein the operations comprise:

storing in the result selection logs of the client system, by the tracking component, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (iv) data representing where in a user interface the search result was presented.

17. The non-transitory computer-readable medium of claim 14, wherein the operations comprise:

storing in the result selection logs of the client system, by the tracking component, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (v) information retrieval scores of all search results presented in a user interface before the search result corresponding to the user click is presented in the user interface.

18. The non-transitory computer-readable medium of claim 14, wherein the operations comprise:

storing in the result selection logs of the client system, by the tracking component, for each user click made by any of the multiple users of the client system and not by other users of the search engine on a search result presented in response to a search query, (vi) titles and snippets of search results presented in a user interface before the search result corresponding to the user click is presented in the user interface.

19. The non-transitory computer-readable medium of claim 13, wherein the operations comprise:

storing in the result selection logs of the client system, by the tracking component, click data that comprises information about user interactions with presentations of search results by any of the multiple users of the client system and not by other users of the search engine, including information identifying document results that were presented in the presentations but not clicked.

20. A computer-implemented method comprising:

storing, by a client system for use by a plurality of users, in result selection logs stored on memory devices of the client system by a tracking component of the client system, information about search result selections made by multiple users of the client system;

submitting, by the client system, a search query received to a search engine that receives search queries from the client system and from other sources of queries;

receiving, by the client system, search results responsive to the search query, the search results including either an initial ranking of the search results from the search engine or actual information retrieval scores for the search results from the search engine;

computing, by the client system, for each of the search results received from the search engine, a respective measure of relevance based on the information stored by the tracking component of the client system about search result selections made by multiple users of the client system and not by other users of the search engine; and

re-ranking, by the client system, the search results returned by the search engine based on the respective measure of relevance from the search result selections stored in the result selection logs of the client system and providing the re-ranked search results in a response to the search query.