Search result ranker

• US 9,501,575 B2
• Filed: 09/25/2015
• Issued: 11/22/2016
• Est. Priority Date: 06/30/2014
• Status: Active Grant

• Current Assignee: YE Hub Armenia LLC
• Original Assignee: Yandex Europe AG (Yandex N.V.)
• Inventors: Gusev, Gleb Gennadievich, Serdyukov, Pavel Viktorovich, Ustinovskiy, Yury Mikhailovich
• Primary Examiner(s): Alam, Shahid
• Application Number:
  US14/866,380
• Publication Number:
  US 20160019219A1
• Time in Patent Office: 424 Days
• Field of Search: 707/723, 707/749, 707/759, 705/348
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

A computerized method for optimizing search result rankings obtained from a search result ranker has the steps of retrieving a first set of query-document pairs, each query-document pair of the first set having an associated post-impression features vector; generating a weight vector having a number of weights corresponding to a number of post-impression features in each of the post-impression feature vector of the first set; generating a target function by using the weight vector and the post-impression features vectors of the first set; using a performance metric associated with the target function, optimizing the weights of the weight vector using the first set of query-document pairs to obtain an optimized target function; optimizing the search result ranker using the optimized target function; and using the optimized search result ranker to rank search results.

Claims

1. A computerized method for optimizing search result rankings obtained from a search result ranker, the search result ranker executing a search result ranking algorithm, the method comprising:

retrieving a first set of query-document pairs, each query-document pair of the first set having an associated post-impression features vector representative of a number of post-impression features associated with a document of the query-document pair;

generating an original weight vector having a number of originally-assigned weights corresponding to the number of post-impression features in each of the post-impression features vector of the first set;

generating a target function by using the weight vector and the post-impression features vectors of the first set, the original weight vector having the number of originally-assigned weights of the first set;

using a performance metric associated with the search ranking algorithm, optimizing the originally-assigned weights of the weight vector using the first set of query-document pairs to obtain an optimized target function, the optimized target function including a number of optimized weights, the optimized weights of the optimized target function representative of an impact of each post-impression feature of the post-impression features vector to a determination of a relevance of the document, the optimizing being executed such that the performance metric of the search ranking algorithm is maximized;

using the optimized target function, generating a relevance label for each query-document pair of the first set;

optimizing the search result ranking algorithm using a relevance label of each query-document pair of the first set generated by the optimized target function; and

using the optimized search result ranking algorithm to rank search results.

2. The method of claim 1, further comprising selecting the performance metric based on features of the target function.

3. The method of claim 1, wherein optimizing the search result ranking algorithm using the relevance label of each query-document pair of the first set generated by the optimized target function includes optimizing the search result ranking algorithm using the relevance label of each query-document pair of a second set of query-document pairs generated by the optimized target function.

4. The method of claim 3, wherein, after optimizing the search result ranking algorithm and before using the search result ranking algorithm, the method further comprises:

evaluating the optimized search result ranking algorithm using a third set of query-document pairs.

5. The method of claim 1, wherein the post-impression features include at least one of at least one document level feature and at least one search engine result page (SERP) level feature.

6. The method of claim 5, wherein the at least one document level feature includes at least one of:

a click feature identifying if a document has been clicked;

a dwell feature identifying a dwell time on a document;

a long dwell feature identifying if the dwell time on the document is greater than a predetermined amount;

a last clicked feature identifying if the document has been clicked last;

a first clicked feature identifying if the document has been clicked first;

a satisfied click feature identifying if the document has been clicked last or if the dwell time is greater than the predetermined amount;

a position feature identifying the original position of the document in the SERP;

a skip feature identifying if the document has been skipped;

a previous skipped feature identifying if another document prior in a ranking to the document has been skipped; and

a skipped above feature identifying a number of documents prior in the ranking to the document have been skipped.

7. The method of claim 5, wherein the at least one SERP level feature includes at least one of:

a top click feature identifying a highest ranking of clicked documents in a ranking;

a bottom click feature identifying a lowest ranking of clicked documents in the ranking;

a number of clicks feature identifying a number of clicks on a SERP;

a top three number of clicks feature identifying a number of clicks on a three highest ranked documents of the SERP;

a number of skips feature identifying a number of documents skipped on the SERP;

a last query feature identifying a last query in a search session; and

an examination time feature identifying a time to a first click on the SERP.

8. The method of claim 1, wherein the post-impression features include at least one of:

a click feature identifying if a document has been clicked;

a dwell feature identifying a dwell time on a document;

a long dwell feature identifying if the dwell time on the document is greater than a predetermined amount;

a last clicked feature identifying if the document has been clicked last;

a first clicked feature identifying if the document has been clicked first;

a satisfied click feature identifying if the document has been clicked last or if the dwell time is greater than the predetermined amount;

a position feature identifying the original position of the document in the SERP;

a skip feature identifying if the document has been skipped;

a previous skipped feature identifying if another document prior in a ranking to the document has been skipped;

a skipped above feature identifying a number of documents prior in the ranking to the document have been skipped;

a top click feature identifying a highest ranking of clicked documents in the ranking;

a bottom click feature identifying a lowest ranking of clicked documents in the ranking;

a number of clicks feature identifying a number of clicks on a SERP;

a top three number of clicks feature identifying a number of clicks on a three highest ranked documents of the SERP;

a number of skips feature identifying a number of documents skipped on the SERP;

a last query feature identifying a last query in a search session; and

an examination time feature identifying a time to a first click on the SERP.

9. The method of claim 1, wherein using the search result ranking algorithm comprises:

receiving, by at least one server from a client device via a communications network, a search query;

effecting, by the at least one server, a search in respect of the search query;

ranking, by the at least one server, documents found by the search using the search result ranking algorithm incorporating the optimized target function;

sending, by the at least one server to the client device via the communications network, instructions to cause a display to a user via a user interface of the client device of a search engine results page (SERP) including links to the documents found by the search, the links being ranked on the SERP according to the ranking of the documents found by the search.

10. The method of claim 1, wherein optimizing the originally-assigned weights of the weight vector includes optimizing the originally-assigned weights of the weight vector using a gradient descent technique.

11. The method of claim 1, wherein the performance metric is one of mean reciprocal rank (MRR) and mean average precision (MAP).

12. A system for optimizing search result rankings obtained from a search result ranker, the search result ranker executing a search result ranking algorithm, the system comprising:

a processor; and

a tangible non-transitory computer readable storage medium communicating with the processor and storing instructions that cause the system to perform the steps of;

retrieving a first set of query-document pairs, each query-document pair of the first set having an associated post-impression features vector representative of a number of post-impression features associated with a document of the query-document pair;

generating an original weight vector having a number of originally-assigned weights corresponding to the number of post-impression features in each of the post-impression features vector of the first set;

generating a target function by using the weight vector and the post-impression features vectors of the first set, the original weight vector having the number of originally-assigned weights of the first set;

using a performance metric associated with the search ranking algorithm, optimizing the originally-assigned weights of the weight vector using the first set of query-document pairs to obtain an optimized target function, the optimized target function including a number of optimized weights, the optimized weights of the optimized target function representative of an impact of each post-impression feature of the post-impression features vector to a determination of a relevance of the document, the optimizing being executed such that the performance metric of the search ranking algorithm is maximized;

using the optimized target function, generating a relevance label for each query-document pair of the first set;

optimizing the search result ranking algorithm using a relevance label of each query-document pair of the first set generated by the optimized target function; and

using the optimized search result ranking algorithm to rank search results.

13. The system of claim 12, wherein the tangible non-transitory computer readable storage medium also stores instructions that cause the system to perform the step of selecting the performance metric based on features of the target function.

14. The system of claim 12, wherein optimizing the search result ranking algorithm using the relevance label of each query-document pair of the first set generated by the optimized target function includes optimizing the search result ranking algorithm using the relevance label of each query-document pair of a second set of query-document pairs generated by the optimized target function.

15. A tangible non-transitory computer readable storage medium storing instructions for optimizing search result rankings obtained from a search result ranker, the search result ranker executing a search result ranking algorithm, the instructions, when executed by a computer, cause the computer to perform operations comprising:

retrieving a first set of query-document pairs, each query-document pair of the first set having an associated post-impression features vector representative of a number of post-impression features associated with a document of the query-document pair;

generating an original weight vector having a number of originally-assigned weights corresponding to the number of post-impression features in each of the post-impression features vector of the first set;

generating a target function by using the weight vector and the post-impression features vectors of the first set, the original weight vector having the number of originally-assigned weights of the first set;

using a performance metric associated with the search ranking algorithm, optimizing the originally-assigned weights of the weight vector using the first set of query-document pairs to obtain an optimized target function, the optimized target function including a number of optimized weights, the optimized weights of the optimized target function representative of an impact of each post-impression feature of the post-impression features vector to a determination of a relevance of the document, the optimizing being executed such that the performance metric of the search ranking algorithm is maximized;

optimizing the search result ranking algorithm using a relevance label of each query-document pair of the first set generated by the optimized target function; and

using the optimized search result ranking algorithm to rank search results.

16. The non-transitory storage medium of claim 15, wherein the operations further comprise selecting the performance metric based on features of the target function.

17. The non-transitory storage medium of claim 15, wherein optimizing the search result ranking algorithm using the relevance label of each query-document pair of the first set generated by the optimized target function includes optimizing the search result ranking algorithm using the relevance label of each query-document pair of a second set of query-document pairs generated by the optimized target function.