Incorporating recency in network search using machine learning

• US 8,886,641 B2
• Filed: 10/15/2009
• Issued: 11/11/2014
• Est. Priority Date: 10/15/2009
• Status: Expired due to Fees

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Dong, Anlei, Chang, Yi, Zhang, Ruiqiang, Zheng, Zhaohui, Mishne, Gilad Avraham, Bai, Jing, Buchner, Karolina Barbara, Liao, Ciya, Ji, Shihao, Leung, Gilbert, Dupret, Georges-Eric Albert Marie Robert, Liu, Ling
• Primary Examiner(s): PANNALA, SATHYANARAYA R
• Application Number:
  US12/579,855
• Publication Number:
  US 20110093459A1
• Time in Patent Office: 1,853 Days
• Field of Search: 707/999.004, 707/736, 707/721, 707/768, 707/725, 715/738
• US Class Current: 707/725
• CPC Class Codes: G06F 16/9532 Query formulation
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

In one embodiment, access a set of recency ranking data comprising one or more recency search queries and one or more recency search results, each of the recency search queries being recency-sensitive with respect to a particular time period and being associated with a query timestamp representing the time at which the recency search query is received at a search engine, each of the recency search results being generated by the search engine for one of the recency search queries and comprising one or more recency network resources. Construct a plurality of recency features from the set of recency ranking data. Train a first ranking model via machine learning using at least the recency features.

Claims

1. A method, implemented on at least one machine having at least one processor, storage, and a communication platform connected to a network for ranking a search result, comprising:

accessing, by the at least one machine, a set of recency ranking data comprising one or more past search queries, one or more past search results, and one or more recency features, wherein the recency features comprise;

a time-sensitive feature representing a particular time period that was determined based on content of at least some of the past search queries, wherein the at least some of the past search queries were sensitive to the particular time period, and

a query timestamp feature representing the time at which the past search queries were received at a search engine;

training, by the at least one machine, a first ranking model via machine learning based on the recency features; and

determining when recency is to be utilized for ranking a search result based on the ranking model.

2. The method of claim 1, further comprising:

accessing, by the at least one machine, a plurality of first search queries and a plurality of first search results, each of the first search results being generated by the search engine for one of the first search queries and comprising one or more first network resources; and

for each of the first search queries, by the at least one machine,

automatically determining whether the first search query is sensitive with respect to a first time period during which the first search query is received at the search engine; and

if the first search query is sensitive, then including the first search query and the first search result generated for the first search query in the set of recency ranking data.

3. The method of claim 2, wherein automatically determining whether a current search query is sensitive with respect to a current time period during which the current search query is received at the search engine comprises:

defining one or more reference time periods for the current time period, each of the reference time periods being prior to the current time period;

constructing one or more current language models for the current time period, each of the current language models representing a particular type of search-related activity occurring during the current time period;

constructing one or more reference language models for each of the reference time periods, each of the reference language models corresponding to one of the current language models and representing the particular type of search-related activity represented by the corresponding current language model occurring during the corresponding reference time period;

computing a score for the current search query that represents a difference between the current search query'"'"'s affinity to the current language models for the current time period and the current search query'"'"'s affinity to the reference language models for each of the reference time periods; and

if the score satisfies a predetermined threshold requirement, then classifying the current search query as sensitive with respect to the current time period.

4. The method of claim 1, wherein

the recency features comprise:

one or more resource timestamp features each of which representing a timestamp extracted from one of the recency network resources;

one or more resource age features each of which representing, for one of the recency query-resource pairs, a difference between the query timestamp of the recency query and the resource timestamp of the recency resource;

one or more resource link-time features each of which representing a time when a link to one of the recency network resources is discovered;

one or more web-buzz features each of which representing one of the recency network resources that is popular during one particular time period; and

one or more click-buzz features each of which representing one of the recency network resources that receives a higher amount of clicks during one particular time period.

5. The method of claim 1, wherein:

each of the recency features is with respect to a recency query-resource pair comprising one of the past search queries and one of recency network resources identified for the one past search query; and

constructing the recency features comprises;

for each of the recency query-resource pairs,

assigning, by a human judge, a grade to the recency query-resource pair based on a level of relevance that the recency network resource has with respect to the recency search query; and

adjusting, by the human judge, the grade based on the recency of the recency network resource with respect to the recency search query.

6. The method of claim 1, wherein training the first ranking model via machine learning comprises training the first ranking model using only the recency features.

7. The method of claim 1, wherein training the first ranking model via machine learning comprises training the first ranking model using the recency features and a plurality of relevance features.

8. The method of claim 7, wherein training the first ranking model using the recency features and the relevance features comprises:

for each of the past search queries, determining a ranking score for each of the recency network resources comprised in the past search result generated for the past search query using a second ranking model that has been trained without any recency feature;

constructing the relevance features from the ranking scores; and

applying the recency features and the relevance features to the first ranking model via machine learning.

9. The method of claim 7, wherein training the first ranking model using the recency features and the relevance features comprises:

accessing a set of relevance ranking data comprising one or more search queries and one or more search results, each of the search results being generated for one of the search queries and comprising one or more network resources;

constructing the relevance features from the set of relevance ranking data;

determining a first weight for the recency features and a second weight for the relevance features, the first weight indicating a first level of influence the recency features have on training the first ranking model, the second weight indicating a second level of influence the relevance features have on training the first ranking model; and

applying the recency features according to the first weight and the relevance features according to the second weight to the first ranking model via machine learning.

10. The method of claim 7, wherein training the first ranking model using the recency features and the relevance features comprises:

adjusting a second ranking model that has been trained with the relevance features to incorporate the recency features.

11. The method of claim 1, further comprising:

accessing, by the at least one machine, a second search query received at the search engine;

identifying, by the at least one machine, a plurality of second network resources for the second search query;

automatically determining, by the at least one machine, whether the second search query is sensitive with respect to a second time period during which the second search query is received at the search engine;

if the second search query is sensitive, then ranking, by the at least one machine, the second network resources using the first ranking model that has been trained with at least the recency features; and

if the second search query is not sensitive, then ranking, by the at least one machine, the second network resources using a second ranking model that has been trained without any recency feature.

12. The method of claim 1, wherein each of the recency features is with respect to a recency query-resource pair comprising one of the past search queries and one of recency network resources identified for the one past search query.

13. A system for ranking a search result, comprising:

a memory comprising instructions executable by one or more processors; and

one or more processors coupled to the memory and operable to execute the instructions, the one or more processors being operable when executing the instructions to;

access a set of recency ranking data comprising one or more past search queries, one or more past search results, and one or more recency features, wherein the recency features comprise;

a time-sensitive feature representing a particular time period that was determined based on content of at least some of the past search queries, wherein the at least some of the past search queries were sensitive to the particular time period, and

a query timestamp feature representing the time at which the past search queries were received at a search engine;

train a first ranking model via machine learning based on recency features; and

determine when recency is to be utilized for ranking a search result based on the ranking model.

14. The system of claim 13, wherein the one or more processors are further operable when executing the instructions to:

access a plurality of first search queries and a plurality of first search results, each of the first search results being generated by the search engine for one of the first search queries and comprising one or more first network resources; and

for each of the first search queries

automatically determine whether the first search query is sensitive with respect to a first time period during which the first search query is received at the search engine; and

if the first search query is sensitive, then include the first search query and the first search result generated for the first search query in the set of recency ranking data.

15. The system of claim 14, wherein to automatically determine whether a current search query is sensitive with respect to a current time period during which the current search query is received at the search engine comprises:

define one or more reference time periods for the current time period, each of the reference time periods being prior to the current time period;

construct one or more current language models for the current time period, each of the current language models representing a particular type of search-related activity occurring during the current time period;

construct one or more reference language models for each of the reference time periods, each of the reference language models corresponding to one of the current language models and representing the particular type of search-related activity represented by the corresponding current language model occurring during the corresponding reference time period;

compute a score for the current search query that represents a difference between the current search query'"'"'s affinity to the current language models for the current time period and the current search query'"'"'s affinity to the reference language models for each of the reference time periods; and

if the score satisfies a predetermined threshold requirement, then classify the current search query as sensitive with respect to the current time period.

16. The system of claim 13, wherein

the recency features comprise:

one or more resource timestamp features each of which representing a timestamp extracted from one of the recency network resources;

one or more resource age features each of which representing, for one of the recency query-resource pairs, a difference between the query timestamp of the recency query and the resource timestamp of the recency resource;

one or more resource link-time features each of which representing a time when a link to one of the recency network resources is discovered;

one or more web-buzz features each of which representing one of the recency network resources that is popular during one particular time period; and

one or more click-buzz features each of which representing one of the recency network resources that receives a higher amount of clicks during one particular time period.

17. The system of claim 13, wherein:

each of the recency features is with respect to a recency query-resource pair comprising one of the past search queries and one of recency network resources identified for the one past search query; and

to construct the recency features comprises;

for each of the recency query-resource pairs,

assign, by a human judge, a grade to the recency query-resource pair based on a level of relevance that the recency network resource has with respect to the recency search query; and

adjust, by the human judge, the grade based on the recency of the recency network resource with respect to the recency search query.

18. The system of claim 13, wherein to train the first ranking model via machine learning comprises train the first ranking model using only the recency features.

19. The system of claim 13, wherein to train the first ranking model via machine learning comprises train the first ranking model using the recency features and a plurality of relevance features.

20. The system of claim 19, wherein to train the first ranking model using the recency features and the relevance features comprises:

for each of the past search queries, determine a ranking score for each of the recency network resources comprised in the past search result generated for the past search query using a second ranking model that has been trained without any recency feature;

construct the relevance features from the ranking scores; and

apply the recency features and the relevance features to the first ranking model via machine learning.

21. The system of claim 19, wherein to train the first ranking model using the recency features and the relevance features comprises:

access a set of relevance ranking data comprising one or more search queries and one or more search results, each of the search results being generated for one of the search queries and comprising one or more network resources;

construct the relevance features from the set of relevance ranking data;

determine a first weight for the recency features and a second weight for the relevance features, the first weight indicating a first level of influence the recency features have on training the first ranking model, the second weight indicating a second level of influence the relevance features have on training the first ranking model; and

apply the recency features according to the first weight and the relevance features according to the second weight to the first ranking model via machine learning.

22. The system of claim 19, wherein to train the first ranking model using the recency features and the relevance features comprises:

adjust a second ranking model that has been trained with the relevance features to incorporate the recency features.

23. The system of claim 13, wherein the one or more processors are further operable when executing the instructions to:

access a second search query received at the search engine;

identify a plurality of second network resources for the second search query;

automatically determine whether the second search query is sensitive with respect to a second time period during which the second search query is received at the search engine;

if the second search query is sensitive, then rank the second network resources using the first ranking model that has been trained with at least the recency features; and

if the second search query is not sensitive, then rank the second network resources using a second ranking model that has been trained without any recency feature.

24. One or more machine-readable tangible and non-transitory media having information for ranking a search result, wherein the information, when read by at least one machine, causes the at least one machine to:

access a set of recency ranking data comprising one or more past search queries, one or more past search results, and one or more recency features, wherein the recency features comprise;

a time-sensitive feature representing a particular time period that was determined based on content of at least some of the past search queries, wherein the at least some of the past search queries were sensitive to the particular time period, and

a query timestamp feature representing the time at which the past search queries were received at a search engine;

train a first ranking model via machine learning based on the recency features; and

determine when recency is to be utilized for ranking a search result based on the ranking model.

25. The media of claim 24, wherein the software is further operable when executed by the at least one machine to:

access a plurality of first search queries and a plurality of first search results, each of the first search results being generated by the search engine for one of the first search queries and comprising one or more first network resources; and

for each of the first search queries

automatically determine whether the first search query is sensitive with respect to a first time period during which the first search query is received at the search engine; and

if the first search query is sensitive, then include the first search query and the first search result generated for the first search query in the set of recency ranking data.

26. The media of claim 25, wherein to automatically determine whether a current search query is sensitive with respect to a current time period during which the current search query is received at the search engine comprises:

define one or more reference time periods for the current time period, each of the reference time periods being prior to the current time period;

construct one or more current language models for the current time period, each of the current language models representing a particular type of search-related activity occurring during the current time period;

construct one or more reference language models for each of the reference time periods, each of the reference language models corresponding to one of the current language models and representing the particular type of search-related activity represented by the corresponding current language model occurring during the corresponding reference time period;

compute a score for the current search query that represents a difference between the current search query'"'"'s affinity to the current language models for the current time period and the current search query'"'"'s affinity to the reference language models for each of the reference time periods; and

if the score satisfies a predetermined threshold requirement, then classify the current search query as sensitive with respect to the current time period.

27. The media of claim 24, wherein

the recency features comprise:

one or more resource timestamp features each of which representing a timestamp extracted from one of the recency network resources;

one or more resource age features each of which representing, for one of the recency query-resource pairs, a difference between the query timestamp of the recency query and the resource timestamp of the recency resource;

one or more resource link-time features each of which representing a time when a link to one of the recency network resources is discovered;

one or more web-buzz features each of which representing one of the recency network resources that is popular during one particular time period; and

one or more click-buzz features each of which representing one of the recency network resources that receives a higher amount of clicks during one particular time period.

28. The media of claim 24, wherein:

each of the recency features is with respect to a recency query-resource pair comprising one of the past search queries and one of recency network resources identified for the one past search query; and

to construct the recency features comprises;

for each of the recency query-resource pairs,

assign, by a human judge, a grade to the recency query-resource pair based on a level of relevance that the recency network resource has with respect to the recency search query; and

adjust, by the human judge, the grade based on the recency of the recency network resource with respect to the recency search query.

29. The media of claim 24, wherein to train the first ranking model via machine learning comprises train the first ranking model using only the recency features.

30. The media of claim 24, wherein to train the first ranking model via machine learning comprises train the first ranking model using the recency features and a plurality of relevance features.

31. The media of claim 30, wherein to train the first ranking model using the recency features and the relevance features comprises:

for each of the past search queries, determine a ranking score for each of the recency network resources comprised in the past search result generated for the past search query using a second ranking model that has been trained without any recency feature;

construct the relevance features from the ranking scores; and

apply the recency features and the relevance features to the first ranking model via machine learning.

32. The media of claim 30, wherein to train the first ranking model using the recency features and the relevance features comprises:

access a set of relevance ranking data comprising one or more search queries and one or more search results, each of the search results being generated for one of the search queries and comprising one or more network resources;

construct the relevance features from the set of relevance ranking data;

determine a first weight for the recency features and a second weight for the relevance features, the first weight indicating a first level of influence the recency features have on training the first ranking model, the second weight indicating a second level of influence the relevance features have on training the first ranking model; and

apply the recency features according to the first weight and the relevance features according to the second weight to the first ranking model via machine learning.

33. The media of claim 30, wherein to train the first ranking model using the recency features and the relevance features comprises:

adjust a second ranking model that has been trained with the relevance features to incorporate the recency features.

34. The media of claim 24, wherein the software is further operable when executed by the at least one machine to:

access a second search query received at the search engine;

identify a plurality of second network resources for the second search query;

automatically determine whether the second search query is sensitive with respect to a second time period during which the second search query is received at the search engine;

if the second search query is sensitive, then rank the second network resources using the first ranking model that has been trained with at least the recency features; and

if the second search query is not sensitive, then rank the second network resources using a second ranking model that has been trained without any recency feature.