System and method for development of search success metrics

• US 8,024,336 B2
• Filed: 06/09/2009
• Issued: 09/20/2011
• Est. Priority Date: 06/09/2009
• Status: Expired due to Fees

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Wai, Lawrence
• Primary Examiner(s): Mofiz; Apu
• Assistant Examiner(s): LE, HUNG D
• Application Number:
  US12/481,196
• Publication Number:
  US 20100312786A1
• Time in Patent Office: 833 Days
• Field of Search: None
• US Class Current: 707/731
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

A system and method for development of search success metrics. A plurality of search engine result pages are collected and a target page success metric is determined for each page. A plurality of machine learned page success metrics are trained using a first subset of the search engine result pages and each result page'"'"'s respective target page success metric, wherein each of the machine learned page success metrics is trained to predict the target page success metric for each of the first subset of search engine result pages. A predicted target page success metric is predicted for each of a second subset of the search engine result pages using each of the machine learned page success metrics. The accuracy of each of the machine learned page success metrics in predicting the target page success metric associated with each of the second subset of search engine result pages is then evaluated.

Claims

1. A method comprising the steps of:

collecting, over a network, a plurality of search engine result pages;

determining, using at least one computing device, a target page success metric for each of the plurality of search engine result pages;

training, using the at least one computing device, a plurality of machine learned page success metrics using a first subset of the plurality of search engine result pages and each result page'"'"'s respective target page success metric, wherein each of the machine learned page success metrics is trained to predict the target page success metric for each of the first subset of the plurality of search engine result pages;

predicting, using the at least one computing device, using each of the plurality of machine learned page success metrics, a predicted target page success metric for each of a second subset of the search engine result pages;

evaluating, using the at least one computing device, the accuracy of each of the plurality of machine learned page success metrics in predicting the target page success metric associated with each of the second subset of search engine result pages; and

outputting, using the at least one computing device, a representation of the accuracy of the of each of the plurality of machine learned page success metrics to a display medium.

2. The method of claim 1 wherein each of the search engine result pages is associated with at least one session variable and the target page success metric for each of the search engine result pages is determined using the at least one session variable associated with the respective result page.

3. The method of claim 2 wherein the at least one session variable is a click through rate.

4. The method of claim 3 wherein the target success metric for each of the search result pages is expressed as 1 if a user clicked on at least one result on the respective search result page, and 0 if otherwise.

5. The method of claim 2 wherein the at least one session variable is a dwell time, wherein the dwell time represents a difference in timestamps between actions on each search result page.

6. The method of claim 1 wherein each of the search engine result pages is associated with at least one editorial variable and the target page success metric for each of the search engine result pages is determined using the at least editorial session variable associated with the respective result page.

7. The method of claim 1 wherein each of the search engine result pages is associated with at least one session variable and at least one editorial variable and the target page success metric for each of the search engine result pages is determined using the at least one session variable and at least one editorial variable associated with the respective result page.

8. The method of claim 1 wherein the first subset of search result pages and the second subset of search result pages are selected such that each page within each subset is a result page produced in response to a navigational query.

9. The method of claim 1 wherein the first subset of search result pages and the second subset of search result pages are selected such that each page within each subset is a result page produced in response to a non-navigational query.

10. The method of claim 1 wherein at least one of the plurality machine learned page success metrics is an editorial page success metric.

11. The method of claim 10 wherein the editorial page success metric is a discounted cumulative gain metric defined as follows:

discounted cumulative gain=Σ

r w_r wd_r

where Σ

r is a set of search result pages r

w_r=1/log 2(rank+1), rank=rank of result

wd_r=relevance weight for result at rank r.

12. The method of claim 11 wherein the relevance weight for result at rank r are defined as follow:

wd(perfect)=10, wd(excellent)=7, wd(good)=3, wd(fair)=0.5, wd(bad)=0.

13. The method of claim 10 wherein the editorial page success metric is a best discounted cumulative gain metric defined as follows

best discounted cumulative gain=max{w_r wd_r}.

14. The method of claim 13 wherein the automated page success metric is a decision tree regression based on query search engine results page timestamps.

15. The method of claim 10 wherein the editorial page success metric is a generalized discounted cumulative gain metric.

16. The method of claim 15 wherein the generalized discounted cumulative gain metric is determined as follows,

for a set of search result pages {k}

where wr=rank weight for result at rank r

wdr=relevance weight for result at rank r

the generalized cumulative gain metric (GDCG) is a function of wr and wdr wherein the values of wr and wdr are varied such that the function Λ

is minimized, where Λ

=2 k (GDCGk−

Sk)2, where k is the kth page view and S is the target page success metric.

17. The method of claim 1 wherein at least one of the plurality of machine learned page success metrics is an automated page success metric.

18. The method of claim 1 wherein at least one of the plurality machine learned page success metrics is a hybrid page success metric.

19. The method of claim 1 wherein the accuracy of each of the plurality of plurality of machine learned page success metrics is determined by calculating an error parameter, sigma, where sigma is defined as sqrt({tilde over (Λ

)}N), where Λ

=Σ

k (PSMk−

Sk)2, where N is the number of page views in the second subset of search engine result pages, k is the kth page view in the second subset of search engine result pages, and S is the target page success metric associated with the kth page view of the in the second subset of search engine result pages.

20. A system comprising:

a search engine result pages collection module comprising one or more processors programmed to execute software code retrieved from a computer readable storage medium storing software for collecting, over a network, a plurality of search engine result pages;

a target page success metrics determination module comprising one or more processors programmed to execute software code retrieved from a computer readable storage medium storing software for determining a target page success metric for each of the plurality of search engine result pages;

a machine learned page success metrics training comprising one or more processors programmed to execute software code retrieved from a computer readable storage medium storing software for training a plurality of machine learned page success metrics using a first subset of the plurality of search engine result pages and each result page'"'"'s respective target page success metric, wherein each of the machine learned page success metrics is trained to predict the target page success metric for each of the first subset of the plurality of search engine result pages;

a target page success prediction module comprising one or more processors programmed to execute software code retrieved from a computer readable storage medium storing software for predicting, using each of the plurality of machine learned page success metrics, a predicted target page success metric for each of a second subset of the search engine result pages; and

a page success metrics evaluation module comprising one or more processors programmed to execute software code retrieved from a computer readable storage medium storing software for evaluating the accuracy of each of the plurality of plurality of machine learned page success metrics in predicting the target page success metric associated with each of the second subset of search engine result pages.

21. The system of claim 20 wherein each of the search engine result pages collected by the search engine result pages collection module is associated with at least one session variable and the target page success metrics determination module determines target page success metric for each of the search engine result pages using the at least one session variable associated with the respective result page.

22. The system of claim 20 wherein each of the search engine result pages collected by the search engine result pages collection module is associated with at least one editorial variable and the target page success metrics determination module determines target page success metric for each of the search engine result pages using the at least one editorial variable associated with the respective result page.

23. The system of claim 20 wherein each of the search engine result pages collected by the search engine result pages collection module is associated with at least one session variable and at least one editorial variable and the target page success metrics determination module determines the target page success metric for each of the search engine result pages using the at least one session variable and at least one editorial variable associated with the respective result page.

24. The system of claim 20 wherein the first subset of search result pages and the second subset of search result pages are selected such that each page within each subset is a result page produced in response to a navigational query.

25. The system of claim 20 wherein the first subset of search result pages and the second subset of search result pages are selected such that each page within each subset is a result page produced in response to a non-navigational query.

26. The system of claim 20 wherein at least one of the plurality machine learned page success metrics is an editorial page success metric.

27. The system of claim 20 wherein at least one of the plurality of machine learned ranking optimization metrics is an automated page success metric.

28. The system of claim 20 wherein at least one of the plurality of machine learned ranking optimization metrics is an hybrid page success metric.

29. The system of claim 20 wherein the page success metrics evaluation module additionally comprises software code retrieved from a computer readable storage medium storing software for outputting a representation of the accuracy of the of each of the plurality of machine learned page success metrics to a display medium.

30. A non-transitory computer-readable storage medium having computer-executable instructions comprising the steps of:

collecting, over a network, a plurality of search engine result pages;

determining, using at least one computing device, a target page success metric for each of the plurality of search engine result pages;

training, using the at least one computing device, a plurality of machine learned page success metrics using a first subset of the plurality of search engine result pages and each result page'"'"'s respective target page success metric, wherein each of the machine learned page success metrics is trained to predict the target page success metric for each of the first subset of the plurality of search engine result pages;

predicting, using the at least one computing device, using each of the plurality of machine learned page success metrics, a predicted target page success metric for each of a second subset of the search engine result pages;

evaluating, using the at least one computing device, the accuracy of each of the plurality of machine learned page success metrics in predicting the target page success metric associated with each of the second subset of search engine result pages; and

outputting, using the at least one computing device, a representation of the accuracy of each of the plurality of machine learned page success metrics to a display medium.