Evaluating website properties by partitioning user feedback

• US 8,615,514 B1
• Filed: 02/03/2010
• Issued: 12/24/2013
• Est. Priority Date: 02/03/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Flaster, Michael, Lehman, April R., Fernandes, Neil C.
• Primary Examiner(s): Singh, Amresh
• Application Number:
  US12/699,549
• Time in Patent Office: 1,420 Days
• Field of Search: 707/728
• US Class Current: 707/728
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating website properties by partitioning user feedback data are disclosed. In one aspect, a method includes receiving a plurality of document query pairs (D-Qs) associated with a website, partitioning the plurality of D-Qs into one or more groups according to values for a partition parameter associated with the plurality of D-Qs, evaluating a property parameter of the website based on aggregated user feedback data of the D-Qs included within at least one of the one or more groups, and providing the evaluated property parameter as an input for ranking documents from the website as result documents for searches.

Claims

1. A computer-implemented method, comprising:

obtaining a plurality of document-query pairs associated with a website, each of the document-query pairs being associated with a respective query, a respective document from the website, and respective user feedback data for the respective document as a search result for the respective query;

assigning each of the plurality of document-query pairs to one of a plurality of partitions according to an information retrieval score of a document for the respective query of the pair, each partition being defined by a distinct respective information retrieval score range;

aggregating the respective user feedback data associated with the document-query pairs in two or more of the partitions, wherein at least one of the partitions represents aggregated user feedback data from a plurality of different document-query pairs;

determining a first distribution of the aggregated user feedback data among the two or more partitions;

calculating a skew of the first distribution;

determining an indication of quality of the website based on at least the calculated skew of the first distribution; and

providing the indication of quality of the website as an input for ranking documents from the website as search results.

2. The method of claim 1, wherein each user feedback data includes one or more of a click-count, an impression count, and a click-to-impression ratio.

3. The method of claim 2, wherein the click-count is weighted by click-duration.

4. The method of claim 2, wherein the click-count is weighted by click-recency.

5. The method of claim 1, further comprising:

assigning each of the plurality of document-query pairs to one of a plurality of second partitions according to a length of the respective query of the document-query pair, each second partition being defined by a distinct respective length range.

6. The method of claim 5, wherein determining the indication of quality of the website further comprises:

aggregating the respective user feedback data associated with the document-query pairs in two or more of the second partitions;

determining a distribution of aggregated user feedback data among the two or more second partitions; and

determining the indication of quality of the website based on a combination of the first distribution and the second distribution.

7. The method of claim 1, further comprising:

assigning each of the plurality of document-query pairs to one of a plurality of second partitions according to a measure of popularity of the respective query of the document-query pair, each second partition being defined by a distinct respective measure of popularity range.

8. The method of claim 7, wherein determining the indication of quality of the website further comprises:

aggregating the respective user feedback data associated with the document-query pairs in two or more of the second partitions;

determining a distribution of aggregated user feedback data among the two or more second partitions; and

determining the indication of quality of the website based on a combination of the first distribution and the second distribution.

9. The method of claim 1, further comprising

assigning each of the plurality of document-query pairs to one of a plurality of second partitions according to a measure of commerciality of the respective query of the document-query pair, each second partition being defined by a distinct respective measure of commerciality range.

10. The method of claim 9, wherein determining the indication of quality of the website further comprises:

aggregating the respective user feedback data associated with the document-query pairs in two or more of the second partitions;

determining a distribution of aggregated user feedback data among the two or more second partitions; and

determining the indication of quality of the website based on a combination of the first distribution and the second distribution.

11. The method of claim 1 wherein the score of the document for the respective query of the pair is a score of a top ranked document for the respective query of the pair.

12. A computer-readable storage medium having instructions stored thereon, which, when executed by at least one processor, cause the processor to perform operations comprising:

obtaining a plurality of document-query pairs associated with a website, each of the document-query pairs being associated with a respective query, a respective document from the website, and respective user feedback data for the respective document as a search result for the respective query;

assigning each of the plurality of document-query pairs to one of a plurality of partitions according to an information retrieval score of a document for the respective query of the pair, each partition being defined by a distinct respective information retrieval score range;

aggregating the respective user feedback data associated with the document-query pairs in two or more of the partitions;

determining a first distribution of the aggregated user feedback data among the two or more partitions;

calculating a skew of the first distribution;

determining an indication of quality of the website based on, at least, the calculated skew of the first distribution; and

providing the indication of quality of the website as an input for ranking documents from the website as search results.

13. The computer-readable storage medium of claim 12, wherein each user feedback data includes one or more of a click-count, an impression count, and a click-to-impression ratio.

14. The computer-readable storage medium of claim 13, wherein the click-count is weighted by click-duration.

15. The computer-readable storage medium of claim 13, wherein the click-count is weighted by click-recency.

16. The computer-readable storage medium of claim 13 wherein the score of the document for the respective query of the pair is a score of a top ranked document for the respective query of the pair.

17. The computer-readable storage medium of claim 12, further comprising:

assigning each of the plurality of document-query pairs to one of a plurality of second partitions according to length of the respective query of the document-query pair, each second partition being defined by a distinct respective length range.

18. The computer-readable storage medium of claim 17, wherein determining the indication of quality of the website further comprises:

aggregating the respective user feedback data associated with the document-query pairs in two or more of the second partitions;

determining a distribution of aggregated user feedback data among the two or more second partitions; and

determining the indication of quality of the website based on a combination of the first distribution and the second distribution.

19. The computer-readable storage medium of claim 12, further comprising:

assigning each of the plurality of document-query pairs to one of a plurality of second partitions according to a measure of popularity of the respective query of the document-query pair, each second partition being defined by a distinct respective measure of popularity range.

20. The computer-readable storage medium of claim 19, wherein determining the indication of quality of the website further comprises:

aggregating the respective user feedback data associated with the document-query pairs in two or more of the second partitions;

determining a distribution of aggregated user feedback data among the two or more second partitions; and

determining the indication of quality of the website based on a combination of the first distribution and the second distribution.

21. The computer-readable storage medium of claim 12, further comprising:

assigning each of the plurality of document-query pairs to one of a plurality of second partitions according to a measure of commerciality of the respective query of the document-query pair, each second partition being defined by a distinct respective measure of commerciality range.

22. The computer-readable storage medium of claim 21, wherein determining the indication of quality of the website further comprises:

aggregating the respective user feedback data associated with the document-query pairs in two or more of the second partitions;

determining a distribution of aggregated user feedback data among the two or more second partitions; and

determining the indication of quality of the website based on a combination of the first distribution and the second distribution.

23. A system comprising:

a computer-readable storage medium having instructions stored thereon; and

one or more computers programmed to execute the instructions comprising;

obtaining a plurality of document-query pairs associated with a website, each of the document-query pairs being associated with a respective query, a respective document from the website, and respective user feedback data for the respective document as a search result for the respective query;

assigning each of the plurality of document-query pairs to one of a plurality of partitions according to an information retrieval score of a document for the respective query of the pair, each partition being defined by a distinct respective information retrieval score range;

aggregating the respective user feedback data associated with the document-query pairs in two or more of the partitions;

determining a first distribution of the aggregated user feedback data among the two or more partitions;

calculating a skew of the first distribution;

determining an indication of quality of the website based on, at least, the calculated skew of the first distribution; and

providing the indication of quality of the website as an input for ranking documents from the website as search results.

24. The system of claim 23, wherein each user feedback data includes one or more of a click-count, an impression count, and a click-to-impression ratio.

25. The system of claim 24, wherein the click-count is weighted by click-duration.

26. The system of claim 24, wherein the click-count is weighted by click-recency.

27. The system of claim 24 wherein the score of the document for the respective query of the pair is a score of a top ranked document for the respective query of the pair.

28. The system of claim 23, further comprising:

assigning each of the plurality of document-query pairs to one of a plurality of second partitions according to length of the respective query of the document-query pair, each second partition being defined by a distinct respective length range.

29. The system of claim 28, wherein determining the indication of quality of the website further comprises:

aggregating the respective user feedback data associated with the document-query pairs in two or more of the second partitions;

determining a distribution of aggregated user feedback data among the two or more second partitions; and

determining the indication of quality of the website based on a combination of the first distribution and the second distribution.

30. The system of claim 23, further comprising:

assigning each of the plurality of document-query pairs to one of a plurality of second partitions according to a measure of popularity of the respective query of the document-query pair, each second partition being defined by a distinct respective measure of popularity range.

31. The system of claim 30, wherein determining the indication of quality of the website further comprises:

aggregating the respective user feedback data associated with the document-query pairs in two or more of the second partitions;

determining a distribution of aggregated user feedback data among the two or more second partitions; and

determining the indication of quality of the website based on a combination of the first distribution and the second distribution.

32. The system of claim 23, further comprising:

assigning each of the plurality of document-query pairs to a one of a plurality of second partitions according to a measure of commerciality of the respective query of the document-query pair, each second partition being defined by a distinct respective measure of commerciality range.

33. The system of claim 32, wherein determining the indication of quality of the website further comprises:

aggregating the respective user feedback data associated with the document-query pairs in two or more of the second partitions;

determining a distribution of aggregated user feedback data among the two or more second partitions; and

determining the indication of quality of the website based on a combination of the first distribution and the second distribution.