Modifying scoring data based on historical changes

• US 8,874,555 B1
• Filed: 11/20/2009
• Issued: 10/28/2014
• Est. Priority Date: 11/20/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kim, Hyung-Jin, Lopatenko, Andrei
• Primary Examiner(s): LU, KUEN S
• Application Number:
  US12/623,276
• Time in Patent Office: 1,803 Days
• Field of Search: 707/723, 707/803
• US Class Current: 707/723
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for modifying historical data. One method includes calculating time trend statistics for a document and a query during different time periods. The method further includes modifying a quality of result statistic for the document as a search result for the query by a factor based on the one or more time trend statistics. The method further includes providing the modified quality of result statistic as an input to a document ranking process. Another method includes calculating a difference score for statistics for a group of documents and a query for a first time period and a second time period. The method further includes modifying quality of result statistics for documents responsive to the query based on the difference score. The method further includes providing the modified statistics as an input to a document ranking process.

Claims

1. A computer-implemented method comprising:

calculating one or more time trend statistics for a plurality of quality of result statistics for a first document as a search result for a first query, each of the quality of result statistics corresponding to a different time period, the one or more time trend statistics estimating changes in the quality of result statistics over time, wherein each of the one or more time trend statistics comprises a quality of result difference between a first quality of result statistic for the first document as a search result for the first query during a first time period and a second quality of result statistic for the first document as a search result for the first query during a second time period;

generating a first modified quality of result statistic by modifying a first quality of result statistic for the first document as a search result for the first query by a factor, where the factor is based on the one or more time trend statistics; and

providing the first modified quality of result statistic as an input to a document ranking process for the first document and the first query.

2. The method of claim 1, further comprising verifying that the quality of result difference satisfies a statistically significant threshold before generating the first modified quality of result statistic.

3. The method of claim 2, wherein the factor is further based on how statistically significant the quality of result difference is.

4. The method of claim 1, further comprising:

generating a second modified quality of result statistic for the first document and a second different query according to the one or more time trend statistics; and

providing the second modified quality of result statistic as an input to the document ranking process for the first document and the second query.

5. The method of claim 1, further comprising determining that a first past version of the first document during the first time period is different from a second past version of the first document during the second time period before modifying the first quality of result statistic.

6. The method of claim 5, wherein determining that the first past version of the first document is different from the second past version of the first document comprises calculating a similarity score that is a measure of similarities between shingles extracted from the first past version of the first document and shingles extracted from the second past version of the first document, and determining that the similarity score satisfies a threshold.

7. The method of claim 1, further comprising:

identifying a third query as related to the first document based on a quality of result statistic for the third query and the first document;

generating a third modified quality of result statistic for the third query and a second document by modifying a quality of result statistic for the third query and the second document by a factor, where the factor is based on the one or more time trend statistics; and

providing the third modified quality of result statistic as an input to the document ranking process for the second document and the third query.

8. The method of claim 1, wherein each quality of result statistic is an estimate of a respective percentage of users that found the first document relevant to the first query out of a first total number of users who viewed the first document as a search result for the first query during a respective time period.

9. A computer-implemented method comprising:

calculating a difference score for first quality of result statistics for a group of documents as search results for a first query corresponding to a first time period and second quality of result statistics for the group of documents as search results for the first query corresponding to a different second time period, wherein a quality of result statistic for a particular document in the group of documents and for a particular time period is an estimate of a respective portion of user interactions with the particular document as a search result for the first query during the particular time period out of a total number of user interactions with documents as search results for the first query during the particular time period;

identifying one or more documents that are responsive to the first query and modifying a respective quality of result statistic for each identified document based on the difference score, resulting in a respective modified quality of result statistic for each identified document; and

providing the respective modified quality of result statistic for one of the identified documents as an input to the document ranking process for the identified documents and the first query.

10. A system comprising:

one or more computers, programmed to perform operations comprising;

calculating one or more time trend statistics for a plurality of quality of result statistics for a first document as a search result for a first query, each of the quality of result statistics corresponding to a different time period, the one or more time trend statistics estimating changes in the quality of result statistics over time, wherein each of the one or more time trend statistics include a quality of result difference between a first quality of result statistic for a first document as a search result for a first query during a first time period and a second quality of result statistic for the first document as a search result the first query during a second time period;

generating a first modified quality of result statistic by modifying a first quality of result statistic for the first document as a search result for the first query by a factor, where the factor is based on the one or more time trend statistics; and

providing the first modified quality of result statistic as an input to a document ranking process for the first document and the first query.

11. The system of claim 10, further programmed to perform operations comprising verifying that the quality of result difference satisfies a statistically significant threshold before generating the first modified quality of result statistic.

12. The system of claim 11, wherein the factor is further based on how statistically significant the quality of result difference is.

13. The system of claim 10, further programmed to perform operations comprising:

generating a second modified quality of result statistic for the first document and a second different query according to the one or more time trend statistics; and

providing the second modified quality of result statistic as an input to the document ranking process for the first document and the second query.

14. The system of claim 10, further programmed to perform operations comprising determining that a first past version of the first document during the first time period is different from a second past version of the first document during the second time period before modifying the first quality of result statistic.

15. The system of claim 14, wherein determining that the first past version of the first document is different from the second past version of the first document comprises calculating a similarity score that is a measure of similarities between shingles extracted from the first past version of the first document and shingles extracted from the second past version of the first document, and determining that the similarity score satisfies a threshold.

16. The system of claim 10, further programmed to perform operations comprising:

identifying a third query as related to the first document based on a quality of result statistic for the third query and the first document;

generating a third modified quality of result statistic for the third query and a second document by modifying a quality of result statistic for the third query and the second document by a factor, where the factor is based on the one or more time trend statistics; and

providing the third modified quality of result statistic as an input to the document ranking process for the second document and the third query.

17. The system of claim 10, wherein each quality of result statistic is an estimate of a respective percentage of users that found the first document relevant to the first query out of a first total number of users who viewed the first document as a search result for the first query during a respective time period.

18. A system comprising:

one or more computers programmed to perform operations comprising;

calculating a difference score for first quality of result statistics for a group of documents as search results for a first query corresponding to a first time period and second quality of result statistics for the group of documents as search results for the first query corresponding to a different second time period, wherein a quality of result statistic for a particular document in the group of documents and for a particular time period is an estimate of a respective portion of user interactions with the particular document as a search result for the first query during the particular time period out of a total number of user interactions with documents as search results for the first query during the particular time period;

identifying one or more documents that are responsive to the first query and modifying a respective quality of result statistic for each identified document based on the difference score, resulting in a respective modified quality of result statistic for each identified document; and

providing the respective modified quality of result statistic for one of the identified documents as an input to the document ranking process for the identified documents and the first query.

19. A storage device having instructions stored thereon that, when executed by data processing apparatus, cause the data processing apparatus to perform actions comprising:

calculating one or more time trend statistics for a plurality of quality of result statistics for a first document as a search result for a first query, each of the quality of result statistics corresponding to a different time period, the one or more time trend statistics estimating changes in the quality of result statistics over time, wherein each of the one or more time trend statistics comprises a quality of result difference between a first quality of result statistic for the first document as a search result for the first query during a first time period and a second quality of result statistic for the first document as a search result for the first query during a second time period;

generating a first modified quality of result statistic by modifying a first quality of result statistic for the first document as a search result for the first query by a factor, where the factor is based on the one or more time trend statistics; and

providing the first modified quality of result statistic as an input to a document ranking process for the first document and the first query.

20. The storage device of claim 19 wherein the operations further comprise verifying that the quality of result difference satisfies a statistically significant threshold before generating the first modified quality of result statistic.

21. The storage device of claim 20 wherein the factor is further based on how statistically significant the quality of result difference is.

22. The storage device of claim 19 wherein the operations further comprise:

generating a second modified quality of result statistic for the first document and a second different query according to the one or more time trend statistics; and

providing the second modified quality of result statistic as an input to the document ranking process for the first document and the second query.

23. The storage device of claim 19 wherein the operations further comprise determining that a first past version of the first document during the first time period is different from a second past version of the first document during the second time period before modifying the first quality of result statistic.

24. The storage device of claim 23 wherein determining that the first past version of the first document is different from the second past version of the first document comprises calculating a similarity score that is a measure of similarities between shingles extracted from the first past version of the first document and shingles extracted from the second past version of the first document, and determining that the similarity score satisfies a threshold.

25. The storage device of claim 19 wherein the operations further comprise:

identifying a third query as related to the first document based on a quality of result statistic for the third query and the first document;

generating a third modified quality of result statistic for the third query and a second document by modifying a quality of result statistic for the third query and the second document by a factor, where the factor is based on the one or more time trend statistics; and

providing the third modified quality of result statistic as an input to the document ranking process for the second document and the third query.

26. The storage device of claim 19 wherein each quality of result statistic is an estimate of a respective percentage of users that found the first document relevant to the first query out of a first total number of users who viewed the first document as a search result for the first query during a respective time period.

27. A storage device having instructions stored thereon that, when executed by data processing apparatus, cause the data processing apparatus to perform actions comprising:

calculating a difference score for first quality of result statistics for a group of documents as search results for a first query corresponding to a first time period and second quality of result statistics for the group of documents as search results for the first query corresponding to a different second time period, wherein a quality of result statistic for a particular document in the group of documents and a particular time period is an estimate of a respective portion of user interactions with the particular document as a search result for the first query during the particular time period out of a total number of user interactions with documents as search results for the first query during the particular time period;

identifying one or more documents that are responsive to the first query and modifying a respective quality of result statistic for each identified document based on the difference score, resulting in a respective modified quality of result statistic for each identified document; and

providing the respective modified quality of result statistic for one of the identified documents as an input to the document ranking process for the identified documents and the first query.

28. A computer-implemented method comprising:

calculating one or more time trend statistics for a plurality of quality of result statistics for a first document as a search result for a first query, each of the quality of result statistics corresponding to a different time period and being an estimate of a respective portion of users that found the first document relevant to the first query out of a total number of users who viewed the first document as a search result for the first query during the time period, the one or more time trend statistics estimating changes in the quality of result statistics over time;

generating a first modified quality of result statistic by modifying a first quality of result statistic for the first document as a search result for the first query by a factor, where the factor is based on the one or more time trend statistics; and

providing the first modified quality of result statistic as an input to a document ranking process for the first document and the first query.

29. The method of claim 28, wherein the one or more time trend statistics include a quality of result difference between two of the quality of result statistics.

30. The method of claim 29, further comprising verifying that the quality of result difference satisfies a statistically significant threshold before generating the first modified quality of result statistic.

31. The method of claim 30, wherein the factor is further based on how statistically significant the quality of result difference is.

32. The method of claim 28, further comprising:

generating a second modified quality of result statistic for the first document and a second different query according to the one or more time trend statistics; and

providing the second modified quality of result statistic as an input to the document ranking process for the first document and the second query.

33. The method of claim 28, further comprising determining that a first past version of the first document during a first time period is different from a second past version of the first document during a second time period before modifying the first quality of result statistic.

34. The method of claim 33, wherein determining that the first past version of the first document is different from the second past version of the first document comprises calculating a similarity score that is a measure of similarities between shingles extracted from the first past version of the first document and shingles extracted from the second past version of the first document, and determining that the similarity score satisfies a threshold.

35. The method of claim 28, further comprising:

identifying a second query as related to the first document based on a quality of result statistic for the second query and the first document;

generating a second modified quality of result statistic for the second query and a second document by modifying a quality of result statistic for the second query and the second document by a factor, where the factor is based on the one or more time trend statistics; and

providing the second modified quality of result statistic as an input to the document ranking process for the second document and the second query.

36. A system comprising:

one or more computers programmed to perform operations comprising;

calculating one or more time trend statistics for a plurality of quality of result statistics for a first document as a search result for a first query, each of the quality of result statistics corresponding to a different time period and being an estimate of a respective portion of users that found the first document relevant to the first query out of a total number of users who viewed the first document as a search result for the first query during the time period, the one or more time trend statistics estimating changes in the quality of result statistics over time;

generating a first modified quality of result statistic by modifying a first quality of result statistic for the first document as a search result for the first query by a factor, where the factor is based on the one or more time trend statistics; and

providing the first modified quality of result statistic as an input to a document ranking process for the first document and the first query.

37. The system of claim 36, wherein the one or more time trend statistics include a quality of result difference between two of the quality of result statistics.

38. The system of claim 37, further programmed to perform operations comprising verifying that the quality of result difference satisfies a statistically significant threshold before generating the first modified quality of result statistic.

39. The system of claim 36, wherein the factor is further based on how statistically significant the quality of result difference is.

40. The system of claim 36, further programmed to perform operations comprising:

generating a second modified quality of result statistic for the first document and a second different query according to the one or more time trend statistics; and

providing the second modified quality of result statistic as an input to the document ranking process for the first document and the second query.

41. The system of claim 36, further programmed to perform operations comprising determining that a first past version of the first document during a first time period is different from a second past version of the first document during a second time period before modifying the first quality of result statistic.

42. The system of claim 41, wherein determining that the first past version of the first document is different from the second past version of the first document comprises calculating a similarity score that is a measure of similarities between shingles extracted from the first past version of the first document and shingles extracted from the second past version of the first document, and determining that the similarity score satisfies a threshold.

43. The system of claim 36, further programmed to perform operations comprising:

identifying a second query as related to the first document based on a quality of result statistic for the second query and the first document;

generating a second modified quality of result statistic for the second query and a second document by modifying a quality of result statistic for the second query and the second document by a factor, where the factor is based on the one or more time trend statistics; and

providing the second modified quality of result statistic as an input to the document ranking process for the second document and the second query.

44. A computer storage device having instructions stored thereon that, when executed by data processing apparatus, cause the data processing apparatus to perform operations comprising:

calculating one or more time trend statistics for a plurality of quality of result statistics for a first document as a search result for a first query, each of the quality of result statistics corresponding to a different time period and being an estimate of a respective portion of users that found the first document relevant to the first query out of a total number of users who viewed the first document as a search result for the first query during the time period, the one or more time trend statistics estimating changes in the quality of result statistics over time;

generating a first modified quality of result statistic by modifying a first quality of result statistic for the first document as a search result for the first query by a factor, where the factor is based on the one or more time trend statistics; and

providing the first modified quality of result statistic as an input to a document ranking process for the first document and the first query.

45. The storage device of claim 44, wherein the one or more time trend statistics include a quality of result difference between two of the quality of result statistics.

46. The storage device of claim 44 wherein the operations further comprise verifying that the quality of result difference satisfies a statistically significant threshold before generating the first modified quality of result statistic.

47. The storage device of claim 46 wherein the factor is further based on how statistically significant the quality of result difference is.

48. The storage device of claim 44 wherein the operations further comprise:

generating a second modified quality of result statistic for the first document and a second different query according to the one or more time trend statistics; and

providing the second modified quality of result statistic as an input to the document ranking process for the first document and the second query.

49. The storage device of claim 44 wherein the operations further comprise determining that a first past version of the first document during a first time period is different from a second past version of the first document during a second time period before modifying the first quality of result statistic.

50. The storage device of claim 49 wherein determining that the first past version of the first document is different from the second past version of the first document comprises calculating a similarity score that is a measure of similarities between shingles extracted from the first past version of the first document and shingles extracted from the second past version of the first document, and determining that the similarity score satisfies a threshold.

51. The storage device of claim 44 wherein the operations further comprise:

identifying a second query as related to the first document based on a quality of result statistic for the second query and the first document;

generating a second modified quality of result statistic for the second query and a second document by modifying a quality of result statistic for the second query and the second document by a factor, where the factor is based on the one or more time trend statistics; and

providing the second modified quality of result statistic as an input to the document ranking process for the second document and the second query.