Adaptive crawl rates based on publication frequency

• US 8,255,385 B1
• Filed: 03/22/2011
• Issued: 08/28/2012
• Est. Priority Date: 03/22/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sun, Walter, Zhang, Xiao, Li, Yipeng, Ahmed, Junaid
• Primary Examiner(s): Mahmoudi, Tony
• Assistant Examiner(s): Pham, Tuan A
• Application Number:
  US13/053,772
• Time in Patent Office: 525 Days
• Field of Search: None
• US Class Current: 707/709
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

Methods and systems for determining an adaptive crawl rate for a Web crawler based on historical publication data from a Web source are provided. A frequency of publication of the Web source is determined over a specified period of time, and an adaptive crawl rate is calculated using the frequency of publication. The Web crawler is then deployed at the calculated adaptive crawl rate.

Claims

1. A system for adaptively deploying a Web-crawler to a Web source at a crawl rate based on historical publication data for the Web source, the system comprising:

a computing device associated with a search engine having one or more processors and one or more computer-readable storage media; and

a data store coupled with the search engine,

wherein the search engine;

determines an update frequency estimation for a particular Web page, the update frequency estimation being determined according to;

F_i=Σ

_k=1ⁱw_kF_i-k

A) wherein F_i is the update frequency estimation for the particular Web page for a time period, i,

B) wherein w_k is a weight factor given to a particular time segment, k, included in the time period, and

C) wherein F_i-k is a publication rate for a given time segment included in the current time period; and

calculates the adaptive crawl rate by multiplying the update frequency estimation for the particular Web page by a constant.

2. The system of claim 1, wherein the publication rate is determined from a real simple syndication (RSS) feed, a hypertext markup language (HTML) document, a site map, or any combination thereof, associated with the particular Web page.

3. The system of claim 1, wherein the given time segment varies depending upon how frequently the Web source is updated.

4. The system of claim 3, wherein the given time segment varies from five days to ten days if the particular Web page is updated frequently.

5. The system of claim 4, wherein the given time segment varies from five days to ten days if the particular Web page is updated more than fifty times per day.

6. The system of claim 3, wherein the given time segment varies from 30 days to 90 days if the particular Web page is updated infrequently.

7. The system of claim 6, wherein the given time segment varies from 60 days to 90 days if the particular Web page is updated once a month.

8. A computerized method carried out by a search engine running on a processor for adaptively deploying a Web-crawler at a crawl rate based on historical publication data, the method comprising:

determining, using the processor, an update frequency estimation for a particular Web page, the update frequency estimation being determined according to;

F_i=Σ

_k=1ⁱw_kF_i-k

A wherein F_i is the update frequency estimation for the particular Web page for a time period, i,

a) wherein w_k is a weight factor given to a particular time segment, k, included in the time period, and

C) wherein F_i-k publication rate for a given time segment included in the current time period; and

calculating the adaptive crawl rate by multiplying the update frequency estimation for the particular Web page by a constant.

9. The method of claim 8, wherein the update frequency estimation is determined between once per day and once every fourteen days.

10. The method of claim 8, wherein the time period, i, the weight factor, w_k, and the constant are customized for each of a plurality of Web pages.

11. The method of claim 8, wherein the constant is a value that minimizes staleness of content associated with the particular Web page.

12. The method of claim 11, wherein the constant is two.

13. One or more computer-readable storage devices not consisting of a signal and storing computer-executable instructions, which, when executed by a computing device, cause the computing device to perform a method for determining an adaptive crawl rate based upon historical publication data, the method comprising:

determining an update frequency estimation for a particular Web page, the update frequency estimation being determined according to;

F_i=Σ

_k=1ⁱw_kF_i-k

A) wherein F_i is the update frequency estimation for the particular Web page for a time period, i,

B) wherein w_k is a weight factor given to a particular time segment, k, included in the time period, and

C) wherein F_i-k is a publication rate for a given time segment included in the current time period; and

calculating the adaptive crawl rate by multiplying the update frequency estimation for the particular Web page by a constant.

14. The one or more computer-readable storage devices of claim 13, wherein the update frequency estimation is determined between once per day and once every fourteen days.

15. The one or more computer-readable storage a devices of claim 14, wherein the update frequency estimation is determined once every fourteen days.

16. The one or more computer-readable storage devices of claim 13, wherein the time period, i, the weight factor, w_k, and the constant are customized for each of a plurality of Web pages.

17. The one or more computer-readable storage devices of claim 13, wherein the weight factor, w_k, is equivalent for each equivalent time segment, k, included in the time period, i.

18. The one or more computer-readable storage devices of claim 17, wherein the weight factor, w_k, is 1/i.

19. The one or more computer-readable storage devices of claim 13, wherein the weight factor, w_k, of at least two equivalent time segments, k, included in the time period, i, differ from one another.

20. The one or more computer-readable storage a devices of claim 19, wherein a weight factor, w_k, given to time segments occurring later in the time period, i, is greater than a weight factor, w_k, given to time segments occurring earlier in the time period, i.