System and Method for Enabling Website Owners to Manage Crawl Rate in a Website Indexing System

• US 20100077098A1
• Filed: 09/04/2009
• Published: 03/25/2010
• Est. Priority Date: 10/12/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Amanda Camp, Jeremy J. Lilley, Katherine Jane Lai, Maximilian Ibel, Neal Douglas Cardwell, Patrik Reali, Ted J. Bonkenburg, Vanessa Fox
• Inventors: Camp, Amanda, Reali, Patrik, Ibel, Maximilian, Fox, Vanessa, Lai, Katherine Jane, Cardwell, Neal Douglas, Bonkenburg, Ted J., Lilley, Jeremy J.
• Granted Patent: US 8,032,518 B2
• Time in Patent Office: Days
• US Class Current: 709/233
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.

Claims

1. A computer-implemented method of indexing documents in websites, the method comprising:

on a server system having one or more processors and memory storing programs to be executed by the one or more processors;

for each website of a multiplicity of websites, each website having a corresponding current crawl rate limit;

crawling the respective website, in accordance with the current crawl rate limit corresponding to the respective website, to download documents from the respective website for inclusion in a database;

storing crawl data associated with the crawling of the respective website; and

providing, for display, a crawl rate control mechanism to a respective owner of the respective website, including providing for display to the respective owner at least a portion of the crawl data, and enabling selection of a new crawl rate limit corresponding to the respective website by the respective owner.

2. The computer-implemented method of claim 1, further including crawling the respective website at a rate no greater than the current crawl rate limit.

3. The computer-implemented method of claim 1, further comprising comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website;

and

performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is a limiting factor in crawling the respective website.

4. The computer-implemented method of claim 3, further comprising:

when the current crawl rate limit is not a limiting factor in crawling the respective website, informing the respective owner that a request for a faster crawl rate may not change a current crawl rate for crawling the respective website.

5. The computer-implemented method of claim 3, wherein the current crawl rate limit is a limiting factor only when a difference between the current crawl rate limit and the maximum crawl rate for the respective website over the defined period of time is less than a predefined quantity.

6. The computer-implemented method of claim 1, further comprising:

in response to a request to decrease the current crawl rate for crawling the respective website, decreasing the current crawl rate.

7. The computer-implemented method of claim 1, wherein storing crawl data further comprises determining a number documents of the respective website that are accessed during one or more crawl sessions.

8. The computer-implemented method of claim 7, wherein storing crawl data further comprises determining an average quantity of time expended to access the documents from the respective website during the one or more crawl sessions.

9. The computer-implemented method of claim 1, wherein storing crawl data further comprises determining a number of bytes downloaded from the respective website during one or more crawl sessions.

10. The computer-implemented method of claim 1, including providing, for display, resource usage statistics corresponding to resources of the respective website used during a plurality of prior crawl visits of the respective website.

11. The computer-implemented method of claim 1, wherein the providing includes providing, for concurrent display:

the current crawl rate limit associated with the crawling of the respective website;

crawl data, including statistical information associated with crawling the respective website; and

an interface for enabling the respective owner to select a new crawl rate limit.

12. The computer-implemented method of claim 11, wherein the providing for concurrent display includes providing for display recommendations for selecting the new crawl rate limit, wherein the recommendations are based, at least in part, on whether the current crawl rate limit is a limiting factor in crawling the respective website.

13. A computer system comprising:

memory;

one or more processors; and

at least one program stored in the memory and executed by the one or more processors, the at least one program including;

web crawl control instructions for controlling crawling of each website of a multiplicity of websites, each website having a corresponding current crawl rate limit,

the web crawl control instructions including;

instructions for crawling a respective website of the multiplicity of websites, in accordance with the current crawl rate limit corresponding to the respective website, to download documents from the respective website for inclusion in a database;

instructions for storing crawl data associated with the crawling of the respective website; and

instructions for providing, for display, a crawl rate control mechanism to a respective owner of the respective website, including providing, for display to the respective owner, at least a portion of the crawl data, and enabling selection, by the respective owner, of a new crawl rate limit corresponding to the respective website.

14. The computer system of claim 13, the web crawl control instructions further including instructions for crawling the respective website at a rate no greater than the current crawl rate limit.

15. The computer system of claim 13, the web crawl control instructions further comprising, instructions for:

comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and

performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is a limiting factor in crawling the respective website.

16. The computer system of claim 15, the web crawl control instructions further comprising:

instructions for informing the respective owner that a request for a faster crawl rate may not change a current crawl rate for crawling the respective website when the current crawl rate limit is not a limiting factor in crawling the respective website.

17. The computer system of claim 15, wherein the current crawl rate limit is a limiting factor only when a difference between the current crawl rate limit and the maximum crawl rate for the respective website over the defined period of time is less than a predefined quantity.

18. The computer system of claim 13, the web crawl control instructions further comprising:

instructions for decreasing the current crawl rate in response to a request to decrease the current crawl rate for crawling the respective website.

19. The computer system of claim 13, wherein the instructions for storing crawl data further comprises instructions for determining a number documents of the respective website that are accessed during one or more crawl sessions.

20. The computer system of claim 19, wherein the instructions for storing crawl data further comprises instructions for determining an average quantity of time expended to access the documents from the respective website during the one or more crawl sessions.

21. The computer system of claim 13, wherein the instructions for storing crawl data further comprises instructions for determining a number of bytes downloaded from the respective website during one or more crawl sessions.

22. The computer system of claim 13, including instructions for providing, for display, resource usage statistics corresponding to resources of the respective website used during a plurality of prior crawl visits of the respective website.

23. The computer system of claim 13, wherein the instructions for providing further include instructions providing, for concurrent display:

the current crawl rate associated with the crawling of the respective website;

crawl data, including statistical information associated with crawling the respective website; and

an interface for enabling the respective owner to select a new crawl rate limit.

24. The computer system of claim 23, wherein the instructions for providing further include instructions for providing an information display having recommendations for selecting the new crawl rate limit, wherein the recommendations are based, at least in part, on whether the current crawl rate is a limiting factor in crawling the website.

25. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising:

web crawl control instructions for controlling crawling of each website of a multiplicity of websites, each website having a corresponding current crawl rate limit, the web crawl control instructions including;

instructions for crawling a respective website of the multiplicity of websites, in accordance with the current crawl rate limit corresponding to the respective website, to download documents from the respective website for inclusion in a database;

instructions for storing crawl data associated with the crawling of the respective website; and

instructions for providing, for display, a crawl rate control mechanism to a respective owner of the respective website, including providing, for display to the respective owner, at least a portion of the crawl data, and enabling selection, by the respective owner, of a new crawl rate limit corresponding to the respective website.

26. The computer program product of claim 25, the web crawl control instructions further including instructions for crawling the respective website at a rate no greater than the current crawl rate limit.

27. The computer program product of claim 25, the web crawl control instructions further comprising, instructions for:

comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and

performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is a limiting factor in crawling the respective website.

28. The computer program product of claim 27, the web crawl control instructions further comprising:

instructions for informing the respective owner that a request for a faster crawl rate may not change the current crawl rate for crawling the respective website when the current crawl rate limit is not a limiting factor in crawling the respective website.

29. The computer program product of claim 27, wherein the current crawl rate limit is a limiting factor only when a difference between the current crawl rate limit and the maximum crawl rate for the respective website over the defined period of time is less than a predefined quantity.

30. The computer program product of claim 25, the web crawl control instructions further comprising:

instructions for decreasing the current crawl rate in response to a request to decrease the current crawl rate for crawling the respective website.

31. The computer program product of claim 25, wherein the instructions for storing crawl data further comprises instructions for determining a number of documents of the respective website that are accessed during one or more crawl sessions.

32. The computer program product of claim 25, wherein the instructions for storing crawl data further comprises instructions for determining a number of bytes downloaded from the respective website during one or more crawl sessions.

33. The computer program product of claim 25, including instructions for providing, for display, resource usage statistics corresponding to resources of the respective website used during a plurality of prior crawl visits of the website.

34. The computer program product of claim 25, wherein the crawl data provided for display further includes resource usage statistics corresponding to resources of the respective website used during a plurality of prior crawl sessions of the respective website.

35. The computer program product of claim 25, wherein the instructions for providing further include instructions providing, for concurrent display:

the current crawl rate associated with the crawling of the respective website;

crawl data, including statistical information associated with crawling the respective website; and

an interface for enabling the respective owner to select a new crawl rate limit.

36. The computer program product of claim 35, wherein the instructions for providing for display further include instructions for providing for display recommendations for selecting the new crawl rate limit, wherein the recommendations are based, at least in part, on whether the current crawl rate limit is a limiting factor in crawling the respective website.