System and method for enabling website owner to manage crawl rate in a website indexing system

• US 8,458,163 B2
• Filed: 10/03/2011
• Issued: 06/04/2013
• Est. Priority Date: 10/12/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fox, Vanessa, Camp, Amanda Ann, Ibel, Maximilian, Reali, Patrik Rene Celeste, Lilley, Jeremy J., Lai, Katherine Jane, Bonkenburg, Ted J., Cardwell, Neal Douglas
• Primary Examiner(s): LY, CHEYNE D
• Application Number:
  US13/252,149
• Publication Number:
  US 20120023091A1
• Time in Patent Office: 610 Days
• Field of Search: 707/709, 707/713, 709/224, 709/225, 709/226
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

Web crawlers crawl websites to access documents of the website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodiments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.

Claims

1. A method of indexing documents in websites, the method comprising:

on a server system having one or more processors and memory storing programs to be executed by the one or more processors;

storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit;

comparing a maximum crawl rate for a respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website;

performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is the limiting factor in crawling the respective website; and

providing a crawl rate control mechanism to a respective owner of the respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner.

2. The method of claim 1, further including crawling the respective website at a rate no greater than the current crawl rate limit for crawling the respective website.

3. The method of claim 1, further comprising:

when the current crawl rate limit is not the limiting factor in crawling the respective website, informing the respective owner that a request for a faster crawl rate may not change a current crawl rate for crawling the respective website.

4. The method of claim 1, wherein the current crawl rate limit is the limiting factor only when a difference between the current crawl rate limit and the maximum crawl rate for the respective website over the defined period of time is less than a predefined quantity.

5. The method of claim 1, further comprising:

in response to a request to decrease the current crawl rate limit for crawling the respective website, decreasing the current crawl rate limit for crawling the respective website.

6. The method of claim 1, further comprising determining a number of documents of the respective website that are accessed during one or more crawl sessions and storing corresponding crawl data.

7. The method of claim 6, wherein storing the corresponding crawl data further comprises determining an average quantity of time expended to access the documents from the respective website during the one or more crawl sessions.

8. The method of claim 1, further comprising determining a number of bytes downloaded from the respective website during one or more crawl sessions and storing corresponding crawl data.

9. The method of claim 1, including providing, for display, resource usage statistics corresponding to resources of the respective website used during a plurality of prior crawl visits of the respective website.

10. The method of claim 1, wherein providing the crawl rate mechanism includes providing, for concurrent display:

the current crawl rate limit associated with the crawling of the respective website;

crawl data, including statistical information associated with crawling the respective website; and

an interface for enabling the respective owner to select the new crawl rate limit.

11. The method of claim 10, wherein the providing for concurrent display includes providing for display recommendations for selecting the new crawl rate limit, wherein the recommendations are based, at least in part, on whether the current crawl rate limit is the limiting factor in crawling the respective website.

34. The method of claim 1, including providing, for display, a status log indicating respective statuses of documents of the respective website that are accessed during one or more crawl sessions.

35. The method of claim 1, wherein storing the corresponding crawl rate limit further comprises storing a list of respective time intervals and a crawl rate limit for each time interval.

12. A computer system comprising:

memory;

one or more processors; and

at least one program stored in the memory and executed by the one or more processors, the at least one program including instructions for;

storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit;

comparing a maximum crawl rate for a respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website;

performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is the limiting factor in crawling the respective website; and

providing a crawl rate control mechanism to a respective owner of the respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner.

13. The computer system of claim 12, the at least one program further including instructions for crawling the respective website at a rate no greater than the current crawl rate limit for crawling the respective website.

14. The computer system of claim 12, the at least one program further comprising:

instructions for informing the respective owner that a request for a faster crawl rate may not change a current crawl rate for crawling the respective website when the current crawl rate limit is not the limiting factor in crawling the respective website.

15. The computer system of claim 12, wherein the current crawl rate limit is the limiting factor only when a difference between the current crawl rate limit and the maximum crawl rate for the respective website over the defined period of time is less than a predefined quantity.

16. The computer system of claim 12, the instructions further comprising:

instructions for decreasing the current crawl rate limit for crawling the respective website in response to a request to decrease the current crawl rate limit for crawling the respective website.

17. The computer system of claim 12, wherein the at least one program further comprises instructions for determining a number of documents of the respective website that are accessed during one or more crawl sessions and for storing corresponding crawl data.

18. The computer system of claim 17, wherein the instructions for storing the corresponding crawl data further comprise instructions for determining an average quantity of time expended to access the documents from the respective website during the one or more crawl sessions.

19. The computer system of claim 12, wherein the at least one program further comprises instructions for determining a number of bytes downloaded from the respective website during one or more crawl sessions and for storing corresponding crawl data.

20. The computer system of claim 12, wherein the at least one program further comprises instructions for providing, for display, resource usage statistics corresponding to resources of the respective website used during a plurality of prior crawl visits of the respective website.

21. The computer system of claim 12, wherein the instructions for providing the crawl rate control mechanism further include instructions providing, for concurrent display:

the current crawl rate limit associated with the crawling of the respective website;

crawl data, including statistical information associated with crawling the respective website; and

an interface for enabling the respective owner to select the new crawl rate limit.

22. The computer system of claim 21, wherein the instructions for providing for concurrent display further include instructions for providing an information display having recommendations for selecting the new crawl rate limit, wherein the recommendations are based, at least in part, on whether the current crawl rate is the limiting factor in crawling the website.

36. The computer system of claim 12, the at least one program further including instructions for providing, for display, a status log indicating respective statuses of documents of the respective website that are accessed during one or more crawl sessions.

37. The computer system of claim 12, the instructions for storing the corresponding crawl rate limit further comprising storing a list of respective time intervals and a crawl rate limit for each time interval.

23. A non-transitory computer readable storage medium storing at least one program for execution by a computer system, the at least one program comprising instructions for:

storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit;

comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and

performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is the limiting factor in crawling the respective website; and

providing a crawl rate control mechanism to a respective owner of the respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner.

24. The non-transitory computer program product of claim 23, the at least one program further including instructions for crawling the respective website at a rate no greater than the current crawl rate limit for crawling the respective website.

25. The non-transitory computer program product of claim 23, the at least one program further comprising:

instructions for informing the respective owner that a request for a faster crawl rate may not change a current crawl rate for crawling the respective website when the current crawl rate limit is not the limiting factor in crawling the respective website.

26. The non-transitory computer program product of claim 23, wherein the current crawl rate limit is the limiting factor only when a difference between the current crawl rate limit and the maximum crawl rate for the respective website over the defined period of time is less than a predefined quantity.

27. The non-transitory computer program product of claim 23, wherein the at least one program further comprises:

instructions for decreasing the current crawl rate limit in response to a request to decrease the current crawl rate limit for crawling the respective website for crawling the respective website.

28. The non-transitory computer program product of claim 23, wherein the at least one program further comprises instructions for determining a number of documents of the respective website that are accessed during one or more crawl sessions and for storing corresponding crawl data.

29. The non-transitory computer program product of claim 28, wherein the instructions for storing the corresponding crawl data further comprise instructions for determining an average quantity of time expended to access the documents from the respective website during the one or more crawl sessions.

30. The non-transitory computer program product of claim 23, wherein the instructions for storing crawl data further comprise instructions for determining a number of bytes downloaded from the respective website during one or more crawl sessions and for storing corresponding crawl data.

31. The non-transitory computer program product of claim 23, wherein the at least one program further comprises instructions for providing, for display, resource usage statistics corresponding to resources of the respective website used during a plurality of prior crawl visits of the website.

32. The non-transitory computer program product of claim 23, wherein the instructions for providing the crawl rate control mechanism further include instructions providing, for concurrent display:

the current crawl rate associated with the crawling of the respective website;

crawl data, including statistical information associated with crawling the respective website; and

an interface for enabling the respective owner to select the new crawl rate limit.

33. The non-transitory computer program product of claim 32, wherein the instructions for providing for concurrent display further include instructions for providing for display recommendations for selecting the new crawl rate limit, wherein the recommendations are based, at least in part, on whether the current crawl rate limit is the limiting factor in crawling the respective website.

38. The non-transitory computer readable storage medium of claim 23, the at least one program further including instructions for providing, for display, a status log indicating respective statuses of documents of the respective website that are accessed during one or more crawl sessions.

39. The non-transitory computer readable storage medium of claim 23, the instructions for storing the corresponding crawl rate limit further comprising storing a list of respective time intervals and a crawl rate limit for each time interval.