Web crawler scheduler that utilizes sitemaps from websites

• US 8,417,686 B2
• Filed: 10/11/2011
• Issued: 04/09/2013
• Est. Priority Date: 05/31/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Brawer, Sascha B., Ibel, Maximilian, Keller, Ralph Michael, Shivakumar, Narayanan
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Liu, Hexing
• Application Number:
  US13/271,160
• Publication Number:
  US 20120036118A1
• Time in Patent Office: 546 Days
• Field of Search: 707/708, 707/709, 707/711, 707/706, 707/741, 707/707
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.

Claims

1. A method of scheduling documents for crawling, performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising:

storing sitemap information for a plurality of websites, wherein the information includes a predicted update period for at least a plurality of documents identified by the sitemap information;

analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date;

updating the stored sitemap information for the identified respective website by downloading updated sitemap information for the identified respective website; and

scheduling documents for crawling in accordance with the updated stored sitemap information for the identified respective website.

2. The method of claim 1, wherein analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date comprises analyzing the stored sitemap information to identify a respective website having sitemap information that is older than a shortest predicted update period for at least one document listed in the sitemap information for the respective website.

3. The method of claim 2, wherein analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date comprises subtracting a last modification date in the sitemap from a current date.

4. The method of claim 1, wherein the sitemap information includes last modification dates for at least a plurality of documents identified by the sitemap information;

and

wherein at least some of the documents scheduled for crawling are scheduled in accordance with their last modification dates.

5. The method of claim 4, wherein the scheduling comprises deferring scheduling a respective document for crawling when the last modification date for the respective document corresponds to a download date or a download date and time that is no later than a respective download date or a download date and time at which the respective document was last crawled.

6. The method of claim 1, further comprising receiving a sitemap notification including a sitemap URL corresponding to a sitemap for a respective website of the plurality of websites.

7. The method of claim 1, wherein scheduling documents for crawling comprises generating a list of document identifiers that identify the scheduled documents.

8. A system for scheduling documents for crawling, comprising:

one or more processors; and

memory storing one or more modules;

the one or more modules including instructions for;

storing sitemap information for a plurality of websites, wherein the information includes a predicted update period for at least a plurality of documents identified by the sitemap information;

analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date;

updating the stored sitemap information for the identified respective website by downloading updated sitemap information for the identified respective website; and

scheduling documents for crawling in accordance with the updated stored sitemap information for the identified respective website.

9. The system of claim 8, wherein the instructions for analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date comprise instructions for analyzing the stored sitemap information to identify a respective website having sitemap information that is older than a shortest predicted update period for at least one document listed in the sitemap information for the respective website.

10. The system of claim 9, wherein the instructions for analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date comprise instructions for subtracting a last modification date in the sitemap from a current date.

11. The system of claim 8, wherein the sitemap information includes last modification dates for at least a plurality of documents identified by the sitemap information;

and

wherein the instructions for scheduling documents for crawling comprise instructions for scheduling at least some of the documents for crawling in accordance with their last modification dates.

12. The system of claim 11, wherein the instructions for scheduling comprise instructions for deferring scheduling a respective document for crawling when the last modification date for the respective document corresponds to a download date or a download date and time that is no later than a respective download date or a download date and time at which the respective document was last crawled.

13. The system of claim 8, further comprising instructions for receiving a sitemap notification including a sitemap URL corresponding to a sitemap for a respective website of the plurality of websites.

14. The system of claim 8, wherein instructions for scheduling documents for crawling comprise instructions for generating a list of document identifiers that identify the scheduled documents.

15. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

storing sitemap information for a plurality of websites, wherein the information includes a predicted update period for at least a plurality of documents identified by the sitemap information;

analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date;

updating the stored sitemap information for the identified respective website by downloading updated sitemap information for the identified respective website; and

scheduling documents for crawling in accordance with the updated stored sitemap information for the identified respective website.

16. The non-transitory computer readable storage medium of claim 15, wherein the instructions for analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date comprise instructions for analyzing the stored sitemap information to identify a respective website having sitemap information that is older than a shortest predicted update period for at least one document listed in the sitemap information for the respective website.

17. The non-transitory computer readable storage medium of claim 16, wherein the instructions for analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date comprise instructions for subtracting a last modification date in the sitemap from a current date.

18. The non-transitory computer readable storage medium of claim 15, wherein the sitemap information includes last modification dates for at least a plurality of documents identified by the sitemap information;

and

wherein the instructions for scheduling documents for crawling comprise instructions for scheduling at least some of the documents for crawling in accordance with their last modification dates.

19. The non-transitory computer readable storage medium of claim 18, wherein the instructions for scheduling comprise instructions for deferring scheduling a respective document for crawling when the last modification date for the respective document corresponds to a download date or a download date and time that is no later than a respective download date or a download date and time at which the respective document was last crawled.

20. The non-transitory computer readable storage medium of claim 15, further comprising instructions for receiving a sitemap notification including a sitemap URL corresponding to a sitemap for a respective website of the plurality of websites.

21. The non-transitory computer readable storage medium of claim 15, wherein instructions for scheduling documents for crawling comprise instructions for generating a list of document identifiers that identify the scheduled documents.