Web Crawler Scheduler that Utilizes Sitemaps from Websites

• US 20100262592A1
• Filed: 06/25/2010
• Published: 10/14/2010
• Est. Priority Date: 05/31/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Maximilian Ibel, Ralph Michael Keller, Sascha B. Brawer, Shivakumar Narayanan
• Inventors: Shivakumar, Narayanan, Ibel, Maximilian, Brawer, Sascha B., Keller, Ralph Michael
• Granted Patent: US 8,037,054 B2
• Time in Patent Office: Days
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.

Claims

1. A method of scheduling documents for crawling, performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising:

receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website;

in response to the notification;

accessing the sitemap at the sitemap URL; and

retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website;

scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap; and

downloading at least a subset of the documents scheduled for downloading.

2. The method of claim 1, wherein the metadata retrieved from the sitemap comprises, for at least a plurality of respective documents associated with the website, document modification date information indicating when the respective documents were last modified;

and

wherein the scheduling is performed in accordance with the document modification date information for the respective documents.

3. The method of claim 2, wherein the scheduling comprises deferring scheduling a respective document for downloading when the document modification date information for the respective document corresponds to a download date or a download date and time that is no later than a respective download date or a download date and time at which the respective document was last downloaded.

4. The method of claim 1, the metadata providing information related to at least one of prioritizing documents for crawling by a web crawler and selecting documents for inclusion in a crawl;

the scheduling including at least one of prioritizing documents for crawling and selecting documents for crawling in accordance with the metadata.

5. The method of claim 1, wherein scheduling documents for downloading comprises generating a list of document identifiers that identify the scheduled documents.

6. A system for scheduling documents for crawling, comprising:

one or more processors; and

memory storing one or more modules;

the one or more modules including instructions to;

receive from a website a notification that includes a sitemap URL corresponding to a sitemap for the website;

in response to the notification;

access the sitemap at the sitemap URL; and

retrieve from the sitemap document location information and metadata for a plurality of documents associated with the website;

schedule for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap; and

download at least a subset of the documents scheduled for downloading.

7. The system of claim 6, wherein the metadata retrieved from the sitemap comprises, for at least a plurality of respective documents associated with the website, document modification date information indicating when the respective documents were last modified;

and

wherein the scheduling is performed in accordance with the document modification date information for the respective documents.

8. The system of claim 7, wherein the scheduling comprises deferring scheduling a respective document for downloading when the document modification date information for the respective document corresponds to a download date or a download date and time that is no later than a respective download date or a download date and time at which the respective document was last downloaded.

9. The system of claim 6, the metadata providing information related to at least one of prioritizing documents for crawling by a web crawler and selecting documents for inclusion in a crawl;

the scheduling including at least one of prioritizing documents for crawling and selecting documents for crawling in accordance with the metadata.

10. The system of claim 6, wherein the instructions for scheduling documents for downloading comprise instructions for generating a list of document identifiers that identify the scheduled documents.

11. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising instructions for:

receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website;

in response to the notification;

accessing the sitemap at the sitemap URL; and

retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website;

scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap; and

downloading at least a subset of the documents scheduled for downloading.

12. The computer program product of claim 11, wherein the metadata retrieved from the sitemap comprises, for at least a plurality of respective documents associated with the website, document modification date information indicating when the respective documents were last modified;

and

wherein the scheduling is performed in accordance with the document modification date information for the respective documents.

13. The computer program product of claim 12, wherein the scheduling comprises deferring scheduling a respective document for downloading when the document modification date information for the respective document corresponds to a download date or a download date and time that is no later than a respective download date or a download date and time at which the respective document was last downloaded.

14. The computer program product of claim 11, the metadata providing information related to at least one of prioritizing documents for crawling by a web crawler and selecting documents for inclusion in a crawl;

the scheduling including at least one of prioritizing documents for crawling and selecting documents for crawling in accordance with the metadata.

15. The computer program product of claim 11, wherein the instructions for scheduling documents for downloading comprise instructions for generating a list of document identifiers that identify the scheduled documents.