Web crawler scheduler that utilizes sitemaps from websites
First Claim
Patent Images
1. A method of scheduling documents for crawling, performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising:
- receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website;
in response to the notification;
accessing the sitemap at the sitemap URL; and
retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website, wherein the metadata comprises, for at least a plurality of respective documents associated with the website, document update rate information indicating update frequencies associated with the respective documents;
scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap, wherein the scheduling includes scheduling a respective document for downloading when one of a date or date and time at which the respective document was last downloaded differs from a current date or a current date and time by an amount that is greater than a duration corresponding to the document update rate information for the respective document; and
downloading at least a subset of the documents scheduled for downloading.
2 Assignments
0 Petitions
Accused Products
Abstract
Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.
84 Citations
31 Claims
-
1. A method of scheduling documents for crawling, performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising:
-
receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website; in response to the notification; accessing the sitemap at the sitemap URL; and retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website, wherein the metadata comprises, for at least a plurality of respective documents associated with the website, document update rate information indicating update frequencies associated with the respective documents; scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap, wherein the scheduling includes scheduling a respective document for downloading when one of a date or date and time at which the respective document was last downloaded differs from a current date or a current date and time by an amount that is greater than a duration corresponding to the document update rate information for the respective document; and downloading at least a subset of the documents scheduled for downloading. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A method of scheduling documents for crawling, performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising:
-
receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website; in response to the notification; accessing the sitemap at the sitemap URL; and retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website, wherein the metadata retrieved from the sitemap includes, for at least a plurality of respective documents associated with the website, document importance information indicating relative importance values associated with the respective documents; scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap, wherein the scheduling is performed in accordance with a score assigned to each document deemed eligible for downloading, and wherein the score assigned to a respective document is adjusted by a boost factor corresponding to the relative importance value indicated by the document importance information for the respective document; and downloading at least a subset of the documents scheduled for downloading. - View Dependent Claims (7, 8, 9, 10)
-
-
11. A system for scheduling documents for crawling, comprising:
-
one or more processors; and memory storing one or more modules; the one or more modules including instructions to; receive from a website a notification that includes a sitemap URL corresponding to a sitemap for the website; in response to the notification; access the sitemap at the sitemap URL; and retrieve from the sitemap document location information and metadata for a plurality of documents associated with the website, wherein the metadata comprises, for at least a plurality of respective documents associated with the website, document update rate information indicating update frequencies associated with the respective documents; schedule for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap, wherein the scheduling includes scheduling a respective document for downloading when one of a date or date and time at which the respective document was last downloaded differs from a current date or a current date and time by an amount that is greater than a duration corresponding to the document update rate information for the respective document; and download at least a subset of the documents scheduled for downloading. - View Dependent Claims (12, 13, 14, 15)
-
-
16. A system for scheduling documents for crawling, comprising:
-
one or more processors; and memory storing one or more modules; the one or more modules including instructions to; receive from a website a notification that includes a sitemap URL corresponding to a sitemap for the website; in response to the notification; access the sitemap at the sitemap URL; and retrieve from the sitemap document location information and metadata for a plurality of documents associated with the website, wherein the metadata retrieved from the sitemap includes, for at least a plurality of respective documents associated with the website, document importance information indicating relative importance values associated with the respective documents; schedule for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap, wherein the scheduling is performed in accordance with a score assigned to each document deemed eligible for downloading, and wherein the score assigned to a respective document is adjusted by a boost factor corresponding to the relative importance value indicated by the document importance information for the respective document; and download at least a subset of the documents scheduled for downloading. - View Dependent Claims (17, 18, 19, 20)
-
-
21. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising instructions for:
-
receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website; in response to the notification; accessing the sitemap at the sitemap URL; and retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website, wherein the metadata comprises, for at least a plurality of respective documents associated with the website, document update rate information indicating update frequencies associated with the respective documents; scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap, wherein the scheduling includes scheduling a respective document for downloading when one of a date or date and time at which the respective document was last downloaded differs from a current date or a current date and time by an amount that is greater than a duration corresponding to the document update rate information for the respective document; and downloading at least a subset of the documents scheduled for downloading. - View Dependent Claims (22, 23, 24, 25)
-
-
26. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising instructions for:
-
receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website; in response to the notification; accessing the sitemap at the sitemap URL; and retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website, wherein the metadata retrieved from the sitemap includes, for at least a plurality of respective documents associated with the website, document importance information indicating relative importance values associated with the respective documents; scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap, wherein the scheduling is performed in accordance with a score assigned to each document deemed eligible for downloading, and wherein the score assigned to a respective document is adjusted by a boost factor corresponding to the relative importance value indicated by the document importance information for the respective document; and downloading at least a subset of the documents scheduled for downloading. - View Dependent Claims (27, 28, 29, 30)
-
-
31. A system for scheduling documents for crawling, comprising
one or more processors; - and
memory storing one or more programs to be executed by the one or more processors; the system including; means for receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website; means for, in response to the notification; accessing the sitemap at the sitemap URL; and retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website, wherein the metadata comprises, for at least a plurality of respective documents associated with the website, document update rate information indicating update frequencies associated with the respective documents; means for scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap, wherein the scheduling includes scheduling a respective document for downloading when one of a date or date and time at which the respective document was last downloaded differs from a current date or a current date and time by an amount that is greater than a duration corresponding to the document update rate information for the respective document; and means for downloading at least a subset of the documents scheduled for downloading.
- and
Specification