Web crawler scheduler that utilizes sitemaps from websites

• US 9,355,177 B2
• Filed: 01/27/2015
• Issued: 05/31/2016
• Est. Priority Date: 05/31/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ibel, Max, Brawer, Sascha Benjamin, Keller, Ralph Michael, Shivakumar, Narayanan
• Primary Examiner(s): Liu, Hexing
• Application Number:
  US14/606,882
• Publication Number:
  US 20150242508A1
• Time in Patent Office: 490 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods for scheduling documents for crawling are disclosed in which sitemap information is updated for a first website identified by a sitemap by downloading updated sitemap information for the first website and scheduling documents for crawling in accordance with the updated sitemap information for the first website. The sitemap information includes one or more sitemap indexes, where each respective sitemap index in the one or more sitemap indices includes a list of URLs corresponding to documents stored at a corresponding website in a plurality of websites, the plurality of websites including the first website, and each sitemap index in the one or more sitemap indexes includes information identifying one or more of: a last modification date of a URL in the list of URLs, a change frequency of a document specified by the URL, a document title, an authority of the document, and a priority of the document.

Claims

1. A method of scheduling documents for crawling, performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising:

identifying an updated sitemap using a last change date in a sitemap index, the sitemap index including a list of sitemaps for a website, each sitemap having a URL and a last change date;

updating sitemap information for the sitemap by downloading updated sitemap information, wherein

the sitemap information includes a list of URLs corresponding to documents stored at the website and each URL is associated with two or more of;

a last modification date for the URL, a change frequency of a document specified by the URL, and a priority of the document; and

scheduling documents for crawling in accordance with the updated sitemap information.

2. The method of claim 1, wherein the sitemap is generated using (i) an access log, (ii) a pre-made URL list, and (iii) information obtained from a content management system associated with the website.

3. The method of claim 2, wherein the sitemap is in a format selected from the group consisting of an XML file, a plain-text file, a comma-separated value file, and a semicolon-separated file.

4. The method of claim 1, wherein the sitemap index includes site-specific information for the website.

5. The method of claim 4, wherein the site-specific information includes one or more of:

a list of crawl intervals, a crawl rate, and information identifying geographic location of the website.

6. The method of claim 1, wherein scheduling documents for crawling uses a document'"'"'s change frequency.

7. The method of claim 1, wherein scheduling documents for crawling is independent of a document'"'"'s change frequency.

8. The method of claim 1, wherein scheduling documents for crawling uses a document'"'"'s priority.

9. The method of claim 1, wherein scheduling documents for crawling is independent of a document'"'"'s priority.

10. The method of claim 1, wherein the identifying, updating and scheduling are executed (i) in response to obtaining a notification of a change to the sitemap information or (ii) in accordance with a predefined schedule.

11. The method of claim 2, wherein generating the sitemap using an access log includes:

accessing the access log for the website, the access log having one or more URLs;

applying one or more filters to the one or more URLs, thereby producing one or more filtered URLs; and

generating the sitemap using the one or more filtered URLs.

12. The method of claim 11, wherein the access log includes an error URL and a non-error URL.

13. The method of claim 1, further comprising generating the sitemap by:

performing a database scan or a directory crawl to obtain one or more URLs associated with the website;

applying one or more filters to the one or more URLs, thereby producing one or more filtered URLs, and

generating the sitemap using the one or more filtered URLs.

14. The method of claim 1, further comprising generating the sitemap by:

obtaining one or more URLs associated with the website;

indexing content of a webpage at a respective URL in the one or more URLs;

generating records of out-bound links included in the webpage;

detecting duplicates pages;

creating one or more log records for the webpage; and

generating the sitemap for the website using the one or more log records.

15. The method of claim 14, further comprising generating an anchor map for the website using anchor text included in a respective URL in the one or more URLs.

16. The method of claim 15, wherein the anchor map includes records keyed by a fingerprint of an out-bound link included in the webpage.

17. The method of claim 15, further comprising generating search indices for the website, using the anchor map.

18. The method of claim 1, wherein scheduling documents for crawling in accordance with the updated sitemap information includes:

selecting one or more candidate URLs from the list of URLs;

assigning a score to each candidate URL in the one or more candidate URLs;

applying one or more filtering criteria to each candidate URL; and

scheduling for crawling filtered candidate URLs by a first crawler.

19. The method of claim 18, further comprising distributing a filtered candidate URL or a non-candidate URL in the list of URLs, to a second crawler distinct from the first crawler.

20. A computer system comprising:

one or more processors; and

a memory storing one or more programs for execution by the one or more processors, wherein the one or more programs comprising instructions for;

identifying an updated sitemap using a last change date in a sitemap index, the sitemap index including a list of sitemaps for a website, each sitemap having a URL and a last change date,

updating sitemap information for a first website identified by a sitemap by downloading updated sitemap information, wherein

the sitemap information includes a list of URLs corresponding to documents stored at the website and each URL is associated with two or more of;

a last modification date for the URL, a change frequency of a document specified by the URL, and a priority of the document, and

scheduling documents for crawling in accordance with the updated sitemap information.

21. A non-transitory computer readable storage medium storing one or more programs to be executed by a computer system, the one or more programs comprising instructions for:

identifying an updated sitemap using a last change date in a sitemap index, the sitemap index including a list of sitemaps for a website, each sitemap having a URL and a last change date;

updating sitemap information for the sitemap by downloading updated sitemap information, wherein

the sitemap information includes a list of URLs corresponding to documents stored at the website and each URL is associated with two or more of;

a last modification date for the URL, a change frequency of a document specified by the URL, and a priority of the document; and

scheduling documents for crawling in accordance with the updated sitemap information.