Sitemap Generating Client for Web Crawler

• US 20100318508A1
• Filed: 08/23/2010
• Published: 12/16/2010
• Est. Priority Date: 05/31/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Maximilian Ibel, Ralph Michael Keller, Sascha B. Brawer, Shivakumar Narayanan
• Inventors: Shivakumar, Narayanan, Ibel, Maximilian, Brawer, Sascha B., Keller, Ralph Michael
• Granted Patent: US 8,037,055 B2
• Time in Patent Office: Days
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and systems for a sitemap generating client for web crawlers are described. The client accesses one or more sources of document information about the documents available on a website, such as the file system, access logs, or pre-made URL lists. Document information is extracted from the sources and one or more sitemaps are generated based on the extracted document information. A notification is transmitted to a remote computer, informing that the sitemap(s) are available for access and likely have been updated. If the remote computer is associated with a web crawler, the remote computer may access the sitemap(s) and use the sitemaps to schedule a crawl of documents included or available on the website.

Claims

1. A method of listing documents performed by a website server system having one or more processors and memory storing one or more programs for execution by the one or more processors, comprising:

accessing one or more sources of document information, wherein the one or more sources of document information are associated with a website server;

extracting the document information including metadata from the sources;

generating a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information;

storing the sitemap at a location; and

transmitting a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access.

2. The method of claim 1, wherein the document metadata information comprises document modification date information associated with the plurality of documents.

3. The method of claim 1, wherein the document metadata information comprises document access frequency information associated with the plurality of documents.

4. The method of claim 1, wherein the document metadata information comprises document priority information associated with the plurality of documents, wherein the document priority information indicates a crawling priority.

5. The method of claim 1, wherein the document metadata information comprises document update rate information associated with the plurality of documents.

6. The method of claim 1, wherein the sitemap metadata provides information for at least one of:

prioritizing documents for crawling by a web crawler, and selecting documents for inclusion in a crawl by the web crawler.

7. The method of claim 1, wherein the sources of document information comprise at least one of the group consisting of:

a file system, one or more access logs, and one or more document location lists.

8. The method of claim 1, wherein the document information comprises document location information and the plurality of documents are accessible to other computers via a network.

9. The method of claim 1, wherein generating the sitemap comprises generating a list of documents modified after a particular time.

10. The method of claim 1, further comprising generating a plurality of sitemaps, and generating an index referencing the plurality of sitemaps;

wherein the notification identifies the index.

11. The method of claim 1, wherein the sitemap comprises a current sitemap, the method further comprising:

determining a difference between the current sitemap and a prior sitemap; and

generating a differential sitemap based on the difference.

12. A system for listing documents, comprising:

one or more processors and memory, the memory comprising one or more sources of document information; and

one or more modules including instructions to;

access the sources of document information, wherein the sources are associated with a website server;

extract the document information including metadata from the sources;

generate a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information;

store the sitemap at a location; and

transmit a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access.

13. The system of claim 12, wherein the document metadata information comprises document modification date information associated with the plurality of documents,

14. The system of claim 12, wherein the document metadata information comprises document access frequency information associated with the plurality of documents.

15. The system of claim 12, wherein the document metadata information comprises document priority information associated with the plurality of documents, wherein the document priority information indicates a crawling priority.

16. The system of claim 12, wherein the document metadata information comprises document update rate information associated with the plurality of documents.

17. The system of claim 12, wherein the document information comprises document location information, and the plurality of documents are accessible to other computers via a network.

18. The system of claim 12, wherein the instructions to generate the sitemap include instructions to generate a list of documents modified after a particular time.

19. The system of claim 12, wherein the one or more modules further include instructions to generate a plurality of sitemaps, and to generate an index referencing the plurality of sitemaps;

wherein the notification identifies the index.

20. The system of claim 12, wherein the sitemap comprises a current sitemap, the one or more modules further including instructions to:

determine a difference between the current sitemap and a prior sitemap; and

generate a differential sitemap based on the difference.

21. A non-transitory computer readable storage medium and one or more computer programs embedded therein, the computer programs comprising instructions, which when executed by a computer system, cause the computer system to:

access one or more sources of document information, wherein the sources are associated with a website server;

extract the document information including metadata from the sources;

generate a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information;

store the sitemap at a location; and

transmit a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access.

22. The computer readable storage medium of claim 21, wherein the document metadata information comprises document modification date information associated with the plurality of documents.

23. The computer readable storage medium of claim 21, wherein the document metadata information comprises document access frequency information associated with the plurality of documents.

24. The computer readable storage medium of claim 21, wherein the document metadata information comprises document priority information associated with the plurality of documents, wherein the document priority information indicates a crawling priority.

25. The computer readable storage medium of claim 21, wherein the document metadata information comprises document update rate information associated with the plurality of documents.

26. The computer readable storage medium of claim 21, wherein the document information comprises document location information and the plurality of documents are accessible to other computers via a network.

27. The computer readable storage medium of claim 21, wherein the instructions, which when executed by a computer system, cause the computer system to generate the sitemap comprise instructions for generating a list of documents modified after a particular time.

28. The computer readable storage medium of claim 21, further comprising instructions, which when executed by a computer system, cause the computer system to generate a plurality of sitemaps, and generating an index referencing the plurality of sitemaps;

wherein the notification identifies the index.

29. The computer readable storage medium of claim 21, wherein the sitemap comprises a current sitemap, the computer programs further comprising instructions, which when executed by a computer system, cause the computer system to:

determine a difference between the current sitemap and a prior sitemap; and

generate a differential sitemap based on the difference.

30. A system for listing documents, comprising:

one or more processors and memory, the memory comprising one or more sources of document information;

means for accessing the sources of document information, wherein the sources are associated with a website server;

means for extracting the document information including metadata from the sources;

means for generating a sitemap of a website at the website server, the sitemap including a list of documents and corresponding metadata for each of a plurality of documents in the list of documents based on the document information;

means for storing the sitemap at a location; and

means for transmitting a notification from the website server to a remote computer associated with a web crawler system, the notification including information that identifies the location of the sitemap, the notification functioning as an indication that the sitemap is available for access.