Mobile SiteMaps

• US 20100125564A1
• Filed: 01/25/2010
• Published: 05/20/2010
• Est. Priority Date: 08/29/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hu, Feng, Gil, Elad, Keller, Ralph M., Strohm, Alan C., Shivakumar, Narayanan, Ibel, Maximilian, Brawer, Sascha B.
• Granted Patent: US 8,234,266 B2
• Time in Patent Office: Days
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A method of analyzing documents or relationships between documents includes receiving a notification of an available metadata document containing information about one or more network-accessible documents, obtaining a document format indicator associated with the metadata document, selecting a document crawler using the document format indicator, and crawling at least some of the network-accessible documents using the selected document crawler.

Claims

1. A method of analyzing documents or relationships between documents, comprising:

receiving a notification of an available metadata document containing information about one or more network-accessible documents;

obtaining a document format indicator associated with the metadata document;

selecting a document crawler using the document format indicator; and

crawling at least some of the network-accessible documents using the selected document crawler.

2. The method of claim 1, wherein the one or more network-accessible documents comprise a plurality of web pages at a common domain.

3. The method of claim 1, wherein the metadata document comprises a list of document identifiers.

4. The method of claim 3, wherein the one or more network-accessible documents comprise a plurality of web pages at a common domain.

5. The method of claim 1, wherein the document format indicator indicates one or more mobile content formats.

6. The method of claim 5, wherein the mobile content formats are selected from the group consisting of XHTML, WML, iMode, and HTML.

7. The method of claim 1, further comprising adding information retrieved from crawling at least some of the network-accessible documents to an index.

8. The method of claim 7, further comprising receiving a search request from a mobile device and transmitting search results to the mobile device using information in the index.

9. The method of claim 1, wherein the available metadata document comprises an index referencing a plurality of lists of documents.

10. The method of claim 1, further comprising receiving an indication of document type for the one or more network-accessible documents and classifying the documents using the indication of document type.

11. The method of claim 10, further comprising verifying the identity of a provider of the indication of document type to ensure that the provider is trusted.

12. The method of claim 10, wherein the document type is selected from a group consisting of news, entertainment, commerce, sports, travel, games, and finance.

13. A method of listing network-accessible documents, comprising:

generating a mapping document that represents an organization of related network-accessible documents; and

transmitting to a remote computer a notification that includes an indication that the mapping document is available for access and an indication of the format of the documents.

14. The method of claim 13, wherein the mapping document comprises a list of document identifiers.

15. The method of claim 13, wherein the indication of the format of the documents indicates one or more mobile document formats that affect the ability to interpret the documents.

16. The method of claim 13, wherein the notification includes an indication of the location of the mapping document.

17. The method of claim 13, wherein the notification is transmitted when a user fills out a web-based form.

18. A system for crawling network-accessible documents, comprising:

a memory storing organizational information about network-accessible documents at one or more websites, and format information for the documents;

a crawler configured to access the network-accessible documents using the organizational information; and

a format selector associated with the crawler to cause the crawler to assume a persona compatible with formats indicated by the format information.

19. The system of claim 18, wherein the organizational information comprises a list of URLs.

20. The system of claim 18, further comprising an agent repository that stores parameters for causing the crawler to assume a selected persona.

21. A system for crawling network-accessible documents, comprising:

a memory storing organizational information about network-accessible documents at one or more websites, and format information for the documents;

a crawler configured to access the network-accessible documents using the organizational information; and

means for selecting a crawler persona to present in accessing the network-accessible documents.

22. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising instructions for:

generating a mapping document that represents an organization of related network-accessible documents; and

transmitting to a remote computer a notification that includes an indication that the list is available for access and an indication of the format of the documents.

23. The computer program product of claim 22, wherein the mapping document comprises a list of document identifiers.

24. The computer program product of claim 22, wherein the indication of the format of the documents indicates one or more mobile document formats that affect the ability to interpret the documents.

25. The computer program product of claim 22, wherein the notification includes an indication of the location of the mapping document.

26. The computer program product of claim 22, wherein the notification is transmitted when a user fills out a web-based form.