Mobile sitemaps

• US 7,653,617 B2
• Filed: 05/01/2006
• Issued: 01/26/2010
• Est. Priority Date: 08/29/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Shivakumar, Narayanan, Hu, Feng, Ibel, Maximilian, Brawer, Sascha B., Gil, Elad, Keller, Ralph M., Strohm, Alan C.
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US11/415,947
• Publication Number:
  US 20070050338A1
• Time in Patent Office: 1,366 Days
• Field of Search: 707/2, 707/3, 707/10, 707/200, 725/87, 709/237
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A method of analyzing documents or relationships between documents includes receiving a notification of an available metadata document containing information about one or more network-accessible documents, obtaining a document format indicator associated with the metadata document, selecting a document crawler using the document format indicator, and crawling at least some of the network-accessible documents using the selected document crawler.

Claims

1. A computer-implemented method of analyzing documents or relationships between documents, comprising:

receiving a notification of an available metadata document containing information about one or more network-accessible documents;

obtaining a document format indicator associated with the metadata document, the document format indicator specifying a format in which content of at least one of the network-accessible documents is stored;

selecting, using the document format indicator, a document crawler having an operating mode that defines one or more content formats that the document crawler is capable of accessing, including the format specified by the document format indicator; and

crawling with a computer at least some of the network-accessible documents using the selected document crawler and operating mode.

2. The method of claim 1, wherein the one or more network-accessible documents comprise a plurality of web pages at a common domain.

3. The method of claim 1, wherein the metadata document comprises a list of document identifiers.

4. The method of claim 3, wherein the one or more network-accessible documents comprise a plurality of web pages at a common domain.

5. The method of claim 1, wherein the document format indicator indicates one or more mobile content formats.

6. The method of claim 5, wherein the mobile content formats are selected from the group consisting of XHTML, WML, iMode, and HTML.

7. The method of claim 1, further comprising adding information retrieved from crawling at least some of the network-accessible documents to an index.

8. The method of claim 7, further comprising receiving a search request from a mobile device and transmitting search results to the mobile device using information in the index.

9. The method of claim 1, wherein the available metadata document comprises an index referencing a plurality of lists of documents.

10. The method of claim 1, further comprising receiving an indication of document type for the one or more network-accessible documents and classifying the documents using the indication of document type.

11. The method of claim 10, further comprising verifying the identity of a provider of the indication of document type to ensure that the provider is trusted.

12. The method of claim 10, wherein the document type is selected from a group consisting of news, entertainment, commerce, sports, travel, games, and finance.

13. A system for crawling network-accessible documents, comprising:

a memory storing organizational information about network-accessible documents at one or more websites, and format information for the documents;

a crawler configured to access the network-accessible documents using the organizational information; and

a format selector associated with the crawler to cause the crawler to assume a persona compatible with formats indicated by the format information.

14. The system of claim 13, wherein the organizational information comprises a list of URLs.

15. The system of claim 13, further comprising an agent repository that stores parameters for causing the crawler to assume a selected persona.

16. A system for crawling network-accessible documents, comprising:

a memory storing organizational information about network-accessible documents at one or more websites, and format information for the documents;

a crawler configured to access the network-accessible documents using the organizational information; and

means for selecting a crawler persona to present in accessing the network-accessible documents.