Search engine with multiple crawlers sharing cookies

• US 7,546,370 B1
• Filed: 08/18/2004
• Issued: 06/09/2009
• Est. Priority Date: 08/18/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Louz-On, Michal, Roetter, Alexander C., Acharya, Anurag
• Primary Examiner(s): Abdullahi; Salad
• Assistant Examiner(s): Hussain; Imad
• Application Number:
  US10/921,378
• Time in Patent Office: 1,756 Days
• Field of Search: 707/3, 707/E17, 709/200, 709/203, 709/223, 709/227, 715/745
• US Class Current: 709/227
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A web-crawler system includes a plurality of network crawlers configured to fetch documents from hosts on a network and a cookie database shared by the plurality of network crawlers. The cookie database stores cookies and associated information for use by the plurality of network crawlers. Each network crawler is configured to retrieve one or more cookies from the cookie database so as to enable access to documents on at least one of the hosts on the network. In some embodiments, each of the network crawlers may be configured to detect any of a plurality of predefined cookie errors associated with fetching a document. In some embodiments, each of the network crawlers may also be configured to detect when a cookie in the cookie database has expired and to obtain a replacement cookie.

Claims

1. A web crawler system, comprising:

a plurality of network crawlers each including, one or more processors and memory storing one or more modules to be executed by the one or more processors, the one or more modules having instructions for fetching documents from hosts on a network; and

a cookie database shared by the plurality of network crawlers, the cookie database storing cookies and associated information for use by the plurality of network crawlers;

wherein each network crawler of the plurality of network crawlers further includes instructions for retrieving one or more cookies from the cookie database so as to enable access to documents on at least one of the hosts on the network and each of the network crawlers includes instructions for detecting any of a plurality of predefined cookie errors associated with fetching a document by comparing a fetched document with a plurality of predefined cookie error patterns; and

wherein the cookie database includes cookie acquisition information corresponding to each of at least a plurality of the cookies in the cookie database;

the cookie acquisition information for a respective cookie enabling a respective network crawler to acquire the cookie from an acquisition URL specified by the cookie acquisition information;

wherein the acquisition URL is distinct from a target URL to be accessed using the respective cookie.

2. The web-crawler system of claim 1, wherein the cookie database includes cookie acquisition information corresponding to each cookie in the cookie database.

3. The web-crawler system of claim 1, wherein the web-crawler system includes a history log for storing at least a cookie error associated with a respective document on at least one of the hosts.

4. The web-crawler system of claim 1, wherein the cookie database includes timeout information for each cookie.

5. The web-crawler system of claim 1, wherein the cookie database includes timeout information for at least a plurality of the cookies in the cookie database.

6. The web-crawler system of claim 1, wherein each of the network crawlers includes instructions for:

detecting when a cookie in the cookie database has expired and;

obtaining a replacement cookie for the cookie that has expired, using the acquisition URL for the cookie that has expired.

7. A method of crawling documents on a network, comprising:

providing a plurality of network crawlers configured to fetch documents from hosts on the network; and

at each of the network crawlers, retrieving a respective cookie for a respective host from a shared cookie database that is shared by the plurality of network crawlers, so as to enable the plurality of network crawlers to have access to one or more documents on the respective host;

wherein the cookie database includes cookie acquisition information corresponding to each of at least a plurality of the cookies in the cookie database;

the cookie acquisition information for a respective cookie including an acquisition URL;

the method including,

at a respective network crawler, acquiring a respective cookie from the acquisition URL specified for the cookie in the cookie database, and then accessing a respective target URL from a host on the network, the respective target URL corresponding to the acquired cookie; and

at each of the network crawlers, detecting any of a plurality of predefined cookie errors by comparing a fetched document with a plurality of predefined cookie error patterns.

8. The method of crawling a network in claim 7, including, at each of the network crawlers, determining if the respective cookie has timed out.

9. The method of crawling a network in claim 8, including obtaining a replacement cookie using at least one of the network crawlers if the respective cookie has timed out, and updating the cookie database with the replacement cookie;

wherein the replacement cookie is obtained using the acquisition URL for the respective cookie that has timed out.

10. The method of crawling a network in claim 8, including storing in a history log at least a cookie error associated with at least a document on at least one of the hosts.

11. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising a network crawler module to be executed by a plurality of computer network crawlers in parallel, the network crawler module including instructions for:

retrieving a respective cookie for a respective host from a shared cookie database that is shared by the plurality of network crawlers, so as to enable the plurality of network crawlers to have access to one or more documents on the respective host; and

retrieving a document from the respective host, including sending the respective cookie to the respective host;

wherein the shared cookie database includes cookie acquisition information corresponding to each of at least a plurality of the cookies in the cookie database;

the cookie acquisition information for a respective cookie including an acquisition URL;

the network crawler module further including

instructions for acquiring a respective cookie from the acquisition URL specified for the cookie in the cookie database, and then accessing a respective target URL from a host on the network, the respective target URL corresponding to the acquired cookie; and

instructions for detecting any of a plurality of predefined cookie errors by comparing a fetched document with a plurality of predefined cookie error patterns.