Seeding search engine crawlers using intercepted network traffic

• US 20090313241A1
• Filed: 06/16/2008
• Published: 12/17/2009
• Est. Priority Date: 06/16/2008
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Elsen, Christian
• Granted Patent: US 8,832,052 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method includes monitoring data packets exchanged in a computer network over which documents having respective location identifiers are distributed, so as to detect a request to access a given document. A location identifier of the given document is extracted from the request. The location identifier is provided to a search engine that searches for data in a set of the documents, so as to cause the search engine to add the given document to the set.

Claims

1. A method, comprising:

monitoring data packets exchanged in a computer network over which documents having respective location identifiers are distributed, so as to detect a request to access a given document;

extracting a location identifier of the given document from the request; and

providing the location identifier to a search engine that searches for data in a set of the documents, so as to cause the search engine to add the given document to the set.

2. The method according to claim 1, and comprising, after the given document is added to the set, applying the search engine to search for the data in the set including the given document.

3. The method according to claim 1, wherein one or more documents in the set contain at least one link pointing to at least one additional document, and wherein the search engine incrementally expands the set by following the at least one link in the at least one document and adding the at least one additional document to the set.

4. The method according to claim 3, wherein providing the location identifier comprises adding to the set at least one document that is reachable by following one or more links in the given document but that cannot be reached by following the links in the documents in the set.

5. The method according to claim 1, wherein the given document is already in the set when the location identifier is provided to the search engine, and wherein providing the location identifier comprises causing the search engine to search for the data in the given document irrespective of whether the given document has already been searched.

6. The method according to claim 1, wherein providing the location identifier comprises evaluating a condition related to the extracted location identifier, and providing the location identifier only responsively to meeting the condition.

7. The method according to claim 6, wherein evaluating the condition comprises defining the condition responsively to a time interval between detection of the request and the detection of a previous request to access the given document.

8. The method according to claim 1, wherein monitoring the data packets comprises detecting multiple requests to access the given document, and wherein providing the location identifier comprises reporting an amount of the multiple requests to the search engine.

9. The method according to claim 1, wherein the documents comprise Web pages.

10. The method according to claim 1, wherein the request comprises a Hyper-Text Transfer Protocol (HTTP) request.

11. The method according to claim 1, wherein the location identifier comprises a Uniform Resource Locator (URL).

12. Apparatus, comprising:

an interface, which is operative to communicate with a computer network over which documents having respective location identifiers are distributed; and

a processor, which is configured to monitor data packets exchanged in the computer network so as to detect a request to access a given document stored in the computer network, to extract a location identifier of the given document from the request, and to provide the location identifier to a search engine that searches for data in a set of the documents so as to cause the search engine to add the given document to the set.

13. The apparatus according to claim 12, wherein the given document is already in the set when the location identifier is provided to the search engine, and wherein the processor is configured to cause the search engine to search for the data in the given document irrespective of whether the given document has already been searched.

14. The apparatus according to claim 12, wherein the processor is configured to evaluate a condition related to the extracted location identifier, and to provide the location identifier only responsively to meeting the condition.

15. The apparatus according to claim 14, wherein the processor is configured to define the condition responsively to a time interval between detection of the request and the detection of a previous request to access the given document.

16. The apparatus according to claim 12, wherein the processor is configured to detect multiple requests to access the given document, and to report an amount of the multiple requests to the search engine.

17. The apparatus according to claim 12, wherein the documents comprise Web pages.

18. The apparatus according to claim 12, wherein the request comprises a Hyper-Text Transfer Protocol (HTTP) request.

19. The apparatus according to claim 12, wherein the location identifier comprises a Uniform Resource Locator (URL).

20. A system, comprising:

a network element, which is configured to monitor data packets exchanged in a computer network over which documents having respective location identifiers are distributed, to detect in the monitored data packets a request to access a given document stored in the computer network, to extract a location identifier of the given document from the request and to send the extracted location identifier over the computer network; and

a search engine, which is coupled to search for data in a set of the documents, to accept the location identifier from the network element and, responsively to the location identifier, to add the given document to the set.