INCREMENTAL CRAWLING OF MULTIPLE CONTENT PROVIDERS USING AGGREGATION

• US 20090307211A1
• Filed: 12/23/2008
• Published: 12/10/2009
• Est. Priority Date: 06/05/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: RADCHENKO, Constantin, KENIG, Batya, SHAPIRO, Eitan
• Granted Patent: US 8,799,261 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method for incremental crawling of content stored on a plurality of content providers using aggregation is provided. The method comprises receiving a request to crawl content on one or more associated content providers; retrieving one or more first references to content on a first content provider; retrieving one or more second references to content on one or more second content providers during the same request; aggregating the first and second references; and returning the aggregated first and second references. This is done while taking into consideration opaque timestamp object which is managed in a distributed manner. The opaque timestamp is filled in by the content providers but stored in the crawler side between crawling sessions.

Claims

1. A method for incremental crawling of content stored on a plurality of content providers using aggregation, the method comprising:

receiving a request to crawl content on one or more associated content providers;

retrieving one or more first references to content on a first content provider;

retrieving one or more second references to content on one or more second content providers during the same request;

aggregating the first and second references; and

returning the aggregated first and second references.

2. The method of claim 1, further comprising returning state information, wherein the state information allows passing of internal information between the associated content providers.

3. The method of claim 2, wherein the associated content providers do not save the state information, allowing several crawlers to request access to the plurality of content providers.

4. The method of claim 1, wherein the request comprises a threshold value associated with size of content to be accessed on the associated content providers.

5. The method of claim 4, wherein the second references are returned such that the request is satisfied, in response to determining that content on the first content provider is less than the threshold value.

6. The method of claim 4, wherein the aggregated first and second references are returned such that the request is satisfied to a maximum extent possible, in response to determining that the request cannot be fully satisfied by content on the associated content providers.

7. The method of claim 1, wherein the retrieving comprises indexing all content on the associated content providers for future retrieval.

8. The method of claim 1, wherein the retrieving comprises incremental indexing of updated content on the associated content providers for future retrieval.

9. The method of claim 8, further comprising returning a timestamp, wherein the timestamp identifies when the request was submitted.

10. The method of claim 9, wherein the timestamp is used during a subsequent incremental crawling session to access content on the associated content providers updated after the identified time.

12. The method of claim 1, wherein the validity of the state information is determined by comparing the starting index value to the last received index, which is stored in the state information.

11. A method for incremental crawling of content stored on a plurality of content providers using aggregation, the method comprising:

receiving a request to crawl content on one or more associated content providers, wherein the request comprises at least a starting index value and a range value;

forwarding the request to a first content provider on a list, in response to determining that there is no valid state information;

forwarding the request to a first content provider identified by the state information as a next content provider, in response to determining that there is valid state information;

receiving references, state information, or timing information from the first content provider;

aggregating the received references, state information, and timing information with other references, state information, and timing information, respectively;

forwarding the request to a second content provider on the list, in response to determining that the request has not been satisfied to the maximum extent possible;

updating the state information with the next content provider and corresponding next starting index, in response to determining that the request has been satisfied to the maximum extent possible; and

returning the aggregated references, the updated state information, and the aggregated timestamp, if available.

13. The method of claim 11, wherein forwarding the request to a first content provider on a list further comprises:

incrementing the range value by the starting index value; and

passing timing information corresponding to the first content provider, if any, with the request.

14. The method of claim 11, wherein forwarding the request to a first content provider identified by the state information further comprises:

setting the starting index value to a value indicated by the first state information as a next starting index; and

passing the state information and timing information corresponding to the first content provider, if any, with the request.

15. The method of claim 11, wherein forwarding the request to a second content provider on the list further comprises:

decrementing the range value with the number of references received from the first content provider; and

passing timing information corresponding to the second content provider, if any, with the request.

16. The method of claim 11, wherein updating the state information further comprises:

setting the next content provider to be the next content provider on the list, in response to determining that there are no more references left to receive from the first content provider; and

setting the next content provider to be the first content provider, in response to determining that there are more references left to receive from the first content provider.

17. The method of claim 11, wherein the request does not comprise timing information when the request is received during a full crawling session.

18. The method of claim 11, wherein the request comprises timing information when the request is received during an incremental crawling session.

19. The method of claim 18, wherein the timing information is used to crawl content on the associated content providers that have been updated since a previous crawling session.

20. The method of claim 11, wherein determining that the request has been satisfied to the maximum extent possible comprises comparing the starting index value to a last received index value, which is stored in the state information.