Regulating rates of requests by a spider engine to web sites by creating instances of a timing module

• US 6,681,255 B1
• Filed: 04/19/2000
• Issued: 01/20/2004
• Est. Priority Date: 04/19/2000
• Status: Expired due to Term

• Current Assignee: Benhov GmbH, LLC (Intellectual Ventures LLC)
• Original Assignee: Icplanet Corporation
• Inventors: Foulger, Michael G., Cooper, Jeremy S.
• Primary Examiner(s): EL HADY, NABIL M
• Application Number:
  US09/552,559
• Time in Patent Office: 1,371 Days
• Field of Search: 709/230-235
• US Class Current: 709/232
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method and system for retrieving web-site based information by a spider engine at a target bandwidth is described. A target band width is received from the spider engine. A wait time is calculated by a timing module. Data retrieval from a web site is delayed by the calculated wait time so that data is retrieved at the desired target bandwidth.

Claims

1. A method of retrieving web-site based information using a spider engine at a target bandwidth, comprising the steps of:

(1) receiving a target bandwidth, B_T, at which rate data is desired to be retrieved from a remote web-site to a local processor, wherein the data desired to be retrieved is allocated in storage at the remote web-site in a manner not known to the local processor;

(2) calculating a wait time, T_WAIT; and

(3) delaying data retrieval from the remote web-site to the local processor by said calculated wait time to retrieve at the target bandwidth, B_T, wherein an instance of a timing module is created by the spider engine to calculate said wait time; and

wherein data retrieved to the local processor from the remote web-site passes through the Internet.

2. The method of claim 1, wherein step (2) comprises the steps of:

(A) calculating a start time, T_START;

(B) initiating retrieval of data from the remote web-site across the Internet;

(C) detecting a number of bytes received;

(D) incrementing an aggregate bytes count, bytes_AGG, by the number of bytes received;

(E) calculating a current time, T_NOW; and

(F) calculating the wait time, T_WAIT.

3. The method of claim 2, wherein step (F) comprises calculating T_WAIT according to the equation:

4. The method of claim 2, wherein step (D) comprises the steps of:

(i) incrementing the aggregate bytes count, bytes_AGG, by the number of bytes received; and

(ii) returning to step (B).

5. The method of claim 2, wherein step (1) comprises the step of:

(A) passing the target bandwidth, B_T, to the timing module from the spider engine.

6. The method of claim 5, further comprising the steps of:

(5) implementing steps (2)(A), (2)(C), (2)(D), (2)(E), and (2)(F) in the timing module; and

(6) implementing steps (2)(B) and (3) in the spider engine.

7. The method of claim 6, wherein step (2) further comprises the step of:

(G) passing the calculated wait time, T_WAIT, from the timing module to the spider engine.

8. The method of claim 2, further comprising the step of:

(4) creating a plurality of instances of a timing module with a spider engine.

9. A system for retrieving web-site based information using a spider engine at a target bandwidth, comprising:

receiving means for receiving a target bandwidth, B_T, at which rate data is desired to be retrieved from a remote web-site to a local processor, wherein the data desired to be retrieved is allocated in storage at the remote web-site in a manner not known to the local processor;

calculating means for calculating a wait time, T_WAIT; and

delaying means for delaying data retrieval from the remote web-site to the local processor by the calculated wait time so that data is retrieved at the desired target bandwidth, B_T, wherein an instance of a timing module is created by the spider engine to calculate said wait time; and

wherein data retrieved to the local processor from the remote web-site passes through the Internet.

10. The system of claim 9, wherein said calculating means comprises:

means for calculating a start time, T_START;

means for initiating retrieval of data from the remote web-site across the Internet;

means for detecting a number of bytes received;

means for incrementing an aggregate bytes count, bytes_AGG, by the number of bytes received;

means for calculating a current time, T_NOW; and

wait time calculating means for calculating wait time, T_WAIT.

11. The system of claim 10, wherein said wait time calculating means comprises means for calculating T_WAIT according to the equation:

12. A timing system for retrieving web-site based information using a spider engine at a target bandwidth, comprising:

a data receiver for receiving a target bandwidth, B_T, at which rate data is desired to be retrieved from a remote web-site to a local processor, and at least one bytes count from the spider engine, wherein the data desired to be retrieved is allocated in storage at the remote web-site in a manner not known to the local processor;

a bytes accumulator for accumulating said at least one bytes count received from the spider engine to create an aggregate bytes count, bytes_AGG;

a current time determiner for determining a start time, T_START, and current time, T_NOW, for said at least one received bytes count;

a wait time calculator; and

a wait time transmitter for transmitting a wait time, T_WAIT, calculated by said wait time calculator to the spider engine;

wherein said wait time is the amount of time the spider engine should wait to initiate a next web-site data retrieval from the remote web-site to the local processor to reach said target bandwidth, wherein data retrieved to the local processor from the remote web-site passes through the Internet;

wherein said wait time calculator calculates said wait time as a function of said bytes_AGG, B_T, and an elapsed time (T_NOW−

T_START).

13. The system of claim 12, wherein said wait time, T_WAIT, is calculated according to: