Regulating rates of requests by a spider engine to web sites by creating instances of a timing module

• US 20040210589A1
• Filed: 01/16/2004
• Published: 10/21/2004
• Est. Priority Date: 04/19/2000
• Status: Active Grant

• Current Assignee: Benhov GmbH, LLC (Intellectual Ventures LLC)
• Original Assignee: Archeron Ltd LLC (Intellectual Ventures LLC)
• Inventors: Foulger, Michael G., Cooper, Jeremy S.
• Granted Patent: US 7,401,155 B2
• Time in Patent Office: Days
• US Class Current: 707/100
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method and system for retrieving web-site based information by a spider engine at a target bandwidth is described. A target band width is received from the spider engine. A wait time is calculated by a timing module. Data retrieval from a web site is delayed by the calculated wait time so that data is retrieved at the desired target bandwidth.

Claims

1. A method of retrieving web-site based information at a target bandwidth, comprising the steps of:

(1) receiving a target bandwidth, B_T;

(2) calculating a wait time, T_WAIT; and

(3) delaying data retrieval by said calculated wait time to retrieve at the target bandwidth, B_T.

2. The method of claim 1, wherein step (2) comprises the steps of:

(A) calculating a start time, T_START;

(B) initiating retrieval of data from a remote web-site across a network;

(C) detecting a number of bytes received;

(D) incrementing an aggregate bytes count, bytes_AGG, by the number of bytes received;

(E) calculating a current time, T_NOW; and

(F) calculating the wait time, T_WAIT.

3. The method of claim 2, wherein step (F) comprises calculating T_WAIT according to the equation:

T_WAIT=(bytes_AGG)/B_T−

(T_NOW−

T_START)

4. The method of claim 2, wherein step (D) comprises the steps of:

(i) incrementing the aggregate bytes count, bytes_AGG, by the number of bytes received; and

(ii) returning to step (B).

5. The method of claim 2, further comprising the step of:

(4) creating an instance of a timing module with a spider engine.

6. The method of claim 2, further comprising the step of:

(4) creating a plurality of instances of a timing module with a spider engine.

7. The method of claim 5, wherein step (1) comprises the step of:

(A) passing the target bandwidth, B_T, to the timing module from the spider engine.

8. The method of claim 7, wherein step

(5) implementing steps (2)(A), (2)(C), (2)(D), (2)(E), and (2)(F) in the timing module;

and

(6) implementing steps (2)(B) and (3) in the spider engine.

9. The method of claim 8, wherein step (2) further comprises the step of:

(G) passing the calculated wait time, T_WAIT, from the timing module to the spider engine.

10. A system for retrieving web-site based information at a target bandwidth, comprising:

receiving means for receiving a target bandwidth, B_T;

calculating means for calculating a wait time, T_WAIT; and

delaying means for delaying data retrieval by the calculated wait time so that data is retrieved at the desired target bandwidth, B_T.

11. The system of claim 10, wherein said calculating means comprises:

means for calculating a start time, T_START;

means for initiating retrieval of data from a remote web-site across a network;

means for detecting a number of bytes received;

means for incrementing an aggregate bytes count, bytes_AGG, by the number of bytes received;

means for calculating a current time, T_NOW; and

wait time calculating means for calculating wait time, T_WAIT.

12. The system of claim 11, wherein said wait time calculating means comprises means for calculating T_WAIT according to the equation:

T_WAIT=(bytes_AGG)/B_T−

(T_NOW−

T_START)

13. A timing system for retrieving web-site based information using a spider engine at a target bandwidth, comprising:

a data receiver for receiving a target bandwidth, B_T, and at least one bytes count from the spider engine;

a bytes accumulator for accumulating said at least one bytes count received from the spider engine to create an aggregate bytes count, bytes_AGG;

a current time determiner for determining a start time, T_START, and current time, T_NOW, for said at least one received bytes count;

a wait time calculator; and

a wait time transmitter for transmitting a wait time, T_WAIT, calculated by said wait time calculator to the spider engine;

wherein said wait time is the amount of time the spider engine should wait to initiate a next web-site data retrieval to reach said target bandwidth;

wherein said wait time calculator calculates said wait time as a function of said bytes_AGG, B_T, and an elapsed time (T_NOW−

T_START).

14. The system of claim 13, wherein said wait time, T_WAIT, is calculated according to:

T_WAIT=(bytes_AGG)/B_T−

(T_NOW−

T_START).