Method and system for downloading network data at a controlled data transfer rate

• US 7,401,155 B2
• Filed: 01/16/2004
• Issued: 07/15/2008
• Est. Priority Date: 04/19/2000
• Status: Expired due to Term

• Current Assignee: Benhov GmbH, LLC (Intellectual Ventures LLC)
• Original Assignee: Archeron Ltd LLC (Intellectual Ventures LLC)
• Inventors: Cooper, Jeremy S, Foulger, Michael G
• Primary Examiner(s): Jaroenchonwanit; Bunjob
• Assistant Examiner(s): Chang; Julian
• Application Number:
  US10/758,282
• Publication Number:
  US 20040210589A1
• Time in Patent Office: 1,642 Days
• Field of Search: 709/230, 709/231, 709232-235
• US Class Current: 709/233
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method and system for retrieving web-site based information by a spider engine at a target bandwidth is described. A target band width is received from the spider engine. A wait time is calculated by a timing module. Data retrieval from a web site is delayed by the calculated wait time so that data is retrieved at the desired target bandwidth.

Claims

1. A method of retrieving data over a network at a target bandwidth, B_T, comprising:

(1) transmitting a request for data to a server over the network;

(2) receiving the data from the server over the network;

(3) calculating a wait time based on the target bandwidth and an aggregate bytes count, bytes_AGG, wherein bytes_AGG is an aggregate number of bytes received from the server, and wherein step (3) includes

(A) determining a start time, T_START, based on the request for data of step (1),

(B) incrementing the aggregate bytes count, bytes_AGG, by the number of bytes received in step (2),

(C) determining a current time, T_NOW, and

(D) calculating the wait time according to the equation

wait time=(bytes_AGG)/B_T−

(T_NOW−

T_START);

(4) waiting the calculated wait time;

(5) transmitting a request for additional data to the server over the network after step (4); and

(6) receiving the additional data from the server over the network.

2. The method of claim 1, further comprising:

(7) repeating steps (4)-(6) for further additional data until all desired data is retrieved from the server over the network.

3. The method of claim 1, wherein step (3) further includes

(E) repeating steps (1) and (2) a plurality of times for additional data;

wherein step (B) includes incrementing the aggregate bytes count, bytes_AGG, by the number of bytes received in each step (2).

4. A computer system, comprising:

transmitting means for transmitting requests for data to a server over a network;

receiving means for receiving said data from said server over said network; and

a timing module that calculates a wait time based on an aggregate bytes count, bytes_AGG, and a target bandwidth, B_T, at which rate data is desired to be retrieved from said server over said network, wherein bytes_AGG is an aggregate number of bytes received from said server;

wherein said transmitting means delays transmitting requests for data to said server over said network by said calculated wait time;

wherein said timing module determines a start time, T_START, corresponding to when said transmitting means transmits a request for data to said server over said network;

wherein said timing module detects a number of bytes received by said receiving means due to a transmitted request;

wherein said timing module increments said aggregate bytes count, bytes_AGG, by said number of bytes received;

wherein said timing module determines a current time, T_NOW, after at least one iteration of said timing module detecting a number of bytes received by said receiving means due to a transmitted request; and

wherein said timing module calculates said wait time according to the equation

wait time=(bytes_AGG)/B_T−

(T_NOW−

T_START).

5. The computer system of claim 4, wherein said network is the Internet.

6. A computer program product comprising a tangible computer usable storage medium having computer program logic recorded thereon for enabling a processor to retrieve data over a network at a target bandwidth, B_T, comprising:

calculating means for enabling a processor to calculate a wait time based on an aggregate bytes count, bytes_AGG, and a target bandwidth, B_T, at which rate data is desired to be retrieved from a server over said network, wherein bytes_AGG is an aggregate number of bytes received from said server; and

delaying means for enabling a processor to delay transmitting requests for data to said server over said network by said calculating wait time;

determining means for enabling a processor to determine start time, T_START, corresponding to when a request for data is transmitted to said server over said network;

detecting means for enabling a processor to detect a number of bytes received due to a transmitted request;

incrementing means for enabling a processor to increment said aggregate bytes count, bytes_AGG, said number of bytes received; and

determining means for enabling a processor to determine a current time, T_NOW, after at least one iteration of detecting a number of bytes received due to a transmitted request;

wherein calculating means enables a processor to calculate said wait time according to the equation

wait time=(bytes_AGG)/B_T−

(T_NOW−

T_START).

7. The computer program product of claim 6, wherein the network is the Internet.