Timing Module for Regulating Hits by a Spidering Engine

• US 20080270604A1
• Filed: 07/10/2008
• Published: 10/30/2008
• Est. Priority Date: 04/19/2000
• Status: Active Grant

• Current Assignee: Archeron Ltd LLC (Intellectual Ventures LLC)
• Original Assignee: Archeron Ltd LLC (Intellectual Ventures LLC)
• Inventors: Foulger, Michael G., Cooper, Jeremy S.
• Granted Patent: US 7,949,748 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method and system for retrieving web-site based information by a spider engine at a target bandwidth is described. A target band width is received from the spider engine. A wait time is calculated by a timing module. Data retrieval from a web site is delayed by the calculated wait time so that data is retrieved at the desired target bandwidth.

Claims

1. A method for determining a time between transmitting requests for data over a network to a server, comprising:

establishing, at a processor, a time at which an initial request for the data is transmitted over the network to the server;

determining, at the processor, a count of a number of bytes of the data received over the network from the server in response to the initial request;

establishing, at the processor, a current time, wherein the current time is other than the time at which the initial request for the data is transmitted; and

calculating, at the processor, the time between transmitting the requests, wherein the time between transmitting the requests is equal to a first difference, the first difference is equal to a second difference subtracted from a quotient, the second difference is equal to the time at which the initial request for the data is transmitted subtracted from the current time, and the quotient is equal to the count divided by a target rate of transfer of the data measured in bytes per unit of time.

2. The method of claim 1, further comprising adjusting the time between transmitting the requests to mimic substantially a modem transfer rate.

3. The method of claim 1, further comprising adjusting the time between transmitting the requests as a function of a time of day.

4. An apparatus for determining a time between transmitting requests for data over a network to a server, comprising:

an input configured to receive the data over the network from the server;

an output configured to transmit the requests for the data over the network to the server; and

a processor configured to establish a time at which an initial request for the data is transmitted over the network to the server, to determine a count of a number of bytes of the data received over the network from the server in response to the initial request, to establish a current time, wherein the current time is other than the time at which the initial request for the data is transmitted, and to calculate the time between transmitting the requests, wherein the time between transmitting the requests is equal to a first difference, the first difference is equal to a second difference subtracted from a quotient, the second difference is equal to the time at which the initial request for the data is transmitted subtracted from the current time, and the quotient is equal to the count divided by a target rate of transfer of the data measured in bytes per unit of time.

5. The apparatus of claim 4, wherein the processor is further configured to adjust the time between transmitting the requests to mimic substantially a modem transfer rate.

6. The apparatus of claim 4, wherein the processor is further configured to adjust the time between transmitting the requests as a function of a time of day.

7. A computer program product for determining a time between transmitting requests for data over a network to a server, the computer program product having computer program code embodied in a computer-readable storage medium, the computer program code comprising:

first program code configured to cause a processor to establish a time at which an initial request for the data is transmitted over the network to the server;

second program code configured to cause the processor to determine a count of a number of bytes of the data received over the network from the server in response to the initial request;

third program code configured to cause the processor to establish a current time, wherein the current time is other than the time at which the initial request for the data is transmitted; and

fourth program code configured to cause the processor to calculate the time between transmitting the requests, wherein the time between transmitting the requests is equal to a first difference, the first difference is equal to a second difference subtracted from a quotient, the second difference is equal to the time at which the initial request for the data is transmitted subtracted from the current time, and the quotient is equal to the count divided by a target rate of transfer of the data measured in bytes per unit of time.

8. The computer program product of claim 7, further comprising fifth program code configured to cause the processor to adjust the time between transmitting the requests to mimic substantially a modem transfer rate.

9. The computer program product of claim 7, further comprising fifth program code configured to cause the processor to adjust the time between transmitting the requests as a function of a time of day.

10. The computer program product of claim 7, further comprising:

fifth program code configured to cause the processor to transmit the requests for the data over the network to the server; and

sixth program code configured to cause the processor to transmit the time between transmitting the requests to the fifth program code.

11. A method for retrieving data over a network at a target bandwidth, comprising:

determining, at a local processor, the target bandwidth, wherein the target bandwidth is a desired rate data for retrieving the data from a remote sever via the Internet to the local processor and wherein the data is stored at the remote server in a manner unknown to the local processor;

determining, at the local processor, a time at which retrieval of the data from the remote server to the local processor begins;

determining, at the local processor, a time at which retrieval of the data from the remote server to the local processor ends;

determining, at the local processor, a byte count of the data; and

calculating, at the local processor, a time span between instances of data requests from the local processor to the remote server, wherein the time span is a function of the target bandwidth, the time at which retrieval of the data from the remote server to the local processor begins, the time at which retrieval of the data from the remote server to the local processor ends, and the byte count.

12. The method of claim 11, further comprising adjusting the time span to mimic substantially a modem transfer rate.

13. The method of claim 11, further comprising adjusting the time span as a function of a time of day.

14. The method of claim 11, wherein the function is equal to a first difference, the first difference is equal to a second difference subtracted from a quotient, the second difference is equal to the time at which retrieval of the data from the remote server to the local processor begins subtracted from the time at which retrieval of the data from the remote server to the local processor ends, and the quotient is equal to the byte count divided by the target bandwidth.

15. An apparatus for retrieving data over a network at a target bandwidth, comprising:

an input configured to receive the data from a remote sever via the Internet to a local processor, wherein the data is stored at the remote server in a manner unknown to the local processor; and

a processor configured to determine the target bandwidth, wherein the target bandwidth is a desired rate data for retrieving the data from the remote sever via the Internet to the local processor, to determine a time at which retrieval of the data from the remote server to the local processor begins, to determine a time at which retrieval of the data from the remote server to the local processor ends, to determine a byte count of the data, and to calculate a time span between instances of data requests from the local processor to the remote server, wherein the time span is a function of the target bandwidth, the time at which retrieval of the data from the remote server to the local processor begins, the time at which retrieval of the data from the remote server to the local processor ends, and the byte count.

16. The apparatus of claim 15, wherein the processor is further configured to adjust the time span to mimic substantially a modem transfer rate.

17. The apparatus of claim 15, wherein the processor is further configured to adjust the time span as a function of a time of day.

18. A computer program product for retrieving data over a network at a target bandwidth, the computer program product having computer program code embodied in a computer-readable storage medium, the computer program code comprising:

first program code configured to cause a processor to determine the target bandwidth, wherein the target bandwidth is a desired rate data for retrieving the data from a remote sever via the Internet to the local processor and wherein the data is stored at the remote server in a manner unknown to the local processor;

second program code configured to cause the processor to determine a time at which retrieval of the data from the remote server to the local processor begins;

third program code configured to cause the processor to determine a time at which retrieval of the data from the remote server to the local processor ends;

fourth program code configured to cause the processor to determine a byte count of the data; and

fifth program code configured to cause the processor to calculate a time span between instances of data requests from the local processor to the remote server, wherein the time span is a function of the target bandwidth, the time at which retrieval of the data from the remote server to the local processor begins, the time at which retrieval of the data from the remote server to the local processor ends, and the byte count.

19. The computer program product of claim 18, further comprising sixth program code configured to cause the processor to adjust the time span to mimic substantially a modem transfer rate.

20. The computer program product of claim 18, further comprising sixth program code configured to cause the processor to adjust the time span as a function of a time of day.

21. The computer program product of claim 18, further comprising:

sixth program code configured to cause the processor to transmit a request for the data via the Internet to the remote server; and

seventh program code configured to cause the processor to transmit the time span to the sixth program code.

22. An apparatus for retrieving data over a network at a target bandwidth, comprising:

an input configured to receive the data from a server over the network; and

a processor configured to determine a time at which retrieval of the data from the server begins, to determine a time at which retrieval of the data from the server ends, to determine a byte count of the data, and to calculate a time span between instances of data requests to the server, wherein the time span is equal to a first difference, the first difference is equal to a second difference subtracted from a quotient, the second difference is equal to the time at which retrieval of the data from the server begins subtracted from the time at which retrieval of the data from the server ends, and the quotient is equal to the byte count divided by the target bandwidth.

23. The apparatus of claim 22, wherein the processor is further configured to adjust the time span to mimic substantially a modem transfer rate.

24. The apparatus of claim 22, wherein the processor is further configured to adjust the time span as a function of a time of day.