Timing module for regulating hits by a spidering engine

• US 7,949,748 B2
• Filed: 07/10/2008
• Issued: 05/24/2011
• Est. Priority Date: 04/19/2000
• Status: Expired due to Fees

• Current Assignee: Archeron Ltd LLC (Intellectual Ventures LLC)
• Original Assignee: Archeron Ltd LLC (Intellectual Ventures LLC)
• Inventors: Foulger, Michael G., Cooper, Jeremy S.
• Primary Examiner(s): Bates; Kevin
• Assistant Examiner(s): Eskandarnia; Arvin
• Application Number:
  US12/170,676
• Publication Number:
  US 20080270604A1
• Time in Patent Office: 1,048 Days
• Field of Search: 709/224
• US Class Current: 709/224
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method and system for retrieving web-site based information by a spider engine at a target bandwidth is described. A target bandwidth is received from the spider engine. A wait time is calculated by a timing module. Data retrieval from a web site is delayed by the calculated wait time so that data is retrieved at the desired target bandwidth.

Claims

1. A method for determining a time between transmitting requests for data over a network to a remote server, comprising:

determining, at a processor, a time at which an initial request for the data is transmitted over the network to the remote server;

determining, at the processor, a count of a number of bytes of the data received over the network from the remote server in response to the initial request;

determining, at the processor, a current time, wherein the current time is other than the time at which the initial request for the data is transmitted over the network to the remote server; and

calculating, at the processor, the time between transmitting the initial request and a subsequent request for the data over the network to the remote server, wherein the time between transmitting the requests is calculated as a function of a quotient minus a current time difference, wherein the quotient is equal to the count divided by a target rate of transfer of the data from the remote server measured in bytes per unit of time, and wherein the current time difference is equal to the current time minus the time at which the initial request for the data is transmitted over the network to the remote server.

2. The method of claim 1, further comprising adjusting the time between transmitting the requests for data over the network to the remote server so as to substantially mimic a modem transfer rate.

3. The method of claim 1, further comprising adjusting the time between transmitting the requests for data over the network to the remote server as a function of a time of day.

4. An apparatus for determining a time between transmitting requests for data over a network to a remote server, comprising:

an input configured to receive the data over the network from the remote server;

an output configured to transmit the requests for the data over the network to the remote server; and

a processor configured to determine a time at which an initial request for the data is transmitted over the network to the remote server, to determine a count of a number of bytes of the data received over the network from the remote server in response to the initial request, to determine a current time, wherein the current time is other than the time at which the initial request for the data is transmitted over the network to the remote server, and to calculate the time between transmitting the initial request and a subsequent request for the data over the network to the remote server, wherein the time between transmitting the requests is calculated as a function of a quotient minus a current time difference, wherein the quotient is equal to the count divided by a target rate of transfer of the data from the remote server measured in bytes per unit of time, and wherein the current time difference is equal to the current time minus the time at which the initial request for the data is transmitted over the network to the remote server.

5. The apparatus of claim 4, wherein the processor is further configured to adjust the time between transmitting the requests for data over the network to the remote server so as to substantially mimic a modem transfer rate.

6. The apparatus of claim 4, wherein the processor is further configured to adjust the time between transmitting the requests for data over the network to the remote server as a function of a time of day.

7. A computer-readable storage medium having stored thereon computer executable instructions, execution of which by a processing device causes the processing device to perform operations for determining a time between transmitting requests for data over a network to a remote server, the operations comprising:

determining a time at which an initial request for the data is transmitted over the network to the remote server;

determining a count of a number of bytes of the data received over the network from the remote server in response to the initial request;

determining a current time, wherein the current time is other than the time at which the initial request for the data is transmitted over the network to the remote server; and

calculating the time between transmitting the initial request and a subsequent request for the data over the network to the remote server, wherein the time between transmitting the requests is calculated as a function of a quotient minus a current time difference, wherein the quotient is equal to the count divided by a target rate of transfer of the data from the remote server measured in bytes per unit of time, and wherein the current time difference is equal to the current time minus the time at which the initial request for the data is transmitted over the network to the remote server.

8. The computer-readable storage medium of claim 7, wherein the operations further comprise adjusting the time between transmitting the requests for data over the network to the remote server so as to substantially mimic a modem transfer rate.

9. The computer-readable storage medium of claim 7, wherein the operations further comprise adjusting the time between transmitting the requests for data over the network to the remote server as a function of a time of day.

10. The computer-readable storage medium of claim 7, wherein the operations further comprise:

transmitting the requests for the data over the network to the remote server; and

transmitting, between a timing module of the processing device and a retrieving processor of the processing device, the time between transmitting the requests for data.

11. A method for retrieving data over a network at a target bandwidth, comprising:

determining, at a local processor, the target bandwidth, wherein the target bandwidth is a desired rate for retrieving the data from a remote server over the network to the local processor, and wherein the data is stored at the remote server in a manner unknown to the local processor;

determining, at the local processor, a time at which retrieval of the data from the remote server to the local processor begins;

determining, at the local processor, a time at which retrieval of the data from the remote server to the local processor ends;

determining, at the local processor, a byte count of the data; and

calculating, at the local processor, an amount of time between transmitting requests for data from the local processor to the remote server, wherein the amount of time between transmitting requests for data is calculated as a function of the target bandwidth, the time at which retrieval of the data from the remote server to the local processor begins, the time at which retrieval of the data from the remote server to the local processor ends, and the byte count of the data.

12. The method of claim 11, further comprising adjusting the amount of time between transmitting requests for the data over the network to the remote server so as to substantially mimic a modem transfer rate.

13. The method of claim 11, further comprising adjusting the amount of time between transmitting requests for the data over the network to the remote server as a function of a time of day.

14. The method of claim 11, wherein the function is an equation including a quotient minus a time difference, wherein the quotient is equal to the byte count of the data divided by the target bandwidth measured in bytes per unit of time, and wherein the time difference is equal to an amount of time between the time at which retrieval of the data from the remote server to the local processor begins and the time at which the data from the remote server to the local processor ends.

15. An apparatus for retrieving data over a network at a target bandwidth, comprising:

an input configured to receive the data from a remote server over the network to a local processor, wherein the data is stored at the remote server in a manner unknown to the local processor; and

wherein the local processor is configured to determine the target bandwidth, wherein the target bandwidth is a desired rate for retrieving the data from the remote server over the network to the local processor, to determine a time at which retrieval of the data from the remote server to the local processor begins, to determine a time at which retrieval of the data from the remote server to the local processor ends, to determine a byte count of the data, and to calculate an amount of time between requests for data transmitted from the local processor to the remote server, wherein the amount of time between transmitting requests for the data is a function of the target bandwidth, the time at which retrieval of the data from the remote server to the local processor begins, the time at which retrieval of the data from the remote server to the local processor ends, and the byte count of the data.

16. The apparatus of claim 15, wherein the local processor is further configured to adjust the amount of time between transmitting requests for the data over the network to the remote server so as to substantially mimic a modem transfer rate.

17. The apparatus of claim 15, wherein the local processor is further configured to adjust the amount of time between the requests for the data over the network to the remote server as a function of a time of day.

18. A computer-readable storage medium having stored thereon computer executable instructions, execution of which by a processing device causes the processing device to perform operations for retrieving data over a network at a target bandwidth, the operations comprising:

determining the target bandwidth, wherein the target bandwidth is a desired rate for retrieving the data from a remote server over the network to the processing device, and wherein the data is stored at the remote server in a manner unknown to the processing device;

determining a time at which retrieval of the data from the remote server to the processing device begins;

determining a time at which retrieval of the data from the remote server to the processing device ends;

determining a byte count of the data; and

calculating an amount of time between transmitting requests for the data from the processing device to the remote server, wherein the amount of time between transmitting requests is calculated as a function of the target bandwidth, the time at which retrieval of the data from the remote server to the processing device begins, the time at which retrieval of the data from the remote server to the processing device ends, and the byte count of the data.

19. The computer-readable storage medium of claim 18, wherein the operations further comprise adjusting the amount of time between transmitting requests for the data over the network to the remote server so as to substantially mimic a modem transfer rate.

20. The computer-readable storage medium of claim 18, wherein the operations further comprises adjusting the amount of time between transmitting requests for the data over the network to the remote server as a function of a time of day.

21. The computer-readable storage medium of claim 18, wherein the operations further comprise:

transmitting a request for the data via the network to the remote server; and

transmitting, between a timing module of the processing device and a retrieving processor of the processing device, the amount of time between requests for data.

22. An apparatus for retrieving data over a network at a target bandwidth, comprising:

an input configured to receive the data from a remote server over the network; and

a processor configured to determine a time at which retrieval of the data from the remote server begins, to determine a time at which retrieval of the data from the remote server ends, to determine a byte count of the data, and to calculate an amount of time between transmitting requests for the data to the remote server, wherein the amount of time between transmitting requests for the data is calculated as a function of a quotient minus a time difference, wherein the quotient is equal to the byte count divided by the target bandwidth measured in bytes per unit of time, and wherein the time difference is equal to an amount of time between the time at which retrieval of the data from the remote server to the processor begins and the time at which retrieval of the data from the remote server to the processor ends.

23. The apparatus of claim 22, wherein the processor is further configured to adjust the amount of time between transmitting requests for the data to the remote server so as to substantially mimic a modem transfer rate.

24. The apparatus of claim 22, wherein the processor is further configured to adjust the amount of time between transmitting requests for the data to the remote server as a function of a time of day.

25. An apparatus for determining a time between transmitting requests for data over a network to a remote server, comprising:

means for receiving the data over the network from the remote server;

means for transmitting the requests for the data over the network to the remote server; and

means for determining a time at which an initial request for the data is transmitted over the network to the remote server, for determining a count of a number of bytes of the data received over the network from the remote server in response to the initial request, for determining a current time, wherein the current time is other than the time at which the initial request for the data is transmitted over the network to the remote server, and for calculating the time between transmitting the initial request and a subsequent request for the data over the network to the remote server, wherein the time between transmitting the requests is calculated as a function of a quotient minus a current time difference, wherein the quotient is equal to the count divided by a target rate of transfer of the data from the remote server measured in bytes per unit of time, and wherein the current time difference is equal to the current time minus the time at which the initial request for the data is transmitted over the network to the remote server.

26. An apparatus for retrieving data over a network at a target bandwidth, comprising:

an input configured to receive the data from a remote server over the network; and

means for determining a time at which retrieval of the data from the remote server begins, for determine a time at which retrieval of the data from the remote server ends, for determining a byte count of the data, and for calculating an amount of time between transmitting requests for the data to the remote server, wherein the amount of time between transmitting requests for the data is calculated as a function of a quotient minus a time difference, wherein the quotient is equal to the byte count divided by the target bandwidth measured in bytes per unit of time, and wherein the time difference is equal to an amount of time between the time at which retrieval of the data from the remote server to the processor begins and the time at which retrieval of the data from the remote server to the processor ends.