Method for optimizing off-peak caching of web data

• US 5,961,602 A
• Filed: 02/10/1997
• Issued: 10/05/1999
• Est. Priority Date: 02/10/1997
• Status: Expired due to Term

• Current Assignee: Wistron Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Berstis, Viktors, Thompson, Joseph Raymond
• Primary Examiner(s): Burgess, Glenton B.
• Assistant Examiner(s): TITCOMB, WILLIAM D
• Application Number:
  US08/797,902
• Time in Patent Office: 967 Days
• Field of Search: 711/124, 348/6, 348/10, 395/712, 395/287, 395/704, 395/200.59, 395/200.48, 395/200.68, 395/200.56, 340/825.06, 370/234, 709/229, 709/226, 709/238, 709/218
• US Class Current: 709/229
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/125 by balancing the load, e.g. traffic engineering
• CPC Class Codes: H04L 47/19 at layers above the network layer network arrangements for networked applications for scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/10015 Access to distributed or replicated servers, e.g. using brokers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/101 based on network conditions
• CPC Class Codes: H04L 67/1012 based on compliance of requirements or conditions with available server resources
• CPC Class Codes: H04L 67/1014 based on the content of a request
• CPC Class Codes: H04L 67/1017 based on a round robin mechanism
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/5681 Pre-fetching or pre-delivering data based on network characteristics
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A method of retrieving Web content from a plurality of Web servers for delivery to a Web client connectable to the World Wide Web via a communication link. The Web client is preferably a data processing system connectable to a television or other conventional monitor to provide low cost Internet access. The method begins by having the user define a set of one or more servers from which content is desired to be retrieved and stored in the cache. These servers are preferably identified by a "list" of favorite Web sites. A test is then made to determine whether a given download period has terminated. Typically, this download period occurs during an "off" period, such as in the middle of the night, to avoid traffic congestion at the Web server sites. If the given download period has not terminated, a determination is then made of an activity level for the communication link as content is being downloaded to the cache from the one or more servers. If the activity level for the communication link is less than a given threshold level, additional requests for content are issued to the cache according to a so-called "fairness policy" that ensures that content from as many sites as possible is downloaded during the download period.

Claims

1. A method of retrieving Web content for delivery to a Web client connectable to the World Wide Web via a communication link, the Web client including a cache, comprising the steps of:

a) defining a set of Web servers from which content is desired to be retrieved and stored in the cache;

b) determining an activity level for the communication link as content is being downloaded to the cache from the servers; and

c) if the activity level for the communication link is less than a threshold level, issuing to the cache, additional requests for content according to a policy that ensures that each of the servers of the set has a fair share of opportunities to deliver content to the Web client.

2. The method as described in claim 1 wherein content is downloaded to the cache during a predetermined time period.

3. The method as described in claim 1 wherein according to the policy the additional requests for content are issued to the set of servers as one request per server in an ordered sequence.

4. The method as described in claim 1 wherein according to the policy the additional requests for content are issued to the set of servers up to a predetermined number of requests per server in an ordered sequence.

5. The method as described in claim 1 wherein according to the policy the additional requests for content are issued to the set of servers based on predetermined parameters.

6. The method as described in claim 5 wherein the predetermined parameters include a link depth of a document located on a particular server.

7. The method as described in claim 5 wherein the predetermined parameters include a number of bytes received in the cache from a particular server.

8. The method as described in claim 1 further including the step of processing the content received from a server to remove duplicative links.

9. The method as described in claim 1 further including the step of processing the content received from a server to remove non-local links.

10. The method as described in claim 1 wherein the threshold level of the communication link is a given number of outstanding requests for content.

11. The method as described in claim 1 wherein the threshold level of the communication link is an average link utilization rate for a given monitored interval.

12. A method of retrieving Web content for delivery to a Web client connectable to the World Wide Web via a communication link, the Web client including a cache, comprising the steps of:

a) defining a set of Web servers from which content is desired to be retrieved and stored in the cache;

b) determining whether a given download period has terminated;

c) if the given download period has not terminated, determining an activity level for the communication link as content is being downloaded to the cache from the servers; and

d) if the activity level for the communication link is less than a threshold level, issuing to the cache additional requests for content according to a policy that ensures that each of the servers of the set has a fair share of opportunities to deliver content to the web client.

13. The method as described in claim 12 wherein the given download period is selectable.

14. The method as described in claim 12 wherein the given download period is approximately one hour.

15. The method as described in claim 12 wherein according to the fairness policy the additional requests for content are issued to the servers as one request per server in an ordered sequence.

16. The method as described in claim 12 wherein according to the fairness policy the additional requests for content are issued to the servers up to a predetermined number of requests per server in an ordered sequence.

17. The method as described in claim 12 wherein according to the fairness policy the additional requests for content are issued to the servers based on the number of bytes received in the cache from a particular server.

18. The method as described in claim 12 further including the step of processing the content received from a server to remove duplicative and non-local links.

19. A data processing system, comprising:

a remote control unit; and

a base unit connectable to a monitor for providing Internet access under the control of the remote control unit, the base unit comprising;

a modem connected to a communication link;

a processor;

a memory;

a browser program run by the processor and including means responsive to commands from the remote control unit for generating a list of Web sites;

a cache control program run by the processor and including means for initiating download requests to the communication link based on the list of Web sites; and

a policy control program run by the processor and including means for issuing multiple HTTP GET requests to the cache control program to ensure that each of the Web sites on the list has a fair share of opportunities to contribute content during a given download session.

20. A computer program product for retrieving Web content for delivery to a Web client connectable to the World Wide Web via a communication link, the Web client including a cache, the computer program product comprising:

a computer-readable storage medium having a substrate; and

program data encoded in the substrate of the computer-readable storage medium, wherein the program comprises;

means for defining a set of servers from which content is desired to be retrieved and stored in the cache;

means for monitoring an activity level for the communication link as content is being downloaded to the cache from the servers; and

means responsive to the monitoring means for issuing to the cache additional requests for the content according to a fairness policy that ensures that each of the servers has a fair share of opportunities to contribute content to the Web client during a download session.

21. A computer, comprising:

a processor;

a memory including a cache;

a modem connected to a communication link;

a Web browser program run by the processor for initiating download requests to the communication link based on the list of Web sites and for receiving downloads that are stored in the cache for off-line browsing; and

a fairness policy control program run by the processor issuing multiple HTTP GET requests to the cache control program to ensure that each of the Web sites on the list has a fair share of opportunities to contribute content during a given download session.

22. A computer system, comprising:

a plurality of Web sites;

at least one client computer connectable to each of the plurality of Web sites via a computer network, wherein the client computer comprises;

a processor;

a memory including a cache;

a modem connected to the computer network;

a Web browser program run by the processor for providing World Wide Web information retrieval and including menu means for generating a list of Web sites to be downloaded during off-peak download session;

a cache control program run by the processor for initiating download request to the communication link based on the list of Web sites and for receiving downloads that are stored in the cache for off-line browsing; and

a fairness policy control program run by the processor issuing multiple HTTP GET requests to the cache control program to ensure that each of the Web sites on the list has a fair share of opportunities to contribute content during a given download session.

23. A computer program product for retrieving Web content for delivery to a Web client connectable to the World Wide Web via a communication link, the Web client including a cache, the computer program product comprising:

a computer-readable storage medium having a substrate; and

program data electronically delivered for storage in the substrate of the computer-readable storage medium, wherein the program data comprises;

means for defining a set of servers from which content is desired to be retrieved and stored in the cache;

means for monitoring an activity level for the communication link as content is being downloaded to the cache from the servers; and

means responsive to the monitoring means for issuing to the cache additional requests for content according to a fairness policy that ensures that each of the servers has a fair share of opportunities to contribute content to the Web client during a download session.