Method and apparatus for large payload distribution in a network

• US 6,970,939 B2
• Filed: 05/15/2001
• Issued: 11/29/2005
• Est. Priority Date: 10/26/2000
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Sim, Siew Yong
• Primary Examiner(s): VU, VIET DUY
• Application Number:
  US09/681,644
• Publication Number:
  US 20030026254A1
• Time in Patent Office: 1,659 Days
• Field of Search: 709/217, 709/232, 709/236, 709/238, 709/201, 709/203, 709/219, 709/223, 709/226
• US Class Current: 709/236
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: H04L 47/125 by balancing the load, e.g. traffic engineering
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/101 based on network conditions
• CPC Class Codes: H04L 67/1017 based on a round robin mechanism
• CPC Class Codes: H04L 67/1019 Random or heuristic server selection
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1027 Persistence of sessions during load balancing
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/14 Multichannel or multilink protocols
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node'"'"'s storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user. The download may be in parallel and usually from the least congested nodes. New nodes added to the network learn from other nodes in the network what content they should have and download the required content, in a desired amount, onto their local storage devices from the nearest and least congested nodes without interrupting network operation. Each node manages its local storage and decides what content to prune based on information such as usage patterns.

Claims

1. A method for distributing content to a plurality of network nodes comprising:

obtaining a payload filed having content from a client for distribution in a network having a plurality of nodes, said client providing distributing criteria for said content;

dividing said content into plurality of block files by;

evaluating said payload file to locate a portion having substantive content;

determining if said content from said payload file comprises linear characteristics by determining if said substantive content is located at a starting end of said payload file;

generating a track file if said content comprises said linear characteristics;

generating a plurality of track files if said content does not comprise said linear characteristics;

determining at least one desired block size;

if said content comprises linear characteristics, dividing said track file to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size; and

if said content does not comprise linear characteristics, dividing each of said plurality of track files to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size; and

distributing a subset of said plurality of block files to a plurality of qualified nodes.

2. The method of claim 1, wherein said network comprises a packet-based communication network.

3. The method of claim 1, wherein said plurality of nodes is arranged into a virtual tree for passing control information.

4. The method of claim 3, wherein each node of said virtual tree has a set of attributes and a set of rolled up attributes for identification.

5. The method of claim 4, wherein said set of attributes comprises a bitmap and said set of rolled up attributes comprises a combination of said set of attributes for all nodes that are lineal descendants of said each node.

6. The method of claim 5, wherein said combination comprises a binary OR of said set of attributes for all nodes that are lineal descendants of said node.

7. The method of claim 1, further comprising:

storing said plurality of block files in a plurality of local storage devices such that said plurality of block files are distributed over said plurality of local storage devices.

8. The method of claim 7, wherein said distributed over said plurality of local storage devices comprises distributing said block files amongst said plurality of local storage devices such that said plurality of local storage devices are load balanced during input/output operations.

9. The method of claim 1, wherein said distribution criteria comprises information about which of said qualified network nodes is authorized to receive said content.

10. A method for distributing content to a plurality of network nodes comprising:

obtaining a payload file having content from a client for distribution in a network having a plurality of nodes, said client providing distribution criteria for said content;

dividing said content into a plurality of block files; and

distributing a subset of said plurality of block files to a plurality of qualified nodes by;

an initiating node announcing the existence of said content of said payload file by sending a notification to neighbor nodes;

said neighbor nodes of said initiating node forwarding said notification to nodes that are neighboring said neighbor nodes, wherein propagation of said notification continues until all of said plurality of nodes in said network have received said notification; and

each of said qualified nodes downloading a desired subset of said block files from one or more of said plurality of nodes having said content.

11. The method of claim 10, wherein said initiating node does not receive any subsequent notification of said content.

12. The method of claim 10, wherein said downloading said desired subset of said block files is from the least congested nodes nearest to said qualified node.

13. The method of claim 10, wherein said downloading said desired subset of said block files is by parallel downloading from a plurality of nodes of said network.

14. A method for distributing content to a plurality of network nodes comprising:

obtaining a large payload file having content from a content provider for distribution in a network having a plurality of nodes;

said content provider providing distribution criteria for said content, said distribution criteria having information about which of said plurality of nodes are authorized to have said content;

dividing said content into a plurality of block files each having a block size that maximizes playback of said content;

storing said plurality of block files in a plurality of local storage devices such that said plurality of block files are distributed over said plurality of local storage devices to load balance said plurality of local storage devices during input/output operations;

distributing a subset of said plurality of block files from said local storage devices to each one of said plurality of nodes qualified to receive said file content, wherein said nodes qualified are those nodes that conform to said distribution criteria set by said content provider.

15. A method for distributing content to a plurality of network nodes comprising:

obtaining a large payload file comprising file content from a client for distribution in a network having a plurality of nodes, said client providing distribution criteria for said file content, said distribution criteria having information about which of said plurality of nodes are authorized to have said content;

dividing said file content into a plurality of block files having a block size that maximizes playback of said content;

storing said plurality of block files in a plurality of local storage devices such that said plurality of block files are distributed over said plurality of local storage devices to load balance said plurality of local storage devices during input/output operations;

distributing a subset of said plurality of block files from said local storage devices to each one of said plurality of nodes qualified to receive said file content, wherein said nodes qualified are those nodes that conform to said distribution criteria set by said client, wherein each qualified node receiving said subset of said plurality of block files distributes said subset into a plurality of local storage devices of said receiving node.

16. A computer program product comprising:

a computer usable medium comprising computer readable code for distributing content to a plurality of network nodes, said computer readable program code configured to;

obtain a payload file having content from a client for distribution in a network having a plurality of nodes, said client providing distribution criteria for said content;

divide said content into a plurality of block files by;

evaluating said payload file to locate a portion having substantive content;

determining if said content from said payload file comprises linear characteristics by determining if said substantive content is located at a starting end of said payload file;

generating a track file if said content comprises said linear characteristics;

generating a plurality of track files if said content does not comprise said linear characteristics;

determining at least one desired block size;

if said content comprises linear characteristics, dividing said track file to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size;

if said content does not comprise linear characteristics, dividing each of said plurality of track files to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size.

obtain a payload file having content from a client for distribution; and

distribute a subset of said plurality of block files to a plurality of qualified nodes.

17. The computer program product of claim 16, wherein said network comprises a packet-based communication network.

18. The computer program product of claim 16, wherein said plurality of nodes is arranged into a virtual tree for passing control information.

19. The computer program product of claim 18, wherein each node of said virtual tree has a set of attributes and a set of rolled up attributes for. identification.

20. The computer program product of claim 19, wherein said set of attributes comprises a bitmap and said set of rolled up attributes comprises a combination of said set of attributes for all nodes that are lineal descendants of said each node.

21. The computer program product of claim 20, wherein said combination comprises a binary OR of said set of attributes for all nodes that are lineal descendants of said node.

22. The computer program product of claim 16, further comprising computer readable code configured to:

store said plurality of block files in a plurality of local storage devices such that said plurality of block files are distributed over said plurality of local storage devices.

23. The computer program product of claim 22, wherein said distributed over said plurality of local storage devices comprises distributing said block files amongst said plurality of local storage devices such that said plurality of local storage devices are load balanced during input/output operations.

24. The computer program product of claim 16, wherein said distribution criteria comprises information about which of said qualified network nodes is authorized to receive said content.

25. A computer program product comprising:

a computer usable medium comprising computer readable code for distributing content to a plurality of network nodes, said computer readable program code configured to;

obtain a payload file having content from, a client for distribution in a network having a plurality of nodes, said client providing distribution criteria for said content;

divide said content into a plurality of block files; and

distribute a subset of said plurality of block files to a plurality of qualified nodes by;

an initiating node announcing the existence of said content of said payload file by sending a notification to neighbor nodes;

said neighbor nodes of said initiating node forwarding said notification to nodes that are neighboring said neighbor nodes, wherein propagation of said notification continues until all of said plurality of nodes in said network have received said notification; and

each of said qualified nodes downloading a desired subset of said block files from one or more of said plurality of nodes having said content.

26. The computer program product of claim 25, wherein said initiating node does not receive any subsequent notification of said content.

27. The computer program product of claim 25, wherein said downloading said desired subset of said block files is from the least congested nodes nearest to said qualified node.

28. The computer program product of claim 25, wherein said downloading said desired subset of said block files is by parallel downloading from a plurality of nodes of said network.

29. An apparatus for distributing content to a plurality of network nodes comprising:

a network having a plurality of nodes each having a plurality of components capable of communicating with one another, said plurality of components comprising one or more first servers in a first server cluster; and

a third server obtaining a payload tile having content from a client for distribution in said network, said client providing distribution criteria for said content, said third server transmitting said payload file to a first server of said one or more first servers in said first server cluster, said first server dividing said content into a plurality of block files and distributing a subset of said plurality of block files to a plurality of qualified nodes, wherein said third server transmitting said payload file to said first server comprises said third server;

evaluating said payload file to locate a portion having substantive content;

determining if said content from said payload file comprises linear characteristics by determining if said substantive content is located at a starting end of said payload file;

generating a track file and transmitting said track file to said first server if said content comprises said linear characteristics;

generating a plurality of track files and transmitting said plurality of track files to said first server if said content does not comprise said linear characteristics.

30. The apparatus of claim 29, wherein said plurality of components further comprises:

a fourth server for controlling activities of said node.

31. The apparatus of claim 30, wherein said plurality of components further comprises:

one or more second servers in a second server cluster, and one or more fifth servers in a fifth server cluster for serving said content to external clients, said one or more fifth servers having communication link to said one or more second servers in said second server cluster.

32. The apparatus of claim 29, wherein said network comprises a packet-based communication network.

33. The apparatus of claim 29, wherein said plurality of nodes is arranged into a virtual tree for passing control information.

34. The apparatus of claim 33, wherein each node of said virtual tree has a set of attributes and a set of rolled up attributes for identification.

35. The apparatus of claim 34, wherein said set of attributes comprises a bitmap and said set of rolled up attributes comprises a combination of said set of attributes for all nodes that are lineal descendants of said each node.

36. The apparatus of claim 35, wherein said combination comprises a binary OR of said set of attributes for all nodes that are lineal descendants of said node.

37. The apparatus of claim 29, wherein said first server dividing said content comprises:

determining at least one desired block size;

if said content comprises linear characteristics, dividing said track file to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size;

if said content does not comprise linear characteristics, dividing each of said plurality of track files to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size.

38. The apparatus of claim 29, further comprising:

said plurality of components further having a plurality of shared storage devices, wherein said first server stores said plurality of block files in said plurality of shared storage devices such that said plurality of block files are distributed over said plurality of shared storage devices.

39. The apparatus of claim 38, wherein said distributed over said plurality of shared storage devices comprises distributing said block files amongst said plurality of shared storage devices such that said plurality of shared storage devices are load balanced during input/output operations.

40. The apparatus of claim 29, wherein said distribution criteria comprises information about which of said qualified network nodes is authorized to receive said content.

45. The apparatus of claim 31, wherein at least one of said one or more fifth servers is a streaming media server.

46. The apparatus of claim 31, wherein at least one of said one or more fifth servers is a File Transfer Protocol server.

47. The apparatus of claim 29, wherein said third server is a set of applets running on a computer of said client.

41. An apparatus for distributing content to a plurality of network nodes comprising:

a network having a plurality of nodes, each of said plurality of nodes having a plurality of components capable of communicating with one another, said plurality of components comprising one or more first servers in a first server cluster;

a third server obtaining a payload file having content from a client for distribution in said network, said client providing distribution criteria for said content, said third server transmitting said payload file to a first server of said one or more first servers in said first server cluster, said first server dividing said content into a plurality of block files and distributing a subset of said plurality of block files to a plurality of qualified nodes, wherein said distributing a subset of said plurality of block files comprises;

a first server in an initiating node announcing the existence of said content of said payload file by sending a notification to neighbor nodes;

said neighbor nodes of said initiating node forwarding said notification to nodes that are neighboring said neighbor nodes, wherein propagation of said notification continues until all of said plurality of nodes in said network have received said notification;

one or more first servers in said first server cluster in each of said qualified

nodes downloading a desired subset of said block files from one or more of said plurality of nodes having said content.

42. The apparatus of claim 41, wherein said initiating node does not receive any subsequent notification of said content.

43. The apparatus of claim 41, wherein said downloading said desired subset of said block files is from the least congested nodes nearest to said qualified node.

44. The apparatus of claim 41, wherein said downloading said desired subset of said block files is by parallel downloading from a plurality of nodes of said network.