Method and apparatus for distributing large payload file to a plurality of storage devices in a network

• US 7,165,095 B2
• Filed: 05/17/2001
• Issued: 01/16/2007
• Est. Priority Date: 10/26/2000
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Sim, Siew Yong
• Primary Examiner(s): Chin; Wellington
• Assistant Examiner(s): LY, ANH VU H
• Application Number:
  US09/681,665
• Publication Number:
  US 20030031176A1
• Time in Patent Office: 2,070 Days
• Field of Search: 370/352, 370/389, 370/390, 370/392, 370/394, 370/400, 370/401, 370/412, 370/428, 370/429, 370/473, 370/477, 710/20, 707/2, 707/3, 707/10, 707/205, 709/105, 709/201, 709/213, 709/217, 709/218, 709/219
• US Class Current: 709/217
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: H04L 47/125 by balancing the load, e.g. traffic engineering
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/101 based on network conditions
• CPC Class Codes: H04L 67/1017 based on a round robin mechanism
• CPC Class Codes: H04L 67/1019 Random or heuristic server selection
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1027 Persistence of sessions during load balancing
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/14 Multichannel or multilink protocols
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node'"'"'s storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user. The download may be in parallel and usually from the least congested nodes. New nodes added to the network learn from other nodes in the network what content they should have and download the required content, in a desired amount, onto their local storage devices from the nearest and least congested nodes without interrupting network operation. Each node manages its local storage and decides what content to prune based on information such as usage patterns.

Claims

1. A method for distributing content of a large payload file to a plurality of storage devices in a network comprising:

obtaining a plurality of block files representing content of a large payload file for storage in a plurality of storage devices in a network, including;

evaluating said large payload file to locate a portion having substantive content;

determining if said content from said large payload file comprises linear characteristics by determining if said substantive content is located at a starting end of said large payload file;

generating a track file if said content comprises said linear characteristics;

generating a plurality of track files if said content does not comprise said linear characteristics;

determining at least one desired block size, if said content comprises linear characteristics, dividing a track file to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size; and

if said content does not comprise linear characteristics, dividing each of the plurality of track files to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size;

associating said plurality of block files with said plurality of storage devices;

storing said plurality of block files in said plurality of storage devices based on said associations; and

creating a virtual file, using said associations, for presentation to a client requesting said content, said virtual file providing an illusion to said client that said content is contiguous in said network.

2. The method of claim 1, wherein said network comprises a packet-communication network.

3. The method of claim 1, wherein said obtaining a plurality of block files further comprises:

obtaining said large payload file from a user.

4. The method of claim 3, wherein said desired block size is determined by selecting a block size to maximize space usage within a maximum transmission unit of said network.

5. The method of claim 3, wherein said network has a plurality of distribution nodes and said desired block size is determined by selecting a block size to minimize congestion at said distribution nodes of said network.

6. The method of claim 3, wherein said desired block size is determined by selecting a block size to minimize fragmentation of said plurality of storage devices.

7. The method of claim 3, wherein said at least one block size comprises a fixed value.

8. The method of claim 1, wherein said associating said plurality of block files comprises distributing said plurality of block files amongst said plurality of storage devices such that said plurality of storage devices are load balanced during input/output operations.

9. The method of claim 1, further comprising storing said associations in a file metadata in said plurality of storage devices, said associations comprising information for rebuilding said large payload file from said block files.

10. The method of claim 1, wherein said network comprises a plurality of nodes and said plurality of block files is obtained from other nodes of said network.

11. The method of claim 9, wherein said creating a virtual file comprises:

obtaining information about said large payload file from said file metadata;

using said information to construct a virtual representation of said large payload file.

12. A method for distributing content of a large payload file to a plurality of storage devices in a network comprising:

obtaining a plurality of block files representing content of a large payload file for storage in a plurality of storage devices in a network, including;

evaluating said large payload file to locate a portion having substantive content;

determining if said content from said large payload file comprises linear characteristics by determining if said substantive content is located at a starting end of said large payload file;

generating a track file if said content comprises said linear characteristics;

generating a plurality of track files if said content does not comprise said linear characteristics;

determining at least one desired block size, if said content comprises linear characteristics, dividing a track file to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size; and

if said content does not comprise linear characteristics, dividing each of the plurality of track files to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size;

associating said plurality of block files with said plurality of storage devices;

wherein said associating comprises creating associations by distributing said plurality of block files amongst said plurality of storage devices such that said plurality of storage devices are load balanced during input/output operations;

storing said associations in a file metadata in said plurality of storage devices, said associations comprising information for rebuilding said large payload file from said block files;

storing said plurality of block files in said plurality of storage devices based on said associations;

creating a virtual file, using said associations, for presentation to a client requesting said content, said virtual file providing an illusion to said client that said content is contiguous in said network.

13. A computer program product comprising:

a computer readable medium comprising computer readable code for distributing content of large payload file to a plurality of storage devices in a network, said computer readable program code configured to;

obtain a plurality of block files representing content of a large payload file for storage in a plurality of storage devices in a network, including;

evaluate said large payload file to locate a portion having substantive content;

determine if said content from said large payload file comprises linear characteristics by determining if said substantive content is located at a starting end of said large payload file;

generate a track file if said content comprises said linear characteristics;

generating a plurality of track files if said content does not comprise said linear characteristics;

determine at least one desired block size, if said content comprises linear characteristics, dividing a track file to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size; and

if said content does not comprise linear characteristics, dividing each of the plurality of track files to generate a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size;

associate said plurality of block files with said plurality of storage devices;

store said plurality of block files in said plurality of storage devices based on said associations;

create a virtual file, using said associations, for presentation to a client requesting said content, said virtual file providing an illusion to said client that said content is contiguous in said network.

14. The computer program product of claim 13, wherein said network comprises a packet-communication network.

15. The computer program product of claim 13, wherein said obtain a plurality of block files comprises:

obtain said large payload file from a user.

16. The computer program product of claim 15, wherein said desired block size is determined by selecting a block size to maximize space usage within said network'"'"'s maximum transmission unit.

17. The computer program product of claim 15, wherein said network has a plurality of distribution nodes and said desired block size is determined by selecting a block size to minimize congestion at said distribution nodes of said network.

18. The computer program product of claim 15, wherein said desired block size is determined by selecting a block size to minimize fragmentation of said plurality of storage devices.

19. The computer program product of claim 15, wherein said at least one block size comprises a fixed value.

20. The computer program product of claim 13, wherein said associate said plurality of block files comprises distributing said plurality of block files amongst said plurality of storage devices such that said plurality of storage devices are load balanced during input/output operations.

21. The computer program product of claim 13, further comprising computer readable program code configured to store said associations in a file metadata in said plurality of storage devices, said associations comprising information for rebuilding said large payload file from said block files.

22. The computer program product of claim 13, wherein said network comprises a plurality of nodes and said plurality of block files is obtained from other nodes of said network.

23. The computer program product of claim 21, wherein said create a virtual file comprises:

obtain information about said large payload file from said file metadata;

use said information to construct a virtual representation of said large payload file.

24. An apparatus for distributing content of a large payload file to a plurality of storage devices in a network comprising:

one or more first servers in a first server cluster of a network obtaining a plurality of block files representing content of a large payload file for storage in a plurality of storage devices in said network, said one or more first servers associating said plurality of block files with said plurality of storage devices and storing said plurality of block files in said plurality of storage devices based on said associations;

one or more second servers in a second server cluster of said network creating a virtual file, using said associations, for presentation to a client requesting said content, said virtual file providing an illusion to said client that said content is contiguous in said network; and

a third server obtaining said large payload file from a user, said third server evaluating said large payload file to locate a portion having substantive content and determining if said content from said large payload file comprises linear characteristics by determining if said substantive content is located at a starting end of said large payload file, said third server generating a track file if said content comprises said linear characteristics and transmitting said track file to said one or more first servers, and said third server generating a plurality of track files if said content does not comprise said linear characteristics and transmitting said plurality of track files to said one or more first servers.

25. The apparatus of claim 24, wherein said network comprises a packet-communication network.

26. The apparatus of claim 24, wherein said

one or more first servers receiving said large payload file in the form of one or more track files, said one or more first servers determining at least one desired block size for dividing said one or more track files into a plurality of block files, wherein each of said block files correspond in size to said at least one desired block size.

27. The apparatus of claim 26, wherein said desired block size is determined by selecting a block size to maximize space usage within said network'"'"'s maximum transmission unit.

28. The apparatus of claim 26, wherein said network has a plurality of distribution nodes and said desired block size is determined by selecting a block size to minimize congestion at said distribution nodes of said network.

29. The apparatus of claim 26, wherein said desired block size is determined by selecting a block size to minimize fragmentation of said plurality of storage devices.

30. The apparatus of claim 26, wherein said at least one block size comprises a fixed value.

31. The apparatus of claim 24, wherein said associating said plurality of block files comprises distributing said plurality of block files amongst said plurality of storage devices such that said plurality of storage devices are load balanced during input/output operations.

32. The apparatus of claim 24, further comprising said one or more first servers storing said associations in a file metadata in said plurality of storage devices, said associations comprising information for rebuilding said large payload file from said block files.

33. The apparatus of claim 24, wherein said network comprises a plurality of nodes and said plurality of block files is obtained from other nodes of said network.

34. The apparatus of claim 32, wherein said creating a virtual file comprises:

obtaining information about said large payload file from said file metadata;

using said information to construct a virtual representation of said large payload file.