Method and apparatus for generating a large payload file

• US 20020112069A1
• Filed: 05/18/2001
• Published: 08/15/2002
• Est. Priority Date: 10/26/2000
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Sim, Siew Yong
• Granted Patent: US 7,058,014 B2
• Time in Patent Office: Days
• US Class Current: 709/236
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: H04L 47/125 by balancing the load, e.g. traffic engineering
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/101 based on network conditions
• CPC Class Codes: H04L 67/1017 based on a round robin mechanism
• CPC Class Codes: H04L 67/1019 Random or heuristic server selection
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1027 Persistence of sessions during load balancing
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/14 Multichannel or multilink protocols
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node'"'"'s storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user. The download may be in parallel and usually from the least congested nodes. New nodes added to the network learn from other nodes in the network what content they should have and download the required content, in a desired amount, onto their local storage devices from the nearest and least congested nodes without interrupting network operation. Each node manages its local storage and decides what content to prune based on information such as usage patterns.

Claims

1. A method for generating a large payload file from non-contiguous segments of block files comprising:

receiving a request at a network node for a portion of a large payload file;

obtaining information about assembling said portion of said large payload file;

obtaining a plurality of block files of said large payload file from a plurality of storage devices associated with said network node;

creating a virtual representation of said portion of said large payload file from said plurality of block files using said assembling information;

transmitting said portion of said large payload file in response to said request.

2. The method of claim 1, wherein said assembling information includes information whether said large payload is linear, wherein said large payload file is linear if substantive content of said large payload file is located at a starting end of said large payload file.

3. The method of claim 2, wherein said creating said virtual representation of said portion of said large payload file from said plurality of block files comprises:

creating a virtual assembly of said plurality of block files as said representation if said large payload file is linear.

4. The method of claim 2, wherein said creating said virtual representation of said portion of said large payload file from said plurality of block files comprises:

assembling said plurality of block files into a plurality of virtual track files if said large payload file is not linear; and

multiplexing said plurality of virtual track files to create said representation.

5. The method of claim 1, wherein said request is from an application servicing a client request.

6. The method of claim 5, wherein said application comprises a streaming server.

7. The method of claim 5, wherein said application comprises an FTP host.

8. The method of claim 5, wherein said application comprises a media player.

9. The method of claim 1, wherein said assembling information is contained in a file metadata database stored in said plurality of storage devices of said node.

10. The method of claim 1, wherein said block files are stored in non-contiguous form in said plurality of storage devices.

11. The method of claim 3, wherein said creating a virtual assembly of said plurality of block files comprises linear superposition of representations of said plurality of block files according to said assembling information.

12. The method of claim 4, wherein said assembling said plurality of block files comprises linear superposition of representations of said plurality of block files according to said assembling information.

13. The method of claim 1, wherein said request comprises an offset into said large payload file and a length of said portion of said large payload file.

14. The method of claim 1 wherein said plurality of block files comprises a first subset of block files resident at said node at receipt of said request and a second subset of block files not resident at said node at receipt of said request, wherein both subsets represent the entire large payload file.

15. The method of claim 14, further comprising:

obtaining said second subset of block files from one or more nodes of said network.

16. The method of claim 1, wherein said representation comprises a virtual representation of said large payload file.

17. The method of claim 1, wherein said transmitting said large payload file commences when said first subset of block files resident in said node is adequate to start satisfying said request.

18. A method for generating a large payload file from non-contiguous segments of block files comprising:

receiving a request at a network node for content of a portion of a large payload file from an application servicing a client;

obtaining assembling information from a file metadata database available in a plurality of storage devices of said node, said assembling information providing data on reconstruction of said portion of said large payload file;

obtaining a plurality of block files representing said portion of said large payload file from said plurality of storage devices of said node;

creating a virtual representation of said portion of said large payload file from said plurality of block files using said assembling information;

transmitting said portion of said large payload file in response to said request.

19. A method for generating a large payload file from non-contiguous segments of block files comprising:

receiving a request at a network node for content of a portion of a large payload file from an application servicing a client, said request comprising an offset into said large payload file and a length of said portion of said large payload file;

obtaining assembling information from a file metadata database available in a plurality of storage devices of said node, said assembling information providing data on reconstruction of said portion of said large payload file;

obtaining a plurality of block files representing said portion of said large payload file from said plurality of storage devices of said node, wherein said plurality of block files are stored in non-contiguous form in said plurality of storage devices;

creating a virtual representation of said portion of said large payload file from said plurality of block files using said assembling information;

transmitting said portion of said large payload file in response to said request.

20. A method for satisfying a client request for content with portions of said content distributed amongst a plurality of network nodes comprising:

receiving a first request at a first node from a client for content, wherein said first node comprises one of a plurality of nodes of a network, said first node having a portion of said content, wherein said content is distributed in subsets of block files amongst one or more of said plurality of nodes;

preparing a virtual image of said content using data in a file metadata of said content available in said first node, said file metadata having information about reassembling said file, said virtual image providing an illusion to said client that said content is fully available at said first node;

beginning transmission of said content to said client starting from said portion of said content available at said first node; and

obtaining non-resident portions of said content while said transmission is progressing, said obtaining non-resident portions comprising;

sending a second request to determine nodes in said plurality of nodes that have said non-resident portions of said content, wherein one or more of said plurality of nodes may have same portion of said content;

receiving a response from each responding node having said content, said response specifying the portion of said content and performance characteristics of said responding node;

determining from said response which of said responding nodes are a desired set of nodes to download said non-resident portions of said content from; and

downloading said non-resident portions of said content from said desired set of nodes and storing said non-resident portions onto said requesting node.

21. A computer program product comprising:

a computer usable medium comprising computer readable code for generating a large payload file from non-contiguous segments of block files, said computer readable program code configured to;

receive a request at a network node for content of a large payload file;

obtain information about assembling said large payload file;

obtain a plurality of block files of said large payload file from a plurality of storage devices of said node;

create a virtual representation of said large payload file from said plurality of block files using said assembling information;

transmit said large payload file in response to said request.

22. The computer program product of claim 21, wherein said assembling information includes information whether said large payload is linear, wherein said large payload file is linear if substantive content of said large payload file is located at a starting end of said large payload file.

23. The computer program product of claim 22, wherein said create said virtual representation of said large payload file from said plurality of block files comprises:

create a virtual assembly of said plurality of block files as said representation if said large payload file is linear.

24. The computer program product of claim 22, wherein said create said virtual representation of said large payload file from said plurality of block files comprises:

assembling said plurality of block files into a plurality of virtual track files if said large payload file is not linear; and

multiplexing said plurality of virtual track files to create said representation.

25. The computer program product of claim 21, wherein said request is from an application servicing a client request.

26. The computer program product of claim 25, wherein said application is a streaming server.

27. The computer program product of claim 25, wherein said application is an FTP host.

28. The computer program product of claim 25, wherein said application comprises a media player.

29. The computer program product of claim 21, wherein said assembling information is contained in a file metadata database stored in said plurality of storage devices of said node.

30. The computer program product of claim 21, wherein said block files are stored in non-contiguous form in said plurality of storage devices.

31. The computer program product of claim 23, wherein said create said virtual assembly of said plurality of block files comprises linear superposition of representations of said plurality of block files according to said assembling information.

32. The computer program product of claim 24, wherein said assembling said plurality of block files comprises linear superposition of representations of said plurality of block files according to said assembling information.

33. The computer program product of claim 24, wherein said request comprises an offset into said large payload file and a length of said portion of said large payload file.

34. The computer program product of claim 21, wherein said plurality of block files comprises a first subset of block files resident at said node at receipt of said request and a second subset of block files not resident at said node at receipt of said request, wherein both subsets represent the entire large payload file.

35. The computer program product of claim 34, further comprising computer readable program code configured to:

obtain said second subset of block files from one or more nodes of said network.

36. The computer program product of claim 21, wherein said representation is a virtual representation of said large payload file.

37. The computer program product of claim 21, wherein said transmit said large payload file commences as soon as said first subset of block files resident in said node is adequate to start satisfying said request.

39. The apparatus of claim 38, wherein said assembling information includes information whether said large payload is linear, wherein said large payload file is linear if substantive content of said large payload file is located at a starting end of said large payload file.

40. The apparatus of claim 39, wherein said creating said virtual representation of said large payload file from said plurality of block files comprises:

creating a virtual assembly of said plurality of block files as said representation if said large payload file is linear.

41. The apparatus of claim 39, wherein said creating said virtual representation of said large payload file from said plurality of block files comprises:

assembling said plurality of block files into a plurality of virtual track files if said large payload file is not linear; and

multiplexing said plurality of virtual track files to create said representation.

42. The apparatus of claim 38, wherein said request is from an application servicing a client request.

43. The apparatus of claim 42, wherein said application is a streaming server.

44. The apparatus of claim 42, wherein said application is an FTP host.

45. The apparatus of claim 42, wherein said application comprises a media player.

46. The apparatus of claim 38, wherein said assembling information is contained in a file metadata database stored in said plurality of storage devices of said node.

47. The apparatus of claim 38, wherein said block files are stored in non-contiguous form in said plurality of storage devices.

48. The apparatus of claim 40, wherein said creating a virtual assembly of said plurality of block files comprises linear superposition of representations of said plurality of block files according to said assembling information.

49. The apparatus of claim 41, wherein said assembling said plurality of block files comprises linear superposition of representations of said plurality of block files according to said assembling information.

50. The apparatus of claim 41, wherein said request comprises an offset into said large payload file and a length of said portion of said large payload file.

51. The apparatus of claim 38, wherein said plurality of block files comprises a first subset of block files resident at said node at receipt of said request and a second subset of block files not resident at said node at receipt of said request , wherein both subsets represent the entire large payload file.

52. The apparatus of claim 51, further comprising:

one or more second servers in a second server cluster, and said first server sending a second request to a second server of said one or more second servers to obtain said second subset of block files from other nodes of said network.

53. The apparatus of claim 38, wherein said representation is a virtual representation of said large payload file.

54. The apparatus of claim 38, wherein said transmitting said large payload file commences as soon as said first subset of block files resident in said node is adequate to start satisfying said request.

38. An apparatus for generating a large payload file from non-contiguous segments of block files comprising:

a network with a plurality of nodes;

a plurality of storage devices in each of said plurality of nodes;

a first server in a first server cluster of a node of said network receiving a request for content of a large payload file, said first server cluster having one or more first servers, said first server obtaining information about assembling said large payload file, said first server obtaining a plurality of block files of said large payload file from said plurality of storage devices of said node, said first server creating a virtual representation of said large payload file from said plurality of block files using said assembling information, and transmitting said large payload file in response to said request.