Access to content addressable data over a network

• US 20050283613A1
• Filed: 08/05/2005
• Published: 12/22/2005
• Est. Priority Date: 01/23/1998
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Wave Research NV
• Inventors: Van Riel, Jan F., Teugels, Tom, Carpentier, Paul R.
• Granted Patent: US 7,506,157 B2
• Time in Patent Office: Days
• US Class Current: 713/180
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: H04L 12/1854 with non-centralised forwarding system, e.g. chaincast
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/065 for group communications cryptographic mechanisms or cryptographic arrangements for key management involving conference or group key H04L9/0833
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/1068 Discovery involving direct consultation or announcement among potential requesting and potential source peers
• CPC Class Codes: H04L 67/108 characterised by resources being split in blocks or fragments
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/566 Grouping or aggregating service requests, e.g. for unified processing
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Access to content addressable data on a network is facilitated using digital information storing devices or data repositories (“silos”) that monitor broadcast data requests over the network. A number of silos automatically monitor both data requests and data itself that are broadcast over a network. The silos selectively store data. Each silo responds to data requests broadcast over the network with data the silo has previously intercepted. A content addressable file scheme is used to enable the data repositories to reliably identify data being requested. When a data request is received, each silo evaluates whether it has all or a portion of the data being requested and responds to requests when it has the data. Requests for data are implemented by broadcasting a cryptographic hash data identifier of the data file needed. The data identifier is used by a silo to determine which data to receive and store. A silo includes a network interface, a digital asset collector, an asset request list, asset storage, an asset identifier processor and an asset supplier. The asset identifier processor computes a cryptographic hash asset identifier for a received asset and compares it to an asset identifier on its asset request list to verify it has the correct asset. A hash of a list of assets is also computed and broadcast over the network. When the hash of the list of assets is received by a silo, it places all the assets in its asset request list.

Claims

1. A method of managing a computer system comprising a plurality of nodes coupled via at least one communication medium, the computer system storing a plurality of digital assets, the method comprising acts of:

(A) storing each of the plurality of digital assets via at least one of the plurality of nodes according to a distribution scheme that provides distributed storage of the plurality of digital assets among the plurality of nodes; and

(B) managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications among the plurality of nodes.

2. The method of claim 1, wherein the act (B) comprises managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise requests, issued from at least one of the plurality of nodes, to receive digital assets.

3. The method of claim 1, wherein the act (B) comprises managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise communications, issued from at least one of the plurality of nodes, that comprise digital assets.

4. The method of claim 1, wherein the act (B) comprises managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise at least one multicast communication issued from at least one of the plurality of nodes.

5. The method of claim 4, wherein at least one multicast communication comprises a request to receive at least one of the plurality of digital assets.

6. The method of claim 1, wherein the act (B) comprises managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise the transmission of one of the plurality of digital assets from one of the plurality of nodes to another of the plurality of nodes.

7. The method of claim 1, wherein the computer system comprises a plurality of resources comprising at least the plurality of nodes and the at least one communication medium, and wherein the act (A) comprises an act of storing each of the plurality of digital assets on at least one of the plurality of nodes according to a distribution scheme that evaluates current usage of at least one of the plurality of resources in determining on which of the plurality of nodes to store at least one of the plurality of assets.

8. The method of claim 7, wherein the at least one of the plurality of resources comprises the at least one communication medium.

9. The method of claim 1, wherein the at least one communication medium comprises at least one network.

10. The method of claim 1, wherein the act (A) comprises an act of storing the at least one of the plurality of digital assets on two or more of the plurality of nodes in accordance with the distribution scheme.

11. The method of claim 1, wherein at least one of the plurality of nodes comprises a silo that manages storage of at least some of the plurality of digital assets.

12. The method of claim 1, wherein the computer system comprises a content addressable storage system.

13. The method of claim 1, wherein the computer system comprises a storage system.

14. The method of claim 1, wherein the computer system makes the plurality of digital assets available to at least one client, wherein at least one of the plurality of digital assets comprises content and is identified by the at least one client and the computer system via an identifier that is based, at least in part, on a cryptographic hash of at least some of the content of the at least one of the plurality of digital assets.

15. The method of claim 1, wherein the plurality of nodes are peer computers that are coupled by the at least one communication medium.

16. The method of claim 1, wherein the act (A) comprises storing at least one of the plurality of digital assets on at least one of the plurality of nodes.

17. The method of claim 1, wherein the act (A) comprises storing at least one of the plurality of digital assets on a storage device managed by at least one of the plurality of nodes.

18. At least one computer readable medium encoded with a plurality of instructions for execution on a computer system, the computer system comprising a plurality of nodes coupled via at least one communication medium, the computer system storing a plurality of digital assets, the plurality of instructions, when executed, performing a method of managing the computer system, the method comprising acts of:

(A) storing each of the plurality of digital assets via at least one of the plurality of nodes according to a distribution scheme that provides distributed storage of the plurality of digital assets among the plurality of nodes; and

(B) managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications among the plurality of nodes.

19. The at least one computer readable medium of claim 18, wherein the act (B) comprises managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise requests, issued from at least one of the plurality of nodes, to receive digital assets.

20. The at least one computer readable medium of claim 18, wherein the act (B) comprises managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise communications, issued from at least one of the plurality of nodes, that comprise digital assets.

21. The at least one computer readable medium of claim 18, wherein the act (B) comprises managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise at least one multicast communication issued from at least one of the plurality of nodes.

22. The at least one computer readable medium of claim 21, wherein at least one multicast communication comprises a request to receive at least one of the plurality of digital assets.

23. The at least one computer readable medium of claim 18, wherein the act (B) comprises managing the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise the transmission of one of the plurality of digital assets from one of the plurality of nodes to another of the plurality of nodes.

24. The at least one computer readable medium of claim 18, wherein the computer system comprises a plurality of resources comprising at least the plurality of nodes and the at least one communication medium, and wherein the act (A) comprises an act of storing each of the plurality of digital assets on at least one of the plurality of nodes according to a distribution scheme that evaluates current usage of at least one of the plurality of resources in determining on which of the plurality of nodes to store at least one of the plurality of assets.

25. The at least one computer readable medium of claim 24, wherein the at least one of the plurality of resources comprises the at least one communication medium.

26. The at least one computer readable medium of claim 18, wherein the at least one communication medium comprises at least one network.

27. The at least one computer readable medium of claim 18, wherein the act (A) comprises an act of storing the at least one of the plurality of digital assets on two or more of the plurality of nodes in accordance with the distribution scheme.

28. The at least one computer readable medium of claim 18, wherein at least one of the plurality of nodes comprises a silo that manages storage of at least some of the plurality of digital assets.

29. The at least one computer readable medium of claim 18, wherein the computer system comprises a content addressable storage system.

30. The at least one computer readable medium of claim 18, wherein the computer system comprises a storage system.

31. The at least one computer readable medium of claim 18, wherein the computer system makes the plurality of digital assets available to at least one client, wherein at least one of the plurality of digital assets comprises content and is identified by the at least one client and the computer system via an identifier that is based, at least in part, on a cryptographic hash of at least some of the content of the at least one of the plurality of digital assets.

32. The at least one computer readable medium of claim 18, wherein the plurality of nodes are peer computers that are coupled by the at least one communication medium.

33. The at least one computer readable medium of claim 18, wherein the act (A) comprises storing at least one of the plurality of digital assets on at least one of the plurality of nodes.

34. The at least one computer readable medium of claim 18, wherein the act (A) comprises storing at least one of the plurality of digital assets on a storage device managed by at least one of the plurality of nodes.

35. A computer system for storing a plurality of digital assets, the computer system comprising:

a plurality of nodes;

at least one communication medium that couples the plurality of nodes; and

at least one processor programmed to store each of the plurality of digital assets via at least one of the plurality of nodes according to a distribution scheme that provides distributed storage of the plurality of digital assets among the plurality of nodes, the at least one processor being programmed to manage the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications among the plurality of nodes.

36. The computer system of claim 35, wherein the at least one processor is programmed to manage the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise requests, issued from at least one of the plurality of nodes, to receive digital assets.

37. The computer system of claim 35, wherein the at least one processor is programmed to manage the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise communications, issued from at least one of the plurality of nodes, that comprise digital assets.

38. The computer system of claim 35, wherein the at least one processor is programmed to manage the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise at least one multicast communication issued from at least one of the plurality of nodes.

39. The computer system of claim 38, wherein at least one multicast communication comprises a request to receive at least one of the plurality of digital assets.

40. The computer system of claim 35, wherein the at least one processor is programmed to manage the distribution of the plurality of digital assets among the plurality of nodes, at least in part, via node-to-node communications that comprise the transmission of one of the plurality of digital assets from one of the plurality of nodes to another of the plurality of nodes.

41. The computer system of claim 35, wherein the computer system comprises a plurality of resources comprising at least the plurality of nodes and the at least one communication medium, wherein the at least one processor is programmed to store each of the plurality of digital assets on at least one of the plurality of nodes according to a distribution scheme that evaluates current usage of at least one of the plurality of resources in determining on which of the plurality of nodes to store at least one of the plurality of assets.

42. The computer system of claim 41, wherein the at least one of the plurality of resources comprises the at least one communication medium.

43. The computer system of claim 35, wherein the at least one communication medium comprises at least one network.

44. The computer system of claim 35, wherein the at least one processor is programmed to store the at least one of the plurality of digital assets on two or more of the plurality of nodes in accordance with the distribution scheme.

45. The computer system of claim 35, wherein at least one of the plurality of nodes comprises a silo that manages storage of at least some of the plurality of digital assets.

46. The computer system of claim 35, wherein the computer system comprises a content addressable storage system.

47. The computer system of claim 35, wherein the computer system comprises a storage system.

48. The computer system of claim 35, wherein the computer system makes the plurality of digital assets available to at least one client, wherein at least one of the plurality of digital assets comprises content and is identified by the at least one client and the computer system via an identifier that is based, at least in part, on a cryptographic hash of at least some of the content of the at least one of the plurality of digital assets.

49. The computer system of claim 35, wherein the plurality of nodes are peer computers that are coupled by the at least one communication medium.

50. The computer system of claim 35, wherein the at least one processor is programmed to store at least one of the plurality of digital assets on at least one of the plurality of nodes.

51. The computer system of claim 35, wherein the at least one processor is programmed to store at least one of the plurality of digital assets on a storage device managed by at least one of the plurality of nodes.

52. The computer system of claim 35, wherein the at least one processor is disposed on at least one of the plurality of nodes.

53. The computer system of claim 35, wherein the at least one processor comprises a plurality of processors distributed among the plurality of nodes.