Content addressable information encapsulation, representation, and transfer

• US 7,591,022 B2
• Filed: 06/13/2005
• Issued: 09/15/2009
• Est. Priority Date: 01/23/1998
• Status: Expired due to Fees

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Van Riel, Jan F., Teugels, Tom, Carpentier, Paul R.
• Primary Examiner(s): Smithers; Matthew B
• Assistant Examiner(s): CALLAHAN, PAUL E
• Application Number:
  US11/151,619
• Publication Number:
  US 20050234996A1
• Time in Patent Office: 1,555 Days
• Field of Search: 705/51, 705/2, 713/165, 713/190, 369/30.01, 369/30.03, 369/30.04, 369/27.01, 369/47.1, 711/108, 711/216, 726/30
• US Class Current: 726/30
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: H04L 12/1854 with non-centralised forwarding system, e.g. chaincast
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/065 for group communications cryptographic mechanisms or cryptographic arrangements for key management involving conference or group key H04L9/0833
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/1068 Discovery involving direct consultation or announcement among potential requesting and potential source peers
• CPC Class Codes: H04L 67/108 characterised by resources being split in blocks or fragments
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/566 Grouping or aggregating service requests, e.g. for unified processing
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Representing a number of assets on an originating computer begins with selecting the assets to be represented. Cryptographic hash asset identifiers are generated; each of the asset identifiers is computed using the contents of a particular asset. The asset identifier is a content-based or content-addressable asset name for the asset and is location independent. An asset list is generated that includes the asset identifiers computed from the assets. A cryptographic hash asset list identifier is generated that is computed from the asset list. The asset list identifier is stored for later retrieval. The assets selected are also stored for safekeeping either locally or on a computer network. In the event of loss of the files from the originating computer, the asset list identifier is retrieved. Using the asset list identifier, the original asset list is found and retrieved from its safe location. The asset identifiers from the retrieved asset list are used to find and retrieve the individual assets from their backup locations. The assets are verified by recomputing the cryptographic hash asset identifier for each asset retrieved and comparing it to the asset identifier from the asset list. The MD5 algorithm is used for the cryptographic hash function. Assets are retrieved using a multicast protocol. A series of importer programs searches for assets to retrieve in progressively more remote locations. Assets are retrieved whole or in segments.

Claims

1. A method for use in a computer system comprising at least a first computer and a second computer, the method comprising an act of:

(A) verifying the integrity of at least one digital asset that comprises content and is identified in the computer system by an original unique identifier that was generated by applying a function to the content of the at least one digital asset, wherein an identifier generated by applying a function to the content of the at least one digital asset is the only type of identifier that can be used in communications between the first and second computers to identify the at least one digital asset, the act of verifying comprising;

re-applying the function to the digital asset to generate a new identifier; and

comparing the new identifier to the original unique identifier.

2. The method of claim 1, wherein the at least one digital asset is stored by the first computer and wherein the act (A) is performed in response to a request from the second computer to access the at least one digital asset.

3. The method of claim 1, further comprising an act of receiving the at least one digital asset at the first computer in response to a transmission from the second computer, and wherein the act (A) is performed to ensure that no errors resulted in the transmission of the at least one digital asset from the second computer to the first computer.

4. The method of claim 3, wherein the act (A) is performed in response to receipt of the at least one digital asset at the first computer.

5. The method of claim 3, wherein the second computer transmits the at least one digital asset in response to a request for the at least one digital asset issued from the first computer.

6. The method of claim 1, wherein the at least one digital asset is stored by the first computer and wherein the act (A) is performed by the first computer.

7. The method of claim 1, further comprising an act of:

when it is determined in the act (A) that the new identifier does not match the original unique identifier, automatically reconstructing the at least one digital asset.

8. The method of claim 3, wherein the act (A) is performed by the first computer.

9. The method of claim 1, wherein the first computer is coupled to the second computer by a network, and wherein the method further comprises an act of monitoring the network for a communication including the at least one digital asset.

10. The method of claim 3, wherein the first computer provides content addressable storage.

11. The method of claim 3, wherein the first computer is a silo.

12. The method of claim 3, wherein the first computer is a file server.

13. The method of claim 3, wherein the first computer is a storage system.

14. The method of claim 3, wherein the computer system comprises a plurality of computers in addition to the first and second computers, wherein the plurality of computers are interconnected by at least one network, wherein the computer system stores a plurality of digital assets comprising the at least one digital asset, and wherein each of the plurality of computers manages storage of one or more of the plurality of digital assets.

15. The method of claim 3, wherein the first and second computers are peer computers coupled by at least one network.

16. The method of claim 6, wherein the at least one digital asset is stored on the first computer.

17. The method of claim 6, wherein the at least one digital asset is stored on a storage device managed by the first computer.

18. The method of claim 6, further comprising an act of:

when it is determined in the act (A) that the new identifier does not match the original unique identifier, automatically reconstructing the at least one digital asset.

19. The method of claim 6, wherein the act (A) is performed by the first computer.

20. The method of claim 6, wherein the first computer provides content addressable storage.

21. The method of claim 6, wherein the first computer is a storage system.

22. The method of claim 1, wherein the first computer comprises an importer and wherein the act (A) is performed by the first computer.

23. The method of claim 1, wherein the computer system stores, on at least one computer-readable medium, a plurality of digital assets including the at least one digital asset, wherein the at least one digital asset is a first digital asset comprising data and having associated metadata, and wherein the plurality of digital assets further comprises a second digital asset that comprises at least some of the metadata of the first digital asset.

24. The method of claim 1, wherein the computer system stores, on at least one computer-readable medium, a plurality of digital assets including the at least one digital asset, wherein the at least one digital asset is a first digital asset, wherein the plurality of digital assets further comprises a second digital asset comprising data and having associated metadata, and wherein the first digital asset comprises at least some of the metadata of the second digital asset.

25. At least one computer for use in a computer system, the at least one computer comprising:

at least one processor programmed to verify the integrity of at least one digital asset that comprises content and is identified in the computer system by an original unique identifier that was generated by applying a function to the content of the at least one digital asset, wherein an identifier generated by applying a function to the content of the at least one digital asset is the only type of identifier that can be used in communications between the first and second computers to identify the at least one digital asset, the at least one processor being programmed to verify the integrity of the at least one digital asset by;

re-applying the function to the digital asset to generate a new identifier; and

comparing the new identifier to the original unique identifier.

26. The at least one computer of claim 25, wherein the computer system comprises a first computer and a second computer, wherein the at least one computer comprises the first computer, wherein the at least one digital asset is stored by the first computer, and wherein the at least one processor is programmed to verify the integrity of the at least one digital asset in response to a request from the second computer to access the at least one digital asset.

27. The at least one computer of claim 25, wherein the computer system comprises a first computer and a second computer, wherein the at least one computer comprises the first computer, wherein the at least one digital asset is received at the first computer in response to a transmission from the second computer, and wherein the at least one processor is programmed to verify the integrity of the at least one digital asset to ensure that no errors resulted in the transmission of the at least one digital asset from the second computer to the first computer.

28. The at least one computer of claim 27, wherein the at least one processor is programmed to verify the integrity of the at least one digital asset in response to receipt of the at least one digital asset at the first computer.

29. The at least one computer of claim 27, wherein the second computer transmits the at least one digital asset in response to a request for the at least one digital asset issued from the first computer.

30. The at least one computer of claim 25, wherein the at least one digital asset is stored by the at least one computer.

31. The at least one computer of claim 25, wherein the at least one processor is programmed to, when it is determined that the new identifier does not match the original unique identifier, automatically reconstruct the at least one digital asset.

32. The at least one computer of claim 26, wherein the first computer is coupled to the second computer by a network, and wherein the at least one processor is programmed to monitor the network for a communication including the at least one digital asset.

33. The at least one computer of claim 27, wherein the first computer provides content addressable storage.

34. The at least one computer of claim 27, wherein the first computer is a silo.

35. The at least one computer of claim 27, wherein the first computer is a file server.

36. The at least one computer of claim 27, wherein the first computer is a storage system.

37. The at least one computer of claim 27, wherein the computer system comprises a plurality of computers in addition to the first and second computers, wherein the plurality of computers are interconnected by at least one network, wherein the computer system stores a plurality of digital assets comprising the at least one digital asset, and wherein each of the plurality of computers manages storage of one or more of the plurality of digital assets.

38. The at least one computer of claim 27, wherein the first and second computers are peer computers coupled by at least one network.

39. The at least one computer of claim 30, wherein the at least one digital asset is stored on the at least one computer.

40. The at least one computer of claim 30, wherein the at least one digital asset is stored on a storage device managed by the at least one computer.

41. The at least one computer of claim 30, wherein the at least one processor is programmed to, when it is determined that the new identifier does not match the original unique identifier, automatically reconstruct the at least one digital asset.

42. The at least one computer of claim 30, wherein the first computer comprises the at least one processor.

43. The at least one computer of claim 30, wherein the first computer provides content addressable storage.

44. The at least one computer of claim 30, wherein the first computer is a storage system.

45. The at least one computer of claim 26, wherein the first computer further comprises an importer.

46. The at least one computer of claim 25, wherein the computer system comprises a plurality of digital assets including the at least one digital asset, wherein the at least one digital asset is a first digital asset comprising data and having associated metadata, and wherein the plurality of digital assets further comprises a second digital asset that comprises at least some of the metadata of the first digital asset.

47. The at least one computer of claim 25, wherein the computer system comprises a plurality of digital assets including the at least one digital asset, wherein the at least one digital asset is a first digital asset, wherein the plurality of digital assets further comprises a second digital asset comprising data and having associated metadata, and wherein the first digital asset comprises at least some of the metadata of the second digital asset.

48. A method for use in a computer system comprising at least a first computer and a second computer, the method comprising an act of:

(A) verifying the integrity of at least one digital asset that comprises content and is identified in the computer system by an original unique first identifier that was generated by applying a function to the content of the at least one digital asset, the computer system further storing a unit of data that comprises the first identifier and metadata relating to the at least one digital asset, the unit of data being identified via a second identifier generated based, at least in part, on the content of the unit of data, the act of verifying comprising;

re-applying the function to the at least one digital asset to generate a new first identifier; and

comparing the new first identifier to the original unique first identifier.

49. The method of claim 48, wherein the at least one digital asset is stored by the first computer and wherein the act (A) is performed in response to a request from the second computer to access the at least one digital asset.

50. The method of claim 48, further comprising an act of receiving the at least one digital asset at the first computer in response to a transmission from the second computer, and wherein the act (A) is performed to ensure that no errors resulted in the transmission of the at least one digital asset from the second computer to the first computer.

51. The method of claim 50, wherein the act (A) is performed in response to receipt of the at least one digital asset at the first computer.

52. The method of claim 50, wherein the second computer transmits the at least one digital asset in response to a request for the at least one digital asset issued from the first computer.

53. The method of claim 48, wherein the at least one digital asset is stored by the first computer and wherein the act (A) is performed by the first computer.

54. The method of claim 48, further comprising an act of:

when it is determined in the act (A) that the new first identifier does not match the original unique first identifier, automatically reconstructing the at least one digital asset.

55. The method of claim 50, wherein the act (A) is performed by the first computer.

56. The method of claim 48, wherein the first computer is coupled to the second computer by a network, and wherein the method further comprises an act of monitoring the network for a communication including the at least one digital asset.

57. The method of claim 50, wherein the first computer provides content addressable storage.

58. The method of claim 50, wherein the first computer is a silo.

59. The method of claim 50, wherein the first computer is a file server.

60. The method of claim 50, wherein the first computer is a storage system.

61. The method of claim 50, wherein the computer system comprises a plurality of computers in addition to the first and second computers, wherein the plurality of computers are interconnected by at least one network, wherein the computer system stores a plurality of digital assets comprising the at least one digital asset, and wherein each of the plurality of computers manages storage of one or more of the plurality of digital assets.

62. The method of claim 50, wherein the first and second computers are peer computers coupled by at least one network.

63. The method of claim 53, wherein the at least one digital asset is stored on the first computer.

64. The method of claim 53, wherein the at least one digital asset is stored on a storage device managed by the first computer.

65. The method of claim 53, further comprising an act of:

when it is determined in the act (A) that the new first identifier does not match the original unique first identifier, automatically reconstructing the at least one digital asset.

66. The method of claim 53, wherein the act (A) is performed by the first computer.

67. The method of claim 53, wherein the first computer provides content addressable storage.

68. The method of claim 53, wherein the first computer is a storage system.

69. The method of claim 48, wherein the first computer comprises an importer and wherein the act (A) is performed by the first computer.