Content addressable information encapsulation, representation and transfer

• US 7,930,550 B2
• Filed: 07/19/2004
• Issued: 04/19/2011
• Est. Priority Date: 01/23/1998
• Status: Expired due to Term

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Carpentier, Paul R., Van Riel, Jan F., Teugels, Tom
• Primary Examiner(s): Lanier; Benjamin E
• Assistant Examiner(s): LEMMA, SAMSON B
• Application Number:
  US10/893,839
• Publication Number:
  US 20050010792A1
• Time in Patent Office: 2,465 Days
• Field of Search: 713/193, 713/165, 713/181
• US Class Current: 713/181
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: H04L 12/1854 with non-centralised forwarding system, e.g. chaincast
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/065 for group communications cryptographic mechanisms or cryptographic arrangements for key management involving conference or group key H04L9/0833
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/1068 Discovery involving direct consultation or announcement among potential requesting and potential source peers
• CPC Class Codes: H04L 67/108 characterised by resources being split in blocks or fragments
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/566 Grouping or aggregating service requests, e.g. for unified processing
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Representing a number of assets on an originating computer begins with selecting the assets to be represented. Cryptographic hash asset identifiers are generated; each of the asset identifiers is computed using the contents of a particular asset. The asset identifier is a content-based or content-addressable asset name for the asset and is location independent. An asset list is generated that includes the asset identifiers computed from the assets. A cryptographic hash asset list identifier is generated that is computed from the asset list. The asset list identifier is stored for later retrieval. The assets selected are also stored for safekeeping either locally or on a computer network. In the event of loss of the files from the originating computer, the asset list identifier is retrieved. Using the asset list identifier, the original asset list is found and retrieved from its safe location. The asset identifiers from the retrieved asset list are used to find and retrieve the individual assets from their backup locations. The assets are verified by recomputing the cryptographic hash asset identifier for each asset retrieved and comparing it to the asset identifier from the asset list. The MD5 algorithm is used for the cryptographic hash function. Assets are retrieved using a multicast protocol. A series of importer programs searches for assets to retrieve in progressively more remote locations. Assets are retrieved whole or in segments.

Claims

1. A method of managing information in a content addressable computer system comprising a plurality of computers including at least a first computer and at least two second computers, the method comprising acts of:

(A) creating a first identifier for a unit of data, at least in part, by hashing the content of the unit of data;

(B) sending a multi-cast request including the first identifier from the first computer to the at least two second computers to access the unit of data, the unit of data comprising content comprising a second identifier identifying at least one digital asset and metadata relating to the at least one digital asset, wherein at least one of the at least two second computers uses the first identifier to access the unit of data, wherein the first identifier is the only identifier the first computer can use to identify the unit of data in the request to the at least two second-computers; and

(C) verifying that the content of the unit of data has not changed since the creation of the first identifier.

2. The method of claim 1, further comprising an act of:

(D) using the first identifier to locate a storage location for the unit of data on the computer system.

3. The method of claim 1, wherein the act (A) comprises an act of creating the first identifier based on the entire content of the unit of data.

4. The method of claim 1, wherein the unit of data is a digital asset.

5. The method of claim 1, wherein the unit of data is a descriptor file.

6. The method of claim 1, wherein the unit of data comprises at least one digital asset and at least one descriptor file.

7. The method of claim 1, wherein the act (A) comprises generating the first identifier by applying a cryptographic hash function to the entire content of the unit of data.

8. The method of claim 5, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the digital asset, and wherein the second identifier for the at least one digital asset is based, at least in part, on the content of the digital asset.

9. The method of claim 1, wherein the computer system comprises a plurality of computers in addition to the first computer and the at least two second computers, wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more units of data.

10. The method of claim 1, wherein the first computer and the at least two second computers are peer computers coupled by at least one network.

11. The method of claim 1, wherein storage of the unit of data is managed by at least one of the at least two second computers.

12. The method of claim 11, further comprising an act, performed by the at least one of the at least two second computers, of using the first identifier to locate a storage location for the unit of data on the computer system.

13. The method of claim 11, wherein the unit of data is stored on the at least one of the at least two second computers.

14. The method of claim 11, wherein the unit of data is stored by the at least one of the at least two second computers on a storage device managed by the at least one of the at least two second computers.

15. The method of claim 11, wherein the request is a request to read the unit of data.

16. The method of claim 15, wherein the request identifies the first computer as having issued the request, and wherein the method further comprises an act of transmitting the unit of data from the at least one of the second computers to the first computer.

17. The method of claim 11, wherein the unit of data is a descriptor file, wherein the request is a request to delete the descriptor file, and wherein the method further comprises an act of:

deleting the descriptor file in response to receipt of the request.

18. The method of claim 11, wherein the unit of data is a digital asset, wherein the request is a request to delete the digital asset, and wherein the method further comprises an act of:

deleting the digital asset in response to receipt of the request.

19. The method of claim 12, wherein act (C) comprises:

(D) generating a third identifier for the unit of data based upon the content of the unit of data; and

(E) comparing the third identifier to the first identifier to verify that the first and third identifiers are identical.

20. The method of claim 19, wherein the act (D) comprises an act of generating the third identifier by applying the cryptographic hash function to the at least a portion of the content of the unit of data.

21. The method of claim 19, wherein the act (A) comprises generating the first identifier by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (D) comprises an act of generating the third identifier by applying the cryptographic hash function to the entire content of the unit of data.

22. The method of claim 12, wherein the at least one of the at least two second computers comprises a file system to manage units of data stored by the at least one of the at least two second computers, wherein the first identifier is used as the name of a file in which the unit of data is stored by the at least one of the at least two second computers, and wherein the method further comprises an act of using the first identifier to index into the file system to locate the unit of data.

23. The method of claim 22, wherein the act (A) comprises generating the first identifier by applying a cryptographic hash function to the entire content of the unit of data.

24. The method of claim 1, wherein at least one of the the at least two second computers is a silo.

25. The method of claim 11, wherein at least one of the at least two second computers is a file server.

26. The method of claim 11, wherein at least one of the at least two second computers is a storage system.

27. The method of claim 1, wherein storage of the unit of data is managed by the at least one of the at least two second computers, wherein the act (B) is performed by the first computer.

28. The method of claim 1, wherein the request is a request to have the unit of data stored by the at least one of the at least two second computers.

29. The method of claim 1, further comprising acts, performed by the at least one of the at least two second computers in response to receipt of the request, of:

storing the unit of data in at least one storage location managed by the at least one of the at least two second computers; and

associating the storage location with the first identifier.

30. The method of claim 1, further comprising an act of receiving the first identifier at the at least one of the at least two second computers.

31. The method of claim 1, further comprising an act, performed by the at least one of the at least two second computers in response to receipt of the request, of generating the first identifier.

32. The method of claim 30, wherein the act (C) comprises:

Generating a third identifier for the unit of data; and

comparing the third identifier to the first identifier to verify that the first and third identifiers are identical.

33. The method of claim 15, wherein the first identifier in the request is a first identifier for the digital asset, and wherein the method further comprises acts of:

receiving the digital asset at the first computer;

generating, at the first computer, third identifier for the unit of data received at the first computer; and

comparing the third identifier to the first identifier to verify that the first and third identifiers are identical.

34. The method of claim 15, wherein the request identifies the first computer as having issued the request, and wherein the method further comprises an act of receiving the unit of data at the first computer.

35. The method of claim 28, further comprising an act of associating, at the first computer, the unit of data with the first identifier.

36. The method of claim 28, wherein the act (A) is performed by at least one of the at least two second computers.

37. The method of claim 28, wherein the act (A) is performed by the first computer.

38. At least one non-transitory computer readable medium encoded with instructions that, when executed on a computer system, perform a method of managing information, wherein the computer system is a content addressable computer system comprising a plurality of computers including at least a first computer and at least two second computers, the method comprising acts of:

(A) creating a first identifier for a unit of data, at least in part, by hashing the content of the unit of data;

(B) sending a multi-cast request including the first identifier from the first computer to the at least two second computers to access the unit of data, the unit of data comprising content comprising a second identifier identifying at least one digital asset and metadata relating to the at least one digital asset, wherein at least one of the at least two second computers uses the first identifier to access the unit of data, wherein the first identifier is the only identifier the first computer can use to identify the unit of data in the request to the at least two second computers; and

(C) verifying that the content of the unit of data has not changed since the creation of the first identifier.

39. The at least one non-transitory computer readable medium of claim 38, wherein the method further comprises an act of:

(D) using the first identifier to locate a storage location for the unit of data on the computer system.

40. The at least one non-transitory computer readable medium of claim 38, wherein the act (A) comprises an act of creating the first identifier based on the entire content of the unit of data.

41. The at least one non-transitory computer readable medium of claim 38, wherein the unit of data is a digital asset.

42. The at least one non-transitory computer readable medium of claim 38, wherein the unit of data is a descriptor file.

43. The at least one non-transitory computer readable medium of claim 38, wherein the unit of data comprises at least one digital asset and at least one descriptor file.

44. The at least one non-transitory computer readable medium of claim 38, wherein the act (A) comprises generating the first identifier by applying a cryptographic hash function to the entire content of the unit of data.

45. The at least one non-transitory computer readable medium of claim 42, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the digital asset, and wherein the second identifier for the at least one digital asset is based, at least in part, on the content of the digital asset.

46. The at least one non-transitory computer readable medium of claim 38, wherein the computer system comprises a plurality of computers in addition to the first computer and the at least two second computers, wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more units of data.

47. The at least one non-transitory computer readable medium of claim 38, wherein the first computer and the at least two second computers are peer computers coupled by at least one network.

48. The at least one non-transitory computer readable medium of claim 38, wherein storage of the unit of data is managed by at least one of the at least two second computers.

49. The at least one non-transitory computer readable medium of claim 48, wherein the method further comprises an act of using the first identifier to locate a storage location for the unit of data on the computer system.

50. The at least one non-transitory computer readable medium of claim 48, wherein the unit of data is stored on at least one of the at least two second computers.

51. The at least one non-transitory computer readable medium of claim 48, wherein the unit of data is stored by the at least one of the at least two second computers on a storage device managed by the at least one of the at least two second computers.

52. The at least one non-transitory computer readable medium of claim 48, wherein the request is a request to read the unit of data.

53. The at least one non-transitory computer readable medium of claim 52, wherein the request identifies the first computer as having issued the request, and wherein the method further comprises an act of transmitting the unit of data from the at least one of the at least two second computers to the first computer.

54. The at least one non-transitory computer readable medium of claim 48, wherein the unit of data is a descriptor file, wherein the request is a request to delete the descriptor file, and wherein the method further comprises an act of:

deleting the descriptor file in response to receipt of the request.

55. The at least one non-transitory computer readable medium of claim 48, wherein the unit of data is a digital asset, wherein the request is a request to delete the digital asset, and wherein the method further comprises an act of:

deleting the digital asset in response to receipt of the request.

56. The at least one non-transitory computer readable medium of claim 49, wherein the act (C) comprises acts of:

(D) generating a third identifier for the unit of data based upon the content of the unit of data; and

(E) comparing the third identifier to the first identifier to verify that the first and third identifiers are identical.

57. The at least one non-transitory computer readable medium of claim 56, wherein the act (D) comprises an act of generating the third identifier by applying the cryptographic hash function to the at least a portion of the content of the unit of data.

58. The at least one non-transitory computer readable medium of claim 56, wherein the act (A) comprises generating the first identifier by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (D) comprises an act of generating the third identifier by applying the cryptographic hash function to the entire content of the unit of data.

59. The at least one non-transitory computer readable medium of claim 49, wherein at least one of the at least two second computers comprises a file system to manage units of data stored by the at least one of the at least two second computers, wherein the first identifier is used as the name of a file in which the unit of data is stored by the at least one of the at least two second computers, and wherein the method further comprises an act of using the first identifier to index into the file system to locate the unit of data.

60. The at least one non-transitory computer readable medium of claim 38, wherein the act (A) comprises generating the first identifier by applying a cryptographic hash function to the entire content of the unit of data.

61. The at least one non-transitory computer readable medium of claim 38, wherein at least one of the at least two second computers is a silo.

62. The at least one non-transitory computer readable medium of claim 48, wherein at least one of the at least two second computers is a file server.

63. The at least one non-transitory computer readable medium of claim 48, wherein at least one of the at least two second computers is a storage system.

64. The at least one non-transitory computer readable medium of claim 38, wherein storage of the unit of data is managed by at least one of the at least two second computers, wherein the instructions are executed, at least in part, on the first computer.

65. The at least one non-transitory computer readable medium of claim 38, wherein storage of the unit of data is managed by at least one of the at least two second computers.

66. The at least one non-transitory computer readable medium of claim 65, wherein the request is a request to have the unit of data stored by the at least one of the at least two second computers.

67. The at least one non-transitory computer readable medium of claim 65, wherein the method further comprises acts, performed in response to receipt of the request, of:

storing the unit of data in at least one storage location managed by the at least one of the at least two second computers; and

associating the storage location with the identifier.

68. The at least one non-transitory computer readable medium of claim 65, wherein the method further comprises an act of receiving the first identifier at the at least one of the at least two second computers.

69. The at least one non-transitory computer readable medium of claim 65, wherein the method further comprises an act, performed in response to receipt of the request, of generating the first identifier.

70. The at least one non-transitory computer readable medium of claim 68, wherein the act (C) comprises:

generating a third identifier for the unit of data; and

comparing the third identifier to the first identifier to verify that the first and third identifiers are identical.

71. The at least one non-transitory computer readable medium of claim 52, wherein the first identifier in the request is a first identifier for the digital asset, and wherein the method further comprises acts of:

receiving the digital asset at the first computer;

generating, at the first computer, a third identifier for the unit of data received at the second computer; and

comparing the third identifier to the first identifier to verify that first and third identifiers are identical.

72. The at least one non-transitory computer readable medium of claim 52, wherein the request identifies the first computer as having issued the request, and wherein the method further comprises an act of receiving the unit of data at the first computer.

73. The at least one non-transitory computer readable medium of claim 66, wherein in the method further comprises an act of associating, at the first computer, the unit of data with the first identifier.

74. The at least one non-transitory computer readable medium of claim 66, wherein the instructions are executed, at least in part, on at least one of the at least two second computers.

75. The at least one non-transitory computer readable medium of claim 66, wherein the instructions are executed, at least in part, on the first computer.

76. A content addressable computer system for managing information, comprising:

a first computer;

at least two second computers; and

at least one controller, disposed on the first computer and/or at least one of the at least two second computers, that;

creates a first identifier for a unit of data, at least in part, by hashing the content of the unit of data, the unit of data comprising content comprising a second identifier identifying at least one digital asset and metadata relating to the at least one digital asset;

sends to the at least two second computers a multi-cast request including the first identifier to access the unit of data, wherein at least one of the at least two second-computers uses the first identifier to access the unit of data, wherein the first identifier is the only identifier the first computer can use to identify the unit of data in the request to the at least two second computers; and

verifies that the content of the unit of data has not changed since the creation of the first identifier.

77. The content addressable computer system of claim 76, wherein the at least one controller uses the first identifier to locate a storage location for the unit of data on the computer system.

78. The content addressable computer system of claim 76, wherein the at least one controller creates the first identifier based on the entire content of the unit of data.

79. The content addressable computer system of claim 76, wherein the unit of data is a digital asset.

80. The content addressable computer system of claim 76, wherein the unit of data is a descriptor file.

81. The content addressable computer system of claim 76, wherein the unit of data comprises at least one digital asset and at least one descriptor file.

82. The content addressable computer system of claim 76, wherein the at least one controller generates the first identifier by applying a cryptographic hash function to the entire content of the unit of data.

83. The content addressable computer system of claim 80, wherein the descriptor file includes the second identifier, and wherein the second identifier is based, at least in part, on the content of the digital asset.

84. The content addressable computer system of claim 76, further comprising a plurality of computers in addition to the first computer and the at least two second computers, and a network that interconnects the plurality of computers, and wherein each of the plurality of computers manages storage of one or more units of data.

85. The content addressable computer system of claim 76, wherein the first computer and the at least two second computers are peer computers coupled by at least one network.

86. The content addressable computer system of claim 76, wherein storage of the unit of data is managed by at least one of the at least two second computers, wherein at least one controller issues a request, from the first computer, to access the unit of data, and wherein the request includes the first identifier.

87. The content addressable computer system of claim 86, wherein the at least one controller is disposed, at least in part, on the at least one of the at least two second computers, and wherein the at least one controller uses the first identifier to locate a storage location for the unit of data on the computer system.

88. The content addressable computer system of claim 86, wherein the unit of data is stored on the at least one of the at least two second computers.

89. The content addressable computer system of claim 86, wherein the computer system further comprises a storage device managed by the at least one of the at least two second computers, and wherein the unit of data is stored by the at least one of the at least two second computers on the storage device managed by the at least one of the at least two second computers.

90. The content addressable computer system of claim 86, wherein the request is a request to read the unit of data.

91. The content addressable computer system of claim 90, wherein the request identifies the first computer as having issued the request, and wherein the at least one controller transmits the unit of data from the at least one of the at least two second computers to the first computer.

92. The content addressable computer system of claim 86, wherein the unit of data is a descriptor file that includes the second identifier, wherein the second identifier is based, at least in part, on the content of the digital asset, wherein the request is a request to delete the descriptor file, and wherein the at least one controller deletes the descriptor file in response to receipt of the request.

93. The content addressable computer system of claim 86, wherein the unit of data is a digital asset, wherein the request is a request to delete the digital asset, and wherein the at least one controller deletes the digital asset in response to receipt of the request.

94. The content addressable computer system of claim 87, wherein the request includes the first identifier for the unit of data, and wherein the at least one controller:

generates a third identifier for the unit of data based upon the content of the unit of data stored by the first computer; and

compares the third identifier to the first identifier to verify that the first and third identifiers are identical.

95. The content addressable computer system of claim 94, wherein the at least one controller generates the third identifier by applying the cryptographic hash function to the at least a portion of the content of the unit of data.

96. The content addressable computer system of claim 94, wherein the at least one controller generates the first identifier by applying a cryptographic hash function to the entire content of the unit of data, and generates the third identifier by applying the cryptographic hash function to the entire content of the unit of data.

97. The content addressable computer system of claim 87, wherein the at least one of the at least two second computers comprises a file system to manage units of data stored by the at least one of the at least two second computers, wherein the first identifier is used as the name of a file in which the unit of data is stored by the at least one of the at least two second computers, and wherein the at least one controller uses the first identifier to index into the file system to locate the unit of data.

98. The content addressable computer system of claim 97, wherein the at least one controller generates the first identifier by applying a cryptographic hash function to the entire content of the unit of data.

99. The content addressable computer system of claim 76, wherein at least one of the at least two second computers is a silo.

100. The content addressable computer system of claim 86, wherein at least one of the at least two second computers is a file server.

101. The content addressable computer system of claim 86, wherein at least one of the at least two second computers is a storage system.

102. The content addressable computer system of claim 76, wherein storage of the unit of data is managed by at least one of the at least two second computers, and wherein the at least one controller is disposed, at least in part, on the at least one of the at least two second computers.

103. The content addressable computer system of claim 76, wherein storage of the unit of data is managed by at least one of the at least two second computers, wherein the at least one controller issues a request, from the first computer, to access the unit of data.

104. The content addressable computer system of claim 103, wherein the request is a request to have the unit of data stored by the at least one of the at least two second computers.

105. The content addressable computer system of claim 103, wherein the at least one controller is disposed, at least in part, on the at least one of the at least two second computers, and wherein the at least one controller, in response to receipt of the request, stores the unit of data in at least one storage location managed by the at least one of the at least two second computers and associates the storage location with the first identifier.

106. The content addressable computer system of claim 103, wherein the at least one controller receives the first identifier at the at least one of the at least two second computers.

107. The content addressable computer system of claim 103, wherein the at least one controller, in response to receipt of the request, generates the first identifier.

108. The content addressable computer system of claim 106, wherein the at least one controller:

generates, at the at least one of the at least two second computers, a third identifier for the unit of data; and

compares the third identifier to the first identifier to verify that the first and third identifiers are identical.

109. The content addressable computer system of claim 90, wherein the at least one controller:

receives the unit of data at the first computer;

generates, at the first computer, a third identifier for the unit of data received at the first computer; and

compares the third identifier to the first identifier to verify that first and third identifiers are identical.

110. The content addressable computer system of claim 90, wherein the request identifies the first computer as having issued the request, and wherein the at least one controller receives the unit of data at the first computer.

111. The content addressable computer system of claim 104, wherein the at least one controller associates, at the first computer, the unit of data with the first identifier.