Content addressable information encapsulation, representation, and transfer

• US 20050010794A1
• Filed: 07/19/2004
• Published: 01/13/2005
• Est. Priority Date: 01/23/1998
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Riel, Jan F. Van, Teugels, Tom, Carpentier, Paul R.
• Granted Patent: US 7,415,731 B2
• Time in Patent Office: Days
• US Class Current: 713/193
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: H04L 12/1854 with non-centralised forwarding system, e.g. chaincast
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/065 for group communications cryptographic mechanisms or cryptographic arrangements for key management involving conference or group key H04L9/0833
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/1068 Discovery involving direct consultation or announcement among potential requesting and potential source peers
• CPC Class Codes: H04L 67/108 characterised by resources being split in blocks or fragments
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/566 Grouping or aggregating service requests, e.g. for unified processing
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Representing a number of assets on an originating computer begins with selecting the assets to be represented. Cryptographic hash asset identifiers are generated; each of the asset identifiers is computed using the contents of a particular asset. The asset identifier is a content-based or content-addressable asset name for the asset and is location independent. An asset list is generated that includes the asset identifiers computed from the assets. A cryptographic hash asset list identifier is generated that is computed from the asset list. The asset list identifier is stored for later retrieval. The assets selected are also stored for safekeeping either locally or on a computer network. In the event of loss of the files from the originating computer, the asset list identifier is retrieved. Using the asset list identifier, the original asset list is found and retrieved from its safe location. The asset identifiers from the retrieved asset list are used to find and retrieve the individual assets from their backup locations. The assets are verified by recomputing the cryptographic hash asset identifier for each asset retrieved and comparing it to the asset identifier from the asset list. The MD5 algorithm is used for the cryptographic hash function. Assets are retrieved using a multicast protocol. A series of importer programs searches for assets to retrieve in progressively more remote locations. Assets are retrieved whole or in segments.

Claims

1. A method for use in a computer system comprising at least first and second computers, wherein the first computer provides content addressable storage, the method comprising acts of:

(A) receiving, at the first computer, a request to access a unit of data stored by the first computer, the request identifying the unit of data via an identifier that is based, at least in part, on the content of the digital asset;

(B) using the identifier to locate the unit of data.

2. The method of claim 1, wherein the request identifies the second computer as having issued the request, and wherein the method further comprises an act of:

(C) returning the unit of data to the second computer.

3. The method of claim 1, wherein the unit of data comprises at least one digital asset

4. The method of claim 1, wherein the unit of data comprises at least one descriptor file.

5. The method of claim 1, wherein the digital asset comprises at least one unit of data and metadata relating to at least one unit of data.

6. The method of claim 1, wherein the identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

7. The method of claim 6, wherein the identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

8. The method of claim 4, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the at least one digital asset, wherein the second identifier is based upon the content of the at least one digital asset.

9. The method of claim 8, wherein the request is a request to delete the descriptor file, and wherein the method further comprises an act of:

(C) deleting the descriptor file in response to receipt of the request.

10. The method of claim 1, wherein the unit of data is a digital asset, wherein the request is a request to delete the digital asset, and wherein the method further comprises an act of:

(C) deleting the digital asset in response to receipt of the request.

11. The method of claim 1, wherein the identifier in the request is a first identifier for the unit of data, and wherein the method further comprises acts of:

(C)—

generating, at the first computer, a second identifier for the unit of data based upon the content of the unit of data stored by the first computer; and

(D) comparing the second identifier to the first identifier to verify that the first and second identifiers are identical.

12. The method of claim 11, wherein the first identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data and wherein the act (C) comprises an act of generating the second identifier using the cryptographic hash function on the at least a portion of the content of the unit of data stored by the first computer.

13. The method of claim 11, wherein the first identifier is generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (C) comprises an act of generating the second identifier using the cryptographic hash function on the entire content of the unit of data stored by the first computer.

14. The method of claim 1, wherein the first computer comprises a file system to manage units of data stored by the first computer, wherein the identifier is used as the name of a file in which the unit of data is stored by the storage system, and wherein the act (B) comprises an act of using the identifier to index into the file system to locate the unit of data.

15. The method of claim 14, wherein the identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

16. The method of claim 15, wherein the identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

17. The method of claim 1, wherein the first computer is a silo.

18. The method of claim 1, wherein the first computer is a file server.

19. The method of claim 1, wherein the first computer is a storage system.

20. The method of claim 1, wherein the computer system comprises a plurality of computers in addition to the first and second computers and wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more units of data.

21. The method of claim 1, wherein the first and second computers are peer computers coupled by at least one network.

22. The method of claim 1, wherein the unit of data is stored on the first computer.

23. The method of claim 1, wherein the unit of data is stored by the first computer on a storage device managed by the first computer.

133. The method of claim 1, wherein the request identifies the second computer as having issued the request, and wherein the method further comprises an act of:

returning, to the second computer, information indicating whether the unit of data is stored on the first computer.

24. At least one computer readable medium encoded with instructions that, when executed on a computer system, perform a method, the computer system comprising at least first and second computers, wherein the first computer provides content addressable storage, and wherein the method comprises acts of:

(A) receiving a request to access a unit of data stored by the first computer, the request identifying the unit of data via an identifier that is based, at least in part, on the content of the digital asset;

(B) using the identifier to locate the unit of data.

25. The at least one computer readable medium of claim 24, wherein the request identifies the second computer as having issued the request, and wherein the method further comprises an act of:

(C) returning the unit of data to the second computer.

26. The at least one computer readable medium of claim 24, wherein the unit of data comprises at least one digital asset

27. The at least one computer readable medium of claim 24, wherein the unit of data comprises at least one descriptor file.

28. The at least one computer readable medium of claim 24, wherein the digital asset comprises at least one unit of data and metadata relating to at least one unit of data.

29. The at least one computer readable medium of claim 24, wherein the identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

30. The at least one computer readable medium of claim 29, wherein the identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

31. The at least one computer readable medium of claim 27, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the at least one digital asset, wherein the second identifier is based upon the content of the at least one digital asset.

32. The at least one computer readable medium of claim 31, wherein the request is a request to delete the descriptor file, and wherein the method further comprises an act of:

(C) deleting the descriptor file in response to receipt of the request.

33. The at least one computer readable medium of claim 24, wherein the unit of data is a digital asset, wherein the request is a request to delete the digital asset, and wherein the method further comprises an act of:

(C) deleting the digital asset in response to receipt of the request.

34. The at least one computer readable medium of claim 24, wherein the identifier in the request is a first identifier for the unit of data, and wherein the method further comprises acts of:

(C) generating, at the first computer, a second identifier for the unit of data based upon the content of the unit of data stored by the first computer; and

(D) comparing the second identifier to the first identifier to verify that the first and second identifiers are identical.

35. The at least one computer readable medium of claim 34, wherein the first identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data and wherein the act (C) comprises an act of generating the second identifier using the cryptographic hash function on the at least a portion of the content of the unit of data stored by the first computer.

36. The at least one computer readable medium of claim 34, wherein the first identifier is generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (C) comprises an act of generating the second identifier using the cryptographic hash function on the entire content of the unit of data stored by the first computer.

37. The at least one computer readable medium of claim 24, wherein the first computer comprises a file system to manage units of data stored by the first computer, wherein the identifier is used as the name of a file in which the unit of data is stored by the storage system, and wherein the act (B) comprises an act of using the identifier to index into the file system to locate the unit of data.

38. The at least one computer readable medium of claim 37, wherein the identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

39. The at least one computer readable medium of claim 38, wherein the identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

40. The at least one computer readable medium of claim 24, wherein the first computer is a silo.

41. The at least one computer readable medium of claim 24, wherein the first computer is a file server.

42. The at least one computer readable medium of claim 24, wherein the first computer is a storage system.

43. The at least one computer readable medium of claim 24, wherein the computer system comprises a plurality of computers in addition to the first and second computers and wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more units of data.

44. The at least one computer readable medium of claim 24, wherein the first and second computers are peer computers coupled by at least one network.

45. The at least one computer readable medium of claim 24, wherein the unit of data is stored on the first computer.

46. The at least one computer readable medium of claim 24, wherein the unit of data is stored by the first computer on a storage device managed by the first computer.

134. The at least one computer readable medium of claim 24, wherein the request identifies the second computer as having issued the request, and wherein the method further comprises an act of:

returning, to the second computer, information indicating whether the unit of data is stored on the first computer.

47. A first computer that provides content addressable storage for use in a computer system comprising the first computer and a second computer, the first computer comprising:

an input for receiving access requests; and

at least one controller, coupled to the input, that;

receives, through the input, a request to access a unit of data stored by the first computer, the request identifying the unit of data via an identifier that is based, at least in part, on the content of the digital asset; and

uses the identifier to locate the unit of data.

48. The first computer of claim 47, wherein the request identifies the second computer as having issued the request, and wherein the at least one controller returns the unit of data to the second computer.

49. The first computer of claim 47, wherein the unit of data comprises at least one digital asset

50. The first computer of claim 47, wherein the unit of data comprises at least one descriptor file.

51. The first computer of claim 47, wherein the digital asset comprises at least one unit of data and metadata relating to at least one unit of data.

52. The first computer of claim 47, wherein the identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

53. The first computer of claim 52, wherein the identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

54. The first computer of claim 50, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the at least one digital asset, wherein the second identifier is based upon the content of the at least one digital asset.

55. The first computer of claim 54, wherein the request is a request to delete the descriptor file, and wherein the at least one controller deletes the descriptor file in response to receipt of the request.

56. The first computer of claim 47, wherein the unit of data is a digital asset, wherein the request is a request to delete the digital asset, and wherein the at least one controller deletes the digital asset in response to receipt of the request.

57. The first computer of claim 47, wherein the identifier in the request is a first identifier for the unit of data, and wherein the at least one controller:

generates a second identifier for the unit of data based upon the content of the unit of data stored by the first computer; and

compares the second identifier to the first identifier to verify that the first and second identifiers are identical.

58. The first computer of claim 57, wherein the first identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data and wherein the at least one controller generates the second identifier using the cryptographic hash function on the at least a portion of the content of the unit of data stored by the first computer.

59. The first computer of claim 57, wherein the first identifier is generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the at least one controller generates the second identifier using the cryptographic hash function on the entire content of the unit of data stored by the first computer.

60. The first computer of claim 47, wherein the first computer comprises a file system to manage units of data stored by the first computer, wherein the identifier is used as the name of a file in which the unit of data is stored by the storage system, and wherein the at least one controller uses the identifier to index into the file system to locate the unit of data.

61. The first computer of claim 60, wherein the identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

62. The first computer of claim 61, wherein the identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

63. The first computer of claim 47, wherein the first computer is a silo.

64. The first computer of claim 47, wherein the first computer is a file server.

65. The first computer of claim 47, wherein the first computer is a storage system.

66. The first computer of claim 47, wherein the computer system comprises a plurality of computers in addition to the first and second computers and wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more units of data.

67. The first computer of claim 47, wherein the first and second computers are peer computers coupled by at least one network.

68. The first computer of claim 47, wherein the unit of data is stored on the first computer.

69. The first computer of claim 47, wherein the unit of data is stored by the first computer on a storage device managed by the first computer.

Host Read

135. The first computer of claim 47, wherein the request identifies the second computer as having issued the request, and wherein the at least one controller returns, to the second computer, information indicating whether the unit of data is stored on the first computer.

70. A method for use in a computer system comprising at least first and second computers, wherein the first computer provides content addressable storage, the method comprising an act of:

(A) transmitting, from the second computer to the first computer, a request to access a unit of data stored by the first computer, the request identifying the digital asset via an identifier that is based, at least in part, on the content of the unit of data.

71. The method of claim 70, wherein the unit of data comprises at least one digital asset.

72. The method of claim 70, wherein the unit of data comprises at least one descriptor file.

73. The method of claim 70, wherein the unit of data comprises at least one digital asset and at least one descriptor file.

74. The method of claim 70, wherein the act of transmitting comprises an act of transmitting a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

75. The method of claim 74, wherein the act of transmitting comprises an act of transmitting a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to the entire content of the unit of data.

76. The method of claim 72, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the at least one digital asset, wherein the second identifier is based, at least in part, on the content of the at least one digital asset.

77. The method of claim 76, wherein the act of transmitting the request comprises transmitting a request to delete the descriptor file.

78. The method of claim 81, wherein the act of transmitting the request comprises transmitting a request to delete the digital asset.

79. The method of claim 70, further comprising an act of:

(B) receiving the unit of data at the second computer.

80. The method of claim 79, wherein the identifier in the request is a first identifier for the digital asset, and wherein the method further comprises acts of:

(C) receiving, at the second computer, a second identifier for the digital asset; and

(D) comparing the second identifier to the first identifier to verify that the first and second identifiers are identical.

81. The method of claim 80, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the act of receiving a second identifier comprises an act of receiving a second identifier that was generated by applying the cryptographic hash function to the at least a portion of the content of the unit of data stored by the first computer.

82. The method of claim 80, wherein the first identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act of receiving a second identifier comprises an act of receiving a second identifier that was generated by applying the cryptographic hash function to the entire content of the unit of data stored by the first computer.

83. The method of claim 70, wherein the first computer is a silo.

84. The method of claim 70, wherein the first computer is a file server.

85. The method of claim 70, wherein the first computer is a storage system.

86. The method of claim 70, wherein the computer system comprises a plurality of computers in addition to the first and second computers, wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more digital assets.

87. The method of claim 70, wherein the first and second computers are peer computers coupled by at least one network.

88. The method of claim 79, wherein the identifier in the request is a first identifier for the unit of data, and wherein the method further comprises acts of:

(C) generating, at the second computer, a second identifier for the unit of data received at the second computer; and

(D) comparing the second identifier to the first identifier to verify that the first and second identifiers are identical.

89. The method of claim 88, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the act (C) comprises an act of generating the second identifier by applying the cryptographic hash function to the at least a portion of the content of the unit of data received at the second computer.

90. The method of claim 88, wherein the first identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (C) comprises an act of generating the second identifier by applying the cryptographic hash function to the entire content of the unit of data received by the second computer.

91. At least one computer readable medium encoded with instructions that, when executed on a computer system, perform a method, wherein the computer system comprises at least first and second computers, wherein the first computer provides content addressable storage, and wherein the method comprises an act of:

(A) transmitting, from the second computer to the first computer, a request to access a unit of data stored by the first computer, the request identifying the digital asset via an identifier that is based, at least in part, on the content of the unit of data.

92. The at least one computer readable medium of claim 91, wherein the unit of data comprises at least one digital asset.

93. The at least one computer readable medium of claim 91, wherein the unit of data comprises at least one descriptor file.

94. The at least one computer readable medium of claim 91, wherein the unit of data comprises at least one digital asset and at least one descriptor file.

95. The at least one computer readable medium of claim 91, wherein the act of transmitting comprises an act of transmitting a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

96. The at least one computer readable medium of claim 95, wherein the act of transmitting comprises an act of transmitting a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to the entire content of the unit of data.

97. The at least one computer readable medium of claim 93, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the at least one digital asset, wherein the second identifier is based, at least in part, on the content of the at least one digital asset.

98. The at least one computer readable medium of claim 97, wherein the act of transmitting the request comprises transmitting a request to delete the descriptor file.

99. The at least one computer readable medium of claim 92, wherein the act of transmitting the request comprises transmitting a request to delete the digital asset.

100. The at least one computer readable medium of claim 91, wherein the method further comprises an act of:

(B) receiving the unit of data at the second computer.

101. The at least one computer readable medium of claim 100, wherein the identifier in the request is a first identifier for the digital asset, and wherein the method further comprises acts of:

(C) receiving, at the second computer, a second identifier for the digital asset; and

(D) comparing the second identifier to the first identifier to verify that the first and second identifiers are identical.

102. The at least one computer readable medium of claim 101, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the act of receiving a second identifier comprises an act of receiving a second identifier that-was generated by applying the cryptographic hash function to the at least a portion of the content of the unit of data stored by the first computer.

103. The at least one computer readable medium of claim 101, wherein the first identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act of receiving a second identifier comprises an act of receiving a second identifier that was generated by applying the cryptographic hash function to the entire content of the unit of data stored by the first computer.

104. The at least one computer readable medium of claim 91, wherein the first computer is a silo.

105. The at least one computer readable medium of claim 91, wherein the first computer is a file server.

106. The at least one computer readable medium of claim 91, wherein the first computer is a storage system.

107. The at least one computer readable medium of claim 91, wherein the computer system comprises a plurality of computers in addition to the first and second computers, wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more digital assets.

108. The at least one computer readable medium of claim 91, wherein the first and second computers are peer computers coupled by at least one network.

109. The at least one computer readable medium of claim 100, wherein the identifier in the request is a first identifier for the unit of data, and wherein the method further comprises acts of:

(C) generating, at the second computer, a second identifier for the unit of data received at the second computer; and

(D) comparing the second identifier to the first identifier to verify-that the first and second identifiers are identical.

110. The at least one computer readable medium of claim 109, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the act (C) comprises an act of generating the second identifier by applying the cryptographic hash function to the at least a portion of the content of the unit of data received at the second computer.

111. The at least one computer readable medium of claim 109, wherein the first identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (C) comprises an act of generating the second identifier by applying the cryptographic hash function to the entire content of the unit of data received by the second computer.

112. A first computer that provides content addressable storage for use in a computer system comprising the first computer and a second computer, the first computer comprising:

an output for transmitting access requests;

at least one controller, coupled to the output, that;

transmits, to the second computer, a request to access a unit of data stored by the second computer, the request identifying the digital asset via an identifier that is based, at least in part, on the content of the unit of data.

113. The first computer of claim 112, wherein the unit of data comprises at least one digital asset.

114. The first computer of claim 112, wherein the unit of data comprises at least one descriptor file.

115. The first computer of claim 112, wherein the unit of data comprises at least one digital asset and at least one descriptor file.

116. The first computer of claim 112, wherein the at least one controller transmits a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

117. The first computer of claim 116, wherein at least one controller transmits a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to the entire content of the unit of data.

118. The first computer of claim 114, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the at least one digital asset, wherein the second identifier is based, at least in part, on the content of the at least one digital asset.

119. The first computer of claim 118, wherein the at least one controller transmits a request to delete the descriptor file.

120. The first computer of claim 113, wherein the at least one controller transmits a request to delete the digital asset.

121. The first computer of claim 112, wherein the at least one controller receives the unit of data.

122. The first computer of claim 121, wherein the identifier in the request is a first identifier for the digital asset, and wherein the at least one controller:

receives a second identifier for the digital asset; and

compares the second identifier to the first identifier to verify that the first and second identifiers are identical.

123. The first computer of claim 122, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the at least one controller receives a second identifier that was generated by applying the cryptographic hash function to the at least a portion of the content of the unit of data stored by the second computer.

124. The first computer of claim 122, wherein the first identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein at least one controller receives a second identifier that was generated by applying the cryptographic hash function to the entire content of the unit of data stored by the second computer.

125. The first computer of claim 112, wherein the second computer is a silo.

126. The first computer of claim 112, wherein the second computer is a file server.

127. The first computer of claim 112, wherein the second computer is a storage system.

128. The first computer of claim 112, wherein the computer system comprises a plurality of computers in addition to the first and first computers, wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more digital assets.

129. The first computer of claim 112, wherein the first and second computers are peer computers coupled by at least one network.

130. The first computer of claim 121, wherein the identifier in the request is a first identifier for the unit of data, and wherein the at least one controller:

generates a second identifier for the unit of data received at the first computer; and

compares the second identifier to the first identifier to verify that the first and second identifiers are identical.

131. The first computer of claim 130, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the at least one controller generates the second identifier by applying the cryptographic hash function to the at least a portion of the content of the unit of data.

132. The first computer of claim 112, wherein the second computer is a file server.