Content addressable information encapsulation, representation, and transfer

• US 7,415,731 B2
• Filed: 07/19/2004
• Issued: 08/19/2008
• Est. Priority Date: 01/23/1998
• Status: Expired due to Term

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Van Riel, Jan F., Teugels, Tom, Carpentier, Paul R.
• Primary Examiner(s): Moise; Emmanuel L
• Assistant Examiner(s): CALLAHAN, PAUL E
• Application Number:
  US10/894,450
• Publication Number:
  US 20050010794A1
• Time in Patent Office: 1,492 Days
• Field of Search: 705/2, 705/51, 713/165, 713/190, 369/30.04, 369/30.03, 369/47.1, 711/108, 711/216, 726/30
• US Class Current: 726/30
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: H04L 12/1854 with non-centralised forwarding system, e.g. chaincast
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/065 for group communications cryptographic mechanisms or cryptographic arrangements for key management involving conference or group key H04L9/0833
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/1068 Discovery involving direct consultation or announcement among potential requesting and potential source peers
• CPC Class Codes: H04L 67/108 characterised by resources being split in blocks or fragments
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/566 Grouping or aggregating service requests, e.g. for unified processing
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

One embodiment is directed to a method for use in a computer system comprising at least first and second computers, wherein the first provides content addressable storage. A request to access a unit of data stored by the first computer is issued by the second computer and received by the first. In one embodiment, the unit of data comprises a first identifier identifying at least one digital asset and metadata relating to the at least one digital asset, and the request identifies the unit of data via a second identifier based, at least in part, on the content of the unit of data. In another embodiment, a request to access a unit of data is sent from a second computer and received at a first computer and identifies the unit of data via a content identifier based, at least in part, on the content of the unit of data. The content identifier is the only identifier that can be used in communication between the first and second computers to identify the unit of data.

Claims

1. A method for use in a computer system comprising at least first and second computers, wherein the first computer provides content addressable storage, the method comprising acts of:

(A) receiving, at the first computer, a request to access a unit of data stored by the first computer, the unit of data comprising content comprising a first identifier identifying at least one digital asset and metadata relating to the at least one digital asset, the request identifying the unit of data via a second identifier that is based, at least in part, on the content of the unit of data; and

(B) using the second identifier to locate the unit of data.

2. The method of claim 1, wherein the request identifies the second computer as having issued the request, and wherein the method further comprises an act of:

(C) returning the unit of data to the second computer.

3. The method of claim 1, wherein the request identifies the unit of data via a second identifier that is based, at least in part, on the content of the metadata relating to the at least one digital asset.

4. The method of claim 1, wherein the unit of data comprises at least one descriptor file.

5. The method of claim 4, wherein the request is a request to delete the descriptor file, and wherein the method further comprises an act of:

(C) deleting the descriptor file in response to receipt of the request.

6. The method of claim 1, wherein the second identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

7. The method of claim 6, wherein the second identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

8. The method of claim 1, wherein the method further comprises acts of:

(C) generating, at the first computer, a third identifier for the unit of data based upon the content of the unit of data stored by the first computer; and

(D) comparing the second identifier to the third identifier to verify that the third and second identifiers are identical.

9. The method of claim 8, wherein the second identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data and wherein the act (C) comprises an act of generating the third identifier using the cryptographic hash function on the at least a portion of the content of the unit of data stored by the first computer.

10. The method of claim 8, wherein the second identifier is generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (C) comprises an act of generating the third identifier using the cryptographic hash function on the entire content of the unit of data stored by the first computer.

11. The method of claim 1, wherein the first computer comprises a file system to manage units of data stored by the first computer, wherein the second identifier is used as the name of a file in which the unit of data is stored by the storage system, and wherein the act (B) comprises an act of using the second identifier to index into the file system to locate the unit of data.

12. The method of claim 11, wherein the second identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

13. The method of claim 12, wherein the second identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

14. The method of claim 1, wherein the first computer is a silo.

15. The method of claim 1, wherein the first computer is a file server.

16. The method of claim 1, wherein the first computer is a storage system.

17. The method of claim 1, wherein the computer system comprises a plurality of computers in addition to the first and second computers and wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more units of data.

18. The method of claim 1, wherein the first and second computers are peer computers coupled by at least one network.

19. The method of claim 1, wherein the unit of data is stored on the first computer.

20. The method of claim 1, wherein the unit of data is stored by the first computer on a storage device managed by the first computer.

21. The method of claim 1, wherein the request identifies the second computer as having issued the request, and wherein the method further comprises an act of:

returning, to the second computer, information indicating whether the unit of data is stored on the first computer.

22. A first computer that provides content addressable storage for use in a computer system comprising the first computer and a second computer, the first computer comprising:

an input for receiving access requests; and

at least one controller, coupled to the input, that;

receives, through the input, a request to access a unit of data stored by the first computer, the unit of data comprising content comprising a first identifier identifying at least one digital asset and metadata relating to the at least one digital asset, the request identifying the unit of data via a second identifier that is based, at least in part, on the content of the unit of data; and

uses the second identifier to locate the unit of data.

23. The first computer of claim 22, wherein the request identifies the second computer as having issued the request, and wherein the at least one controller returns the unit of data to the second computer.

24. The first computer of claim 22, wherein the request identifies the unit of data via a second identifier that is based, at least in part, on the content of the metadata relating to the at least one digital asset.

25. The first computer of claim 22, wherein the unit of data comprises at least one descriptor file.

26. The first computer of claim 25, wherein the request is a request to delete the descriptor file, and wherein the at least one controller deletes the descriptor file in response to receipt of the request.

27. The first computer of claim 22, wherein the second identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

28. The first computer of claim 27, wherein the second identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

29. The first computer of claim 22, wherein the unit of data is a digital asset, wherein the request is a request to delete the digital asset, and wherein the at least one controller deletes the digital asset in response to receipt of the request.

30. The first computer of claim 22, wherein the at least one controller:

generates a third identifier for the unit of data based upon the content of the unit of data stored by the first computer; and

compares the second identifier to the third identifier to verify that the third and second identifiers are identical.

31. The first computer of claim 30, wherein the second identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data and wherein the at least one controller generates the third identifier using the cryptographic hash function on the at least a portion of the content of the unit of data stored by the first computer.

32. The first computer of claim 30, wherein the second identifier is generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the at least one controller generates the third identifier using the cryptographic hash function on the entire content of the unit of data stored by the first computer.

33. The first computer of claim 22, wherein the first computer comprises a file system to manage units of data stored by the first computer, wherein the second identifier is used as the name of a file in which the unit of data is stored by the storage system, and wherein the at least one controller uses the second identifier to index into the file system to locate the unit of data.

34. The first computer of claim 33, wherein the second identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

35. The first computer of claim 34, wherein the second identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

36. The first computer of claim 22, wherein the first computer is a silo.

37. The first computer of claim 22, wherein the first computer is a file server.

38. The first computer of claim 22, wherein the first computer is a storage system.

39. The first computer of claim 22, wherein the computer system comprises a plurality of computers in addition to the first and second computers and wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more units of data.

40. The first computer of claim 22, wherein the first and second computers are peer computers coupled by at least one network.

41. The first computer of claim 22, wherein the unit of data is stored on the first computer.

42. The first computer of claim 22, wherein the unit of data is stored by the first computer on a storage device managed by the first computer.

43. The first computer of claim 22, wherein the request identifies the second computer as having issued the request, and wherein the at least one controller returns, to the second computer, information indicating whether the unit of data is stored on the first computer.

44. A method for use in a computer system comprising at least first and second computers, wherein the first computer provides content addressable storage, the method comprising an act of:

(A) transmitting, from the second computer to the first computer, a request to access a unit of data stored by the first computer, the request identifying the digital asset via a content identifier that is based, at least in part, on the content of the unit of data, wherein the content identifier is the only identifier the second computer can use to identify the unit of data in the request to the first computer.

45. The method of claim 44, wherein the unit of data comprises at least one digital asset.

46. The method of claim 44, wherein the unit of data comprises at least one descriptor file.

47. The method of claim 46, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the at least one digital asset, wherein the second identifier is based, at least in part, on the content of the at least one digital asset.

48. The method of claim 47, wherein the act of transmitting the request comprises transmitting a request to delete the descriptor file.

49. The method of claim 44, wherein the unit of data comprises at least one digital asset and at least one descriptor file.

50. The method of claim 44, wherein the act of transmitting comprises an act of transmitting a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

51. The method of claim 50, wherein the act of transmitting comprises an act of transmitting a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to the entire content of the unit of data.

52. The method of claim 44, further comprising an act of:

(B) receiving the unit of data at the second computer.

53. The method of claim 52, wherein the identifier in the request is a first identifier for the digital asset, and wherein the method further comprises acts of:

(C) receiving, at the second computer, a second identifier for the digital asset; and

(D) comparing the second identifier to the first identifier to verify that the first and second identifiers are identical.

54. The method of claim 53, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the act of receiving a second identifier comprises an act of receiving a second identifier that was generated by applying the cryptographic hash function to the at least a portion of the content of the unit of data stored by the first computer.

55. The method of claim 53, wherein the first identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act of receiving a second identifier comprises an act of receiving a second identifier that was generated by applying the cryptographic hash function to the entire content of the unit of data stored by the first computer.

56. The method of claim 54, wherein the act of transmitting the request comprises transmitting a request to delete the digital asset.

57. The method of claim 52, wherein the identifier in the request is a first identifier for the unit of data, and wherein the method further comprises acts of:

(C) generating, at the second computer, a second identifier for the unit of data received at the second computer; and

(D) comparing the second identifier to the first identifier to verify that the first and second identifiers are identical.

58. The method of claim 57, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the act (C) comprises an act of generating the second identifier by applying the cryptographic hash function to the at least a portion of the content of the unit of data received at the second computer.

59. The method of claim 57, wherein the first identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (C) comprises an act of generating the second identifier by applying the cryptographic hash function to the entire content of the unit of data received by the second computer.

60. The method of claim 44, wherein the first computer is a silo.

61. The method of claim 44, wherein the first computer is a file server.

62. The method of claim 44, wherein the first computer is a storage system.

63. The method of claim 44, wherein the computer system comprises a plurality of computers in addition to the first and second computers, wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more digital assets.

64. The method of claim 44, wherein the first and second computers are peer computers coupled by at least one network.

65. A first computer that provides content addressable storage for use in a computer system comprising the first computer and a second computer, the first computer comprising:

an output for transmitting access requests;

at least one controller, coupled to the output, that;

transmits, to the second computer, a request to access a unit of data stored by the second computer, the request identifying the digital asset via a content identifier that is based, at least in part, on the content of the unit of data, wherein the content identifier is the only identifier the first computer can use to identify the unit of data in the request to the second computer.

66. The first computer of claim 65, wherein the unit of data comprises at least one digital asset.

67. The first computer of claim 66, wherein the at least one controller transmits a request to delete the digital asset.

68. The first computer of claim 65, wherein the unit of data comprises at least one descriptor file.

69. The first computer of claim 68, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the at least one digital asset, wherein the second identifier is based, at least in part, on the content of the at least one digital asset.

70. The first computer of claim 69, wherein the at least one controller transmits a request to delete the descriptor file.

71. The first computer of claim 65, wherein the unit of data comprises at least one digital asset and at least one descriptor file.

72. The first computer of claim 65, wherein the at least one controller transmits a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

73. The first computer of claim 72, wherein at least one controller transmits a request that identifies the unit of data via an identifier that was generated by applying a cryptographic hash function to the entire content of the unit of data.

74. The first computer of claim 65, wherein the at least one controller receives the unit of data.

75. The first computer of claim 74, wherein the identifier in the request is a first identifier for the digital asset, and wherein the at least one controller:

receives a second identifier for the digital asset; and

compares the second identifier to the first identifier to verify that the first and second identifiers are identical.

76. The first computer of claim 75, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the at least one controller receives a second identifier that was generated by applying the cryptographic hash function to the at least a portion of the content of the unit of data stored by the second computer.

77. The first computer of claim 75, wherein the first identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein at least one controller receives a second identifier that was generated by applying the cryptographic hash function to the entire content of the unit of data stored by the second computer.

78. The first computer of claim 74, wherein the identifier in the request is a first identifier for the unit of data, and wherein the at least one controller:

generates a second identifier for the unit of data received at the first computer; and

compares the second identifier to the first identifier to verify that the first and second identifiers are identical.

79. The first computer of claim 78, wherein the first identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the at least one controller generates the second identifier by applying the cryptographic hash function to the at least a portion of the content of the unit of data.

80. The first computer of claim 65, wherein the second computer is a silo.

81. The first computer of claim 65, wherein the second computer is a file server.

82. The first computer of claim 65, wherein the second computer is a storage system.

83. The first computer of claim 65, wherein the computer system comprises a plurality of computers in addition to the first and first computers, wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more digital assets.

84. The first computer of claim 65, wherein the first and second computers are peer computers coupled by at least one network.

85. The first computer of claim 65, wherein the second computer is a file server.

86. A method for use in a computer system comprising at least first and second computers, wherein the first computer provides content addressable storage, the method comprising acts of:

(A) receiving, at the first computer, a request from the first computer to access a unit of data stored by the first computer, the request identifying the unit of data via a content identifier that is based, at least in part, on the content of the unit of data, wherein the content identifier is the only identifier the second computer can use to identify the unit of data in the request to the first computer; and

(B) using the content identifier to locate the unit of data.

87. The method of claim 86, wherein the request identifies the computer as having issued the request, and wherein the method further comprises an act of:

(C) returning the unit of data to the second computer.

88. The method of claim 86, wherein the unit of data comprises at least one digital asset.

89. The method of claim 86, wherein the unit of data comprises at least one descriptor file.

90. The method of claim 89, wherein the descriptor file includes a second identifier for at least one digital asset and metadata relating to the at least one digital asset, wherein the second identifier is based upon the content of the at least one digital asset.

91. The method of claim 90, wherein the request is a request to delete the descriptor file, and wherein the method further comprises an act of:

(C) deleting the descriptor file in response to receipt of the request.

92. The method of claim 86, wherein the digital asset comprises at least one unit of data and metadata relating to at least one unit of data.

93. The method of claim 86, wherein the identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

94. The method of claim 93, wherein the identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

95. The method of claim 86, wherein the unit of data is a digital asset, wherein the request is a request to delete the digital asset, and wherein the method further comprises an act of:

(C) deleting the digital asset in response to receipt of the request.

96. The method of claim 86, wherein the identifier in the request is a first identifier for the unit of data, and wherein the method further comprises acts of:

(C) generating, at the first computer, a second identifier for the unit of data based upon the content of the unit of data stored by the first computer; and

(D) comparing the second identifier to the first identifier to verify that the first and second identifiers are identical.

97. The method of claim 96, wherein the first identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data and wherein the act (C) comprises an act of generating the second identifier using the cryptographic hash function on the at least a portion of the content of the unit of data stored by the first computer.

98. The method of claim 96, wherein the first identifier is generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (C) comprises an act of generating the second identifier using the cryptographic hash function on the entire content of the unit of data stored by the first computer.

99. The method of claim 86, wherein the first computer comprises a file system to manage units of data stored by the first computer, wherein the identifier is used as the name of a file in which the unit of data is stored by the storage system, and wherein the act (B) comprises an act of using the identifier to index into the file system to locate the unit of data.

100. The method of claim 99, wherein the identifier is generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

101. The method of claim 100, wherein the identifier is generated by applying a cryptographic hash function to the entire content of the unit of data.

102. The method of claim 86, wherein the first computer is a silo.

103. The method of claim 86, wherein the first computer is a file server.

104. The method of claim 86, wherein the first computer is a storage system.

105. The method of claim 86, wherein the computer system comprises a plurality of computers in addition to the first and second computers and wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more units of data.

106. The method of claim 86, wherein the first and second computers are peer computers coupled by at least one network.

107. The method of claim 86, wherein the unit of data is stored on the first computer.

108. The method of claim 86, wherein the unit of data is stored by the first computer on a storage device managed by the first computer.

109. A method for use in a computer system comprising at least first and second computers, wherein the first computer provides content addressable storage, the method comprising an act of:

(A) transmitting, from the second computer to the first computer, a request to access a unit of data stored by the first computer, the unit of data comprising content comprising a first identifier identifying at least one digital asset and metadata relating to the at least one digital asset, the request identifying the digital asset via a second identifier that is based, at least in part, on the content of the unit of data.

110. The method of claim 109, wherein the first computer is a storage system.

111. The method of claim 109, wherein the unit of data comprises at least one descriptor file.

112. The method of claim 111, wherein the second identifier is based, at least in part, on the content of the at least one digital asset.

113. The method of claim 112, wherein the act of transmitting the request comprises transmitting a request to delete the descriptor file.

114. The method of claim 109, wherein the first computer is a file server.

115. The method of claim 109, wherein the act of transmitting comprises an act of transmitting a request that identifies the unit of data via a second identifier that was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data.

116. The method of claim 115, wherein the act of transmitting comprises an act of transmitting a request that identifies the unit of data via a second identifier that was generated by applying a cryptographic hash function to the entire content of the unit of data.

117. The method of claim 109, wherein the act of transmitting the request comprises transmitting a request to delete the unit of data.

118. The method of claim 109, wherein the method further comprises an act of:

(B) receiving the unit of data at the second computer.

119. The method of claim 118, further comprising acts of:

(C) receiving, at the second computer, a third identifier for the digital asset; and

(D) comparing the second identifier to the third identifier to verify that the second and third identifiers are identical.

120. The method of claim 119, wherein the second identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the act of receiving a third identifier comprises an act of receiving a third identifier that was generated by applying the cryptographic hash function to the at least a portion of the content of the unit of data stored by the first computer.

121. The method of claim 119, wherein the second identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act of receiving a third identifier comprises an act of receiving a third identifier that was generated by applying the cryptographic hash function to the entire content of the unit of data stored by the first computer.

122. The method of claim 118, further comprising acts of:

(C) generating, at the second computer, a third identifier for the unit of data received at the second computer; and

(D) comparing the second identifier to the third identifier to verify that the second and third identifiers are identical.

123. The method of claim 122, wherein the second identifier was generated by applying a cryptographic hash function to at least a portion of the content of the unit of data, and wherein the act (C) comprises an act of generating the third identifier by applying the cryptographic hash function to the at least a portion of the content of the unit of data received at the second computer.

124. The method of claim 122, wherein the second identifier was generated by applying a cryptographic hash function to the entire content of the unit of data, and wherein the act (C) comprises an act of generating the third identifier by applying the cryptographic hash function to the entire content of the unit of data received by the second computer.

125. The method of claim 109, wherein the first computer is a silo.

126. The method of claim 109, wherein the computer system comprises a plurality of computers in addition to the first and second computers, wherein the plurality of computers are interconnected by at least one network, and wherein each of the plurality of computers manages storage of one or more digital assets.

127. The method of claim 109, wherein the first and second computers are peer computers coupled by at least one network.