SYSTEM AND METHOD FOR DISTRIBUTED DEDUPLICATIONOF ENCRYPTED CHUNKS

• US 20150227757A1
• Filed: 04/16/2015
• Published: 08/13/2015
• Est. Priority Date: 07/18/2012
• Status: Active Grant

• Current Assignee: Nexenta by DDN, Inc.
• Original Assignee: Nexenta Systems Incorporated (DataDirect Networks Incorporated)
• Inventors: BESTLER, Caitlin, AIZMAN, Alexander
• Granted Patent: US 9,547,774 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 21/6227 where protection concerns the structure of data, e.g. records, types, queries
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0623 in relation to content
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 2209/76 Proxy, i.e. using intermediary entity to perform cryptographic operations
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 9/0863 involving passwords or one-time passwords network architectures or network communication protocols for using one-time keys in a packet data network H04L63/067
• CPC Class Codes: H04L 9/3239 involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD

Abstract

The present disclosure relates to an advantageous system and related methods for distributed deduplication of encrypted chunks. One embodiment relates to a method for storing encrypted chunks in which an encryption key is generated independently from a chunk payload. With this method, two encrypted chunks are identifiable as having identical chunk payloads even when the chunk payloads are encrypted with different encryption keys. Other embodiments, aspects and features are also disclosed.

Claims

1. A method for retrieving chunks of data from a distributed deduplication storage system, the method comprising:

receiving a request for a chunk from a client, wherein the request for the chunk includes a chunk identifier;

retrieving an encrypted chunk payload and an encrypted chunk key for the chunk;

de-crypting the encrypted chunk key to obtain an unencrypted chunk key;

de-crypting the encrypted chunk payload using the unencrypted chunk key to obtain a decrypted chunk payload;

re-encrypting the decrypted chunk payload to obtain a re-encrypted chunk payload that is encrypted for the client to be able to decrypt;

returning the re-encrypted chunk payload to the client in response to the request for the chunk.

2. The method of claim 1, wherein the client comprises a proxy acting on behalf of the client.

3. The method of claim 1, wherein the encrypted chunk payload comprises a chunk payload that is compressed and then encrypted.

4. The method of claim 1 further comprising:

receiving the re-encrypted chunk payload at the client;

decrypting the re-encrypted chunk payload to obtain the decrypted chunk payload;

applying a cryptographic hash to the decrypted chunk payload to generate a fingerprint;

comparing the fingerprint against the chunk identifier to validate the decrypted chunk payload; and

sending an invalid chunk message to the storage server if the fingerprint and the chunk identifier do not match.

5. The method of claim 4, further comprising:

if the invalid chunk message is received, then performing a scrub on the chunk at the storage server.

6. The method of claim 4, further comprising:

in further response to the request for the chunk, returning a metadata identifier of a specific chunk image that was read.

7. The method of claim 6, further comprising:

if the invalid chunk message is received, then validating the encrypted chunk payload of the specific chunk image; and

if the encrypted chunk payload of the specific chunk image is determined to be invalid, then using other replicas of the chunk, either stored locally or elsewhere in the distributed deduplication storage system, to repair or replace the encrypted chunk payload of the specific chunk image.

8. The method of claim 1, further comprising:

encoding in a transaction identifier whether chunks of an object are encrypted, wherein said encoding is performed by a metadata tracking portion of the distributed deduplication storage system, and the metadata tracking portion is not involved in key management for chunk encryption.

9. The method of claim 1 further comprising:

validating a transaction identifier in the request by verifying that the chunk has a back-reference to an object identified within the transaction identifier.

10. A distributed deduplication storage system for storing encrypted chunks of data, the storage system comprising:

a client; and

a storage server which

receives a request for a chunk from a client, wherein the request for the chunk includes a chunk identifier;

retrieves an encrypted chunk payload and an encrypted chunk key for the chunk;

de-crypts the encrypted chunk key to obtain an unencrypted chunk key;

de-crypts the encrypted chunk payload using the unencrypted chunk key to obtain a decrypted chunk payload;

re-encrypts the decrypted chunk payload to obtain a re-encrypted chunk payload that is encrypted for the client to be able to decrypt; and

returns the re-encrypted chunk payload to the client in response to the request for the chunk.

11. A method of putting an encrypted chunk to an object storage system supporting distributed deduplication, the method comprising:

calculating a cryptographic hash to fingerprint an unencrypted payload by a client/proxy (a client, or proxy acting on behalf of the client), after optional compression has been applied at the client/proxy;

sending a chunk put request to a chunk server that identifies the chunk to be put by a fingerprint, wherein the chunk server was selected either because it is co-located with the client/proxy or by a consistent hashing algorithm.

12. The method of claim 11 further comprising:

determining by the chunk server whether a specified chunk identifier is already stored within a federated set of chunk servers by checking local storage of the chunk server and at least one of the chunk servers designated to store the specified chunk identifier by the consistent hashing algorithm.

13. The method of claim 12 further comprising:

indicating by the chunk server to the client/proxy whether the chunk was already stored, or whether a chunk payload is to be provided.

14. The method of claim 13 further comprising:

when the chunk payload is to be provided, the client/proxy performs steps including generating an effectively random chunk key that is not correlated with the chunk payload, encrypting the chunk payload with the generated chunk key, sending the encrypted chunk payload to the chunk server, encrypting the chunk key for the chunk server, and sending encrypted chunk key to the chunk server.

15. The method of claim 14 further comprising:

receiving the encrypted chunk payload and the encrypted chunk key by the chunk server; and

validating the encrypted chunk payload by the chunk server, wherein the validating is performed by decrypting the encrypted chunk key, decrypting the received payload with the decrypted chunk key, calculating the cryptographic hash fingerprint of the decrypted chunk payload, comparing the calculated fingerprint with the fingerprint identifying the chunk, and declaring the chunk to be invalid if said fingerprints do not match.

16. The method of claim 15, wherein if said fingerprints do match, then

storing the encrypted chunk payload as a chunk image at the chunk server,

encrypting the chunk key with a private key shared with no other server,

including the encrypted chunk key in chunk metadata for the chunk image, and

writing the chunk metadata at the chunk server.

17. A method of retrieving a chunk, the method comprising:

receiving a chunk get request from a client/proxy to get a chunk from a chunk server of a chunk storage system;

using a chunk identifier to determine if the chunk is already stored locally as a local chunk image at the chunk server;

if the chunk is already stored locally at the chunk server, then performing a first procedure to use the local chunk image;

if the chunk is not already stored locally at the chunk server, and the chunk server is not designated to store the chunk identifier by a consistent hashing algorithm, then performing a second procedure to obtain a copy of a chunk image from a designated chunk server of the chunk storage system; and

if the chunk is not stored in the chunk storage system, then returning an error message from the chunk server to the client/proxy.

18. The method of claim 17, wherein the first procedure comprises:

reading the encrypted chunk payload;

forwarding the encrypted chunk payload as a payload in response to the chunk get request;

reading chunk metadata to obtain the encrypted chunk key;

decrypting the encrypted chunk key to obtain an unencrypted chunk key;

re-encrypting the unencrypted chunk key for the requesting client/proxy or user to obtain a re-encrypted chunk key;

appending a metadata field to the chunk metadata, the metadata field identifying the local chunk image; and

sending the re-encrypted chunk key in the chunk metadata in further response to the chunk get request.

19. The method of claim 18, wherein the second procedure comprises:

sending a chunk get request to a designated chunk server;

receiving the copy of the chunk image from the designated chunk server;

saving the copy of the chunk image locally;

forwarding a chunk payload of the chunk image to the client/proxy in response to the chunk get request;

decrypting a chunk key of the chunk image;

re-encrypting the chunk key for the client/proxy or user;

appending a metadata field to the chunk image, wherein the metadata field identifies the chunk image as being a copy of the chunk image at the designated chunk server; and

sending the re-encrypted chunk key to the client/proxy as metadata in further response to the chunk get request.

20. The method of claim 19 further comprising:

receiving the re-encrypted chunk key at the client/proxy;

decrypting the re-encrypted chunk key by the client/proxy to obtain the chunk key;

receiving the encrypted chunk payload at the client/proxy; and

decrypting the encrypted chunk payload at the client/proxy using the chunk key to obtain a chunk payload.

21. The method of claim 20 further comprising:

validating the chunk payload at the client/proxy by generating a fingerprint of the chunk payload and comparing the fingerprint against the chunk identifier;

if validation fails because the fingerprint does not match the chunk identifier, then sending an invalid payload notification to the chunk server, and re-sending the chunk get request to the chunk server.

22. The method of claim 21, wherein if the invalid payload notification is received by the chunk server, the chunk server responds by

if the chunk payload was obtained from the designated chunk server, then relaying the invalid payload notification to the designated chunk server and erasing all local copies of the chunk;

and

if the chunk payload is from a local chunk image, then validating the local chunk image, and if the local chunk image is invalid, then repairing or replacing the local chunk image using other local and/or network replicas of the chunk.