Sub-volume level security for deduplicated data

• US 8,117,464 B1
• Filed: 04/30/2008
• Issued: 02/14/2012
• Est. Priority Date: 04/30/2008
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Kogelnik, Christoph
• Primary Examiner(s): Homayounmehr, Farid
• Assistant Examiner(s): Lavelle, Gary
• Application Number:
  US12/113,122
• Time in Patent Office: 1,385 Days
• Field of Search: 380/45, 380/277, 380/282, 380/284, 380/285, 713/153, 713/160, 713/165, 713/171, 713/180, 713/193, 726/26, 726/30
• US Class Current: 713/193
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A network storage server receives write requests from clients via a network and internally buffers data blocks written by the write requests. At a consistency point, the storage server commits the data blocks to nonvolatile mass storage. In the consistency point process, a storage operating system in the network storage server compresses the data blocks, encrypts selected data blocks, and stores the compressed and (possibly) encrypted data blocks in the nonvolatile mass storage facility. Data blocks can also be fingerprinted in parallel with compression and/or encryption, to facilitate deduplication. Data blocks can be indexed and classified according to content or attributes of the data. Encryption can be applied at different levels of logical container granularity, where a separate, unique cryptographic key is used for each encrypted data container. To facilitate deduplication, the system creates an additional, shared encryption key for each data block duplicated between two or more logical containers.

Claims

1. A network storage system comprising:

a hardware-implemented processor;

a storage manager, operatively coupled to the processor, to service requests to access data stored in a persistent mass storage facility;

a duplicate detection unit to detect a duplicate block of a first logical container of data and to determine whether the duplicate block is also part of a second logical container of data; and

a cryptographic engine to

encrypt a unique block of the first logical container of data by using a unique cryptographic key,

encrypt the duplicate block by using a shared cryptographic key and the unique cryptographic key only if the duplicate block is determined also to be part of the second logical container of data, and otherwise to encrypt the duplicate block using the unique cryptographic key but not the shared cryptographic key if the duplicate block is determined not also to be part of the second logical container of data, and

in conjunction with servicing a request to access the first logical container of data, decrypt the unique block of the first logical container of data by using the unique cryptographic key and decrypt the duplicate block of the first logical container of data by using the shared cryptographic key.

2. A network storage system as recited in claim 1, wherein the shared cryptographic key is shared with the second logical container of data.

3. A network storage system as recited in claim 2, wherein the cryptographic engine further is to:

use the shared cryptographic key to decrypt the duplicate block as part of accessing the second logical container of data.

4. A network storage system as recited in claim 1, wherein unique portions of the second logical container are encrypted using a second unique cryptographic key.

5. A network storage system as recited in claim 1, wherein the cryptographic engine further is to:

use the unique cryptographic key to encrypt and decrypt the shared cryptographic key.

6. A network storage system as recited in claim 1, wherein the cryptographic engine further is to:

encrypt each of a plurality of unique portions of the first logical container of data with said unique cryptographic key; and

use said unique cryptographic key to decrypt each of the plurality of unique portions as part of accessing the first logical container of data.

7. A method comprising:

receiving a first logical container of data at a storage system, the first logical container of data including a first plurality of blocks;

generating a fingerprint for each of the first plurality of blocks and storing results of said generating as a plurality of stored fingerprints;

encrypting blocks of a first logical container of data with a unique first cryptographic key;

storing the first logical container of data as an encrypted first plurality of blocks in a persistent storage facility;

receiving a second logical container of data at the storage system, the second logical container of data including a second plurality of blocks;

generating a fingerprint for a particular block of the second plurality of blocks;

determining whether the particular block is a duplicate of a stored block, by comparing the fingerprint of the particular block with the stored fingerprints;

determining whether the particular block belongs to a logical container to which the stored block belongs;

if the particular block is a duplicate of the stored block and does not belong to a file to which the stored block belongs, then using a second cryptographic key that is shared by a plurality of files and a unique third cryptographic key that is not shared between files to encrypt the particular block, and otherwise using the unique third cryptographic key without using the second cryptographic key to encrypt the particular block;

encrypting other blocks of the second logical container of data with the unique third cryptographic key;

storing the encrypted other blocks of the second logical container in the persistent storage facility;

in response to a request to access data in the first logical container including the particular block when the particular block is a duplicate block and belongs to both the first logical container and the second logical container, using the first cryptographic key and the shared second cryptographic key to access the particular block, and using the first cryptographic key without the shared second cryptographic key to access other blocks of the first logical container; and

in response to a request to access data in the second logical container including the particular block when the particular block is a duplicate block and belongs to both the first logical container and the second logical container, using the shared second cryptographic key and the third cryptographic key to access said block, and using the third cryptographic key without the shared second cryptographic key to access other blocks of the second logical container.

8. A method as recited in claim 7, wherein using the second cryptographic key to encrypt the particular block comprises:

reading the particular block from the first logical container of data in the persistent storage facility;

decrypting the particular block with the first cryptographic key;

re-encrypting the particular block with the second cryptographic key;

writing the particular block re-encrypted with the second cryptographic key back to the persistent storage facility as part of the first logical container; and

setting a pointer associated with the second logical container to point to the particular block, encrypted with the second cryptographic key, in the persistent storage facility.

9. A method as recited in claim 7, wherein using the second cryptographic key to encrypt the particular block comprises:

encrypting the particular block with the second cryptographic key;

writing the particular block encrypted with the second cryptographic key to the persistent storage facility as part of the second logical container; and

updating a pointer associated with the first logical container to point to the particular block, encrypted with the second cryptographic key, in the persistent storage facility.

10. A method as recited in claim 7, wherein using the second cryptographic key that is shared by the first and second logical containers of data comprises:

storing the second cryptographic key or a reference thereto in metadata of the first logical container of data; and

storing the second cryptographic key or a reference thereto in metadata of the second logical container of data.

11. A method comprising:

generating a fingerprint for each of a first plurality of blocks of data to be written to a persistent storage facility in a storage system;

determining whether a first block of the first plurality of blocks is a duplicate of any block of a second plurality of blocks of data stored in the storage system, by comparing a fingerprint of the first block with a plurality of stored fingerprints, the stored fingerprints including a fingerprint for each block of the second plurality of blocks;

if the first block is found to be a duplicate of a block of the second plurality of blocks, then determining whether the first block is part of a same logical container as said block of the second plurality of blocks;

if the first block is found to be a duplicate of a block of the second plurality of blocks but is not part of the same logical container as said block of the second plurality of blocks, then using a shared cryptographic key that is shared by a plurality of logical containers that each include the first block to encrypt the first block;

but if the first block is a duplicate of a block of the second plurality of blocks and is part of the same logical container as said block of the second plurality of blocks, then using a second cryptographic key different from the shared cryptographic key, and not using the shared cryptographic key, to encrypt the first block; and

storing the encrypted first block in the persistent storage facility.

12. A method as recited in claim 11, wherein determining whether the first block is part of the same logical container as said block of the second plurality of blocks comprises:

determining whether the first block is part of a same file as said block of the second plurality of blocks.

13. A method as recited in claim 12, further comprising, when the first block is a duplicate of a block of the second plurality of blocks but is not part of the same file as said block of the second plurality of blocks:

using the shared cryptographic key and the second cryptographic key to access the block as part of accessing said same file; and

using the shared cryptographic key and a third cryptographic key to access the first block as part of accessing a second file that contains the first block.

14. A method as recited in claim 13, wherein using the shared cryptographic key to encrypt the block comprises:

using the second cryptographic key to encrypt the shared cryptographic key and associating a result thereof with said same file as a first encrypted shared key; and

using the third cryptographic key to encrypt the shared cryptographic key and associating a result thereof with the second file as a second encrypted shared key.

15. A method as recited in claim 14, wherein using the shared cryptographic key to access the block further comprises:

using the third cryptographic key to decrypt the first encrypted shared key, and using a result thereof to decrypt the first block as part of as part of accessing said same logical container; and

using the fourth cryptographic key to decrypt the second encrypted shared key, and using a result thereof to decrypt the first block as part of accessing the second logical container.

16. A method as recited in claim 12, further comprising:

storing the shared cryptographic key or a reference thereto in metadata of said same logical container of data; and

storing the shared cryptographic key or a reference thereto in metadata of the second logical container.

17. A method as recited in claim 12, further comprising:

maintaining a plurality of logical containers at a plurality of levels of granularity in the storage system; and

allowing logical containers at each of the plurality of levels of granularity to be encrypted, by allowing separate encryption keys to be assigned to logical containers in each of the plurality of levels of granularity, wherein the plurality of levels of granularity include file level, directory level and volume level.

18. A method as recited in claim 17, wherein allowing logical containers at each of the plurality of levels of granularity to be encrypted comprises:

encrypting data blocks of a file with a first encryption key;

encrypting a directory that contains the file with a second encryption key; and

encrypting a volume that contains the directory with a third encryption key.

19. A network storage controller comprising:

a communication interface through which to receive a plurality of data access requests from a plurality of hosts;

a storage interface through which to access a persistent storage facility;

a hardware-implemented processor; and

a storage medium storing code which, when executed by the processor, causes the storage controller to perform operations including

generating a fingerprint for each of a first plurality of blocks of data to be written to a persistent storage facility in a storage system;

determining whether a first block of the first plurality of blocks is a duplicate of any block of a second plurality of blocks of data stored in the storage system, by comparing a fingerprint of the first block with a plurality of stored fingerprints, the stored fingerprints including a fingerprint for each block of the second plurality of blocks;

if the first block is found to be a duplicate of a block of the second plurality of blocks, then determining whether the first block is part of a same file as said block of the second plurality of blocks;

if the first block is found to be a duplicate of a block of the second plurality of blocks but is not part of the same file as said block of the second plurality of blocks, then using a shared cryptographic key that is shared by a plurality of files that each include the first block and a second cryptographic key different from the shared cryptographic key, to encrypt the first block;

but if the first block is a duplicate of a block of the second plurality of blocks and is part of the same file as said block of the second plurality of blocks, then using the second cryptographic key and not using the shared cryptographic key, to encrypt the first block; and

storing the encrypted first block in the persistent storage facility.

20. A network storage controller as recited in claim 19, further comprising:

maintaining a plurality of logical containers at a plurality of levels of granularity in the storage system; and

allowing logical containers at each of the plurality of levels of granularity to be encrypted, by allowing separate encryption keys to be assigned to logical containers in each of the plurality of levels of granularity, wherein the plurality of levels of granularity include file level, directory level and volume level.

21. A network storage controller as recited in claim 20, wherein allowing logical containers at each of the plurality of levels of granularity to be encrypted comprises:

encrypting data blocks of a file with a first encryption key;

encrypting a directory that contains the file with a second encryption key; and

encrypting a volume that contains the directory with a third encryption key.

22. A method as recited in claim 19, further comprising, when the first block is a duplicate of a block of the second plurality of blocks but is not part of the same file as said block of the second plurality of blocks:

using the shared cryptographic key and the second cryptographic key to access the block as part of accessing said same file; and

using the shared cryptographic key and a third cryptographic key to access the first block as part of accessing a second file that contains the first block.

23. A method as recited in claim 19, wherein using the shared cryptographic key to encrypt the block comprises:

using the second cryptographic key to encrypt the shared cryptographic key and associating a result thereof with said same file as a first encrypted shared key; and

using the third cryptographic key to encrypt the shared cryptographic key and associating a result thereof with the second file as a second encrypted shared key.