Segment deduplication system with encryption and compression of segments

• US 8,731,190 B2
• Filed: 06/09/2009
• Issued: 05/20/2014
• Est. Priority Date: 06/09/2009
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Lumb, Christopher R.
• Primary Examiner(s): Rahman, Mohammad L
• Application Number:
  US12/455,956
• Publication Number:
  US 20100313040A1
• Time in Patent Office: 1,806 Days
• Field of Search: 380/217, 380/37, 380/269
• US Class Current: 380/37
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0623 in relation to content
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 2209/30 Compression, e.g. Merkle-Damgard construction
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 9/0618 Block ciphers, i.e. encrypting groups of characters of a plain text message using fixed encryption transformation

Abstract

A system for storing encrypted compressed data comprises a processor and a memory. The processor is configured to determine whether an encrypted compressed segment has been previously stored. The encrypted compressed segment was determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments. The processor is further configured to store the encrypted compressed segment in the event that the encrypted compressed segment has not been previously stored. The memory is coupled to the processor and configured to provide the processor with instructions.

Claims

1. A system for deduplicated storage of encrypted compressed data, comprising:

a processor configured to;

determine whether an encrypted compressed segment has been previously stored based at least in part on an identifier associated with the encrypted compressed segment, wherein the encrypted compressed segment was determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, and wherein one segment of the one or more segments is used to reconstruct more than one data stream, data block, or data file, wherein for a first segment and a second segment that is identical to the first segment, an encrypted compressed first segment is identical to an encrypted compressed second segment;

in the event that the encrypted compressed segment has not been previously stored, store the encrypted compressed segment;

in the event that the encrypted compressed segment has been previously stored, store a reference to the previously stored encrypted compressed segment, wherein the reference is used to retrieve the previously stored encrypted compressed segment to reconstruct the data stream, data block, or data file; and

a memory coupled to the processor and configured to provide the processor with instructions.

2. A system as in claim 1, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

3. A system as in claim 1, wherein compressing comprises compressing using one of the following:

a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.

4. A system as in claim 1, wherein encrypting comprises encrypting using one of the following:

a stream cipher system, a cipher feedback system, an electronic code book system, an AES system, or a DES system.

5. A method for deduplicated storage of encrypted compressed data, comprising:

determining, using a processor, whether an encrypted compressed segment has been previously stored based at least in part on an identifier associated with the encrypted compressed segment, wherein the encrypted compressed segment is determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, and wherein one segment of the one or more segments is used to reconstruct more than one data stream, data block, or data file, wherein for a first segment and a second segment that is identical to the first segment, an encrypted compressed first segment is identical to an encrypted compressed second segment;

in the event that the encrypted compressed segment has not been previously stored, storing the encrypted compressed segment; and

in the event that the encrypted compressed segment has been previously stored, storing a reference to the previously stored encrypted compressed segment, wherein the reference is used to retrieve the previously stored encrypted compressed segment to reconstruct the data stream, data block, or data file.

6. A method as in claim 5, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

7. A method as in claim 5, wherein compressing comprises compressing using one of the following:

a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.

8. A method as in claim 5, wherein encrypting comprises encrypting using one of the following:

a stream cipher system, a cipher feedback system, an electronic code book system, an AES system, or a DES system.

9. A computer program product for deduplicated storage of encrypted compressed data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

determining, using a processor, whether an encrypted compressed segment has been previously stored based at least in part on an identifier associated with the encrypted compressed segment, wherein the encrypted compressed segment is determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, and wherein one segment of the one or more segments is used to reconstruct more than one data stream, data block, or data file, wherein for a first segment and a second segment that is identical to the first segment, an encrypted compressed first segment is identical to an encrypted compressed second segment;

in the event that the encrypted compressed segment has not been previously stored, storing the encrypted compressed segment; and

in the event that the encrypted compressed segment has been previously stored, storing a reference to the previously stored encrypted compressed segment, wherein the reference is used to retrieve the previously stored encrypted compressed segment to reconstruct the data stream, data block, or data file.

10. A system for reading encrypted compressed data, comprising:

a deduplicated storage device;

a processor configured to;

decrypt one or more encrypted compressed segments stored in the deduplicated storage device, wherein the one or more encrypted compressed segments were determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, wherein one segment of the one or more segments is used to reconstruct more than one data stream, data block, or data file, and wherein the one or more encrypted compressed segments are each associated with a stored identifier, wherein for a first encrypted compressed segment and a second encrypted compressed segment that is identical to the first encrypted compressed segment, a decrypted decompressed first encrypted compressed segment is identical to a decrypted decompressed second encrypted compressed segment;

decompress the one or more decrypted compressed segments; and

assemble the one or more decrypted decompressed segments to reconstruct the data stream, the data block, or the data file, wherein one of the one or more decrypted decompressed segments is used to reconstruct a second data stream, a second data block, or a second data file; and

a memory coupled to the processor and configured to provide the processor with instructions.

11. A system as in claim 10, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

12. A system as in claim 10, wherein compressing comprises compressing using one of the following:

a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.

13. A system as in claim 10, wherein decrypting comprises decrypting using one of the following:

a stream cipher system, a cipher feedback system, an electronic code book system, an AES system, or a DES system.

14. A method for reading encrypted compressed data, comprising:

decrypting, using a processor, one or more encrypted compressed segments stored in a deduplicated storage device, wherein the one or more encrypted compressed segments were determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, wherein one segment of the one or more segments is used to reconstruct more than one data stream, data block, or data file, and wherein the one or more encrypted compressed segments are each associated with a stored identifier, wherein for a first encrypted compressed segment and a second encrypted compressed segment that is identical to the first encrypted compressed segment, a decrypted decompressed first encrypted compressed segment is identical to a decrypted decompressed second encrypted compressed segment;

decompressing the one or more decrypted compressed segments; and

assembling the one or more decrypted decompressed segments to reconstruct the data stream, the data block, or the data file, wherein at least one of the one or more decrypted decompressed segments is used to reconstruct a second data stream, a second data block, or a second data file.

15. A method as in claim 14, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

16. A method as in claim 14, wherein compressing comprises compressing using one of the following:

a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.

17. A method as in claim 14, wherein decrypting comprises decrypting using one of the following:

a stream cipher system, a cipher feedback system, an electronic code book system, an AES system, or a DES system.

18. A computer program product for reading encrypted compressed data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

decrypting, using a processor, one or more encrypted compressed segments stored on a deduplicated storage device, wherein the one or more encrypted compressed segments were determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, wherein one segment of the one or more segments is used to reconstruct more than one data stream, data block, or data file, and wherein the one or more encrypted compressed segments are each associated with a stored identifier, wherein for a first encrypted compressed segment and a second encrypted compressed segment that is identical to the first encrypted compressed segment, a decrypted decompressed first encrypted compressed segment is identical to a decrypted decompressed second encrypted compressed segment;

decompressing the one or more decrypted compressed segments; and

assembling the one or more decrypted decompressed segments to reconstruct the data stream, the data block, or the data file, wherein one of the one or more decrypted decompressed segments is used to reconstruct a second data stream, a second data block, or a second data file.