Segment deduplication system with encryption and compression of segments

• US 9,275,251 B2
• Filed: 04/03/2014
• Issued: 03/01/2016
• Est. Priority Date: 06/09/2009
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Lumb, Christopher R.
• Primary Examiner(s): Rahman, Mohammad L
• Application Number:
  US14/244,777
• Publication Number:
  US 20140325216A1
• Time in Patent Office: 698 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0623 in relation to content
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 2209/30 Compression, e.g. Merkle-Damgard construction
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 9/0618 Block ciphers, i.e. encrypting groups of characters of a plain text message using fixed encryption transformation

Abstract

A system for storing encrypted compressed data comprises a processor and a memory. The processor is configured to determine whether an encrypted compressed segment has been previously stored. The encrypted compressed segment was determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments. The processor is further configured to store the encrypted compressed segment in the event that the encrypted compressed segment has not been previously stored. The memory is coupled to the processor and configured to provide the processor with instructions.

Claims

1. A system for storing encrypted compressed data, comprising:

a processor configured to;

determine whether an encrypted compressed segment has been previously stored, wherein the encrypted compressed segment was determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, wherein the determining of whether the encrypted compressed segment has been previously stored comprises;

compare a first identifier corresponding to the encrypted compressed segment with a set of identifiers corresponding to a plurality of previously stored encrypted compressed segments;

in the event that the first identifier does not match a second identifier of the set of identifiers, store the encrypted compressed segment; and

in the event that the first identifier matches the second identifier of the set of identifiers, store a reference for the encrypted compressed segment, wherein the reference relates to a previously stored encrypted compressed segment corresponding to the second identifier; and

a memory coupled to the processor and configured to provide the processor with instructions.

2. A system as in claim 1, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

3. A system as in claim 1, wherein compressing comprises compressing using one of the following:

a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.

4. A system as in claim 1, wherein encrypting comprises encrypting using one of the following:

a stream cipher system, a cipher feedback system, an electronic code book system, an AES system, or a DES system.

5. A system as in claim 1, wherein for a first segment and a second segment that is identical to the first segment, an encrypted compressed first segment is identical to an encrypted compressed second segment.

6. A method for storing encrypted compressed data, comprising:

determining, using a processor, whether an encrypted compressed segment has been previously stored, wherein the encrypted compressed segment is determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, wherein the determining of whether the encrypted compressed segment has been previously stored comprises;

comparing a first identifier corresponding to the encrypted compressed segment with a set of identifiers corresponding to a plurality of previously stored encrypted compressed segments;

in the event that the first identifier does not match a second identifier of the set of identifiers, storing the encrypted compressed segment; and

in the event that the first identifier matches the second identifier of the set of identifiers, storing a reference for the encrypted compressed segment, wherein the reference relates to a previously stored encrypted compressed segment corresponding to the second identifier.

7. A method as in claim 6, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

8. A method as in claim 6, wherein compressing comprises compressing using one of the following:

a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.

9. A method as in claim 6, wherein encrypting comprises encrypting using one of the following:

a stream cipher system, a cipher feedback system, an electronic code book system, an AES system, or a DES system.

10. A method as in claim 6, wherein for a first segment and a second segment that is identical to the first segment, an encrypted compressed first segment is identical to an encrypted compressed second segment.

11. A computer program product for storing encrypted compressed data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

determining, using a processor, whether an encrypted compressed segment has been previously stored, wherein the encrypted compressed segment is determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, wherein the determining of whether the encrypted compressed segment has been previously stored comprises;

comparing a first identifier corresponding to the encrypted compressed segment with a set of identifiers corresponding to a plurality of previously stored encrypted compressed segments;

in the event that the first identifier does not match a second identifier of the set of identifiers, storing the encrypted compressed segment; and

in the event that the first identifier matches the second identifier of the set of identifiers, storing a reference for the encrypted compressed segment, wherein the reference relates to a previously stored encrypted compressed segment corresponding to the second identifier.

12. A system for reading encrypted compressed data, comprising:

a deduplicated storage device;

a processor configured to;

decrypt one or more encrypted compressed segments stored in the deduplicated storage device, wherein the one or more encrypted compressed segments were determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, wherein the encrypting of the each of the one or more segments comprises to store the each of the one or more segments, comprising to;

determine whether an encrypted compressed segment has been previously stored, comprising to

compare a first identifier corresponding to the encrypted compressed segment with a set of identifiers corresponding to a plurality of stored encrypted compressed segments;

in the event that the first identifier does not match a second identifier of the set of identifiers, store the encrypted compressed segment in the deduplicated storage device; and

in the event that the first identifier matches the second identifier of the set of identifiers, store a reference for the encrypted compressed segment in the deduplicated storage device, wherein the reference relates to a previously stored encrypted compressed segment corresponding to the second identifier;

decompress the one or more decrypted compressed segments; and

assemble the one or more decrypted decompressed segments to reconstruct the data stream, the data block, or the data file; and

a memory coupled to the processor and configured to provide the processor with instructions.

13. A system as in claim 12, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

14. A system as in claim 12, wherein compressing comprises compressing using one of the following:

a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.

15. A system as in claim 12, wherein decrypting comprises decrypting using one of the following:

a stream cipher system, a cipher feedback system, an electronic code book system, an AES system, or a DES system.

16. A system as in claim 12, wherein for a first encrypted compressed segment and a second encrypted compressed segment that is identical to the first encrypted compressed segment, a decrypted decompressed first encrypted compressed segment is identical to a decrypted decompressed second encrypted compressed segment.

17. A method for reading encrypted data, comprising:

decrypting, using a processor, one or more encrypted compressed segments stored in a deduplicated storage device, wherein the one or more encrypted compressed segments were determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, wherein the encrypting of the each of the one or more segments comprises storing the each of the one or more segments, comprising;

determining whether an encrypted compressed segment has been previously stored, comprising;

comparing a first identifier corresponding to the encrypted compressed segment with a set of identifiers corresponding to a plurality of stored encrypted compressed segments;

in the event that the first identifier does not match a second identifier of the set of identifiers, storing the encrypted compressed segment in the deduplicated storage device; and

in the event that the first identifier matches the second identifier of the set of identifiers, storing a reference for the encrypted compressed segment in the deduplicated storage device, wherein the reference relates to a previously stored encrypted compressed segment corresponding to the second identifier;

decompressing the one or more decrypted compressed segments; and

assembling the one or more decrypted decompressed segments to reconstruct the data stream, the data block, or the data file.

18. A method as in claim 17, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

19. A method as in claim 17, wherein for a first encrypted compressed segment and a second encrypted compressed segment that is identical to the first encrypted compressed segment, a decrypted decompressed first encrypted compressed segment is identical to a decrypted decompressed second encrypted decompressed segment.

20. A computer program product for reading encrypted compressed data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

decrypting, using a processor, one or more encrypted compressed segments stored on a deduplicated storage device, wherein the one or more encrypted compressed segments were determined by breaking a data stream, a data block, or a data file into one or more segments and compressing and then encrypting each of the one or more segments, wherein the encrypting of the each of the one or more segments comprises storing the each of the one or more segments, comprising;

determining whether an encrypted compressed segment has been previously stored, comprising;

comparing a first identifier corresponding to the encrypted compressed segment with a set of identifiers corresponding to a plurality of stored encrypted compressed segments;

in the event that the first identifier does not match a second identifier of the set of identifiers, storing the encrypted compressed segment in the deduplicated storage device;

an

in the event that the first identifier matches the second identifier of the set of identifiers, storing a reference for the encrypted compressed segment in the deduplicated storage device, wherein the reference relates to a previously stored encrypted compressed segment corresponding to the second identifier

decompressing the one or more decrypted compressed segments;

an

assembling the one or more decrypted decompressed segments to reconstruct the data stream, the data block, or the data file.