Segment deduplication system with compression of segments

• US 9,436,723 B2
• Filed: 05/06/2014
• Issued: 09/06/2016
• Est. Priority Date: 06/09/2009
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Lumb, Christopher R.
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): Le, Jessica N
• Application Number:
  US14/271,246
• Publication Number:
  US 20140330797A1
• Time in Patent Office: 854 Days
• Field of Search: 707/674, 707/687
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0683 Plurality of storage devices
• CPC Class Codes: H03M 7/30 Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
• CPC Class Codes: H03M 7/3064 Segmenting

Abstract

A system for storing compressed data comprises a processor and a memory. The processor is configured to receive a compressed segment. The compressed segment is determined by breaking a data stream, a data block, or a data file into one or more segments and compressing each of the one or more segments. The processor is further configured to determine whether the compressed segment has been previously stored, and in the event that the compressed segment has not been previously stored, store the compressed segment. The memory is coupled to the processor and configured to provide the processor with instructions.

Claims

1. A system for reading compressed data, comprising:

a deduplicated storage device;

a processor configured to;

receive a request for a data stream, a data block, or a data file to be retrieved;

retrieve one or more compressed segments and corresponding metadata information for each compressed segment,

wherein the corresponding metadata information comprises a compression type and a compression state from the deduplicated storage device using mapping information,

wherein the compression state comprises a state used for compressing the compressed segment, wherein the compression state comprises a 128 bit key, a 256 bit key, or an alphanumeric key,

wherein the one or more compressed segments were determined by breaking the data stream, the data block, or the data file into one or more segments and compressing each of the one or more segments,

wherein the deduplicated storage device stores compressed segments that have not been previously stored and references to compressed segments that have been determined to be identical to a previously stored compressed segment,

wherein the deduplicated storage device stores compressed segments that have not been previously stored with an index entry generated from the compressed segments,

wherein the index entry comprises one or more of the following;

a digital fingerprint, SHA-1 hash, or Rabin Hash of each unique compressed segment,

wherein the deduplicated storage device further retrieves the one or more compressed segments indicated by the mapping information using index entries of compressed segments in the deduplicated storage;

decompress the one or more compressed segments using the segment compression information retrieved from the deduplicated storage device,

wherein the segment compression information comprises a segment identifier and a compression type; and

assemble the one or more compressed segments to reconstruct the data stream, the data block, or the data file using the mapping information,

wherein the mapping information comprises a mapping of segment identifiers and the data stream, the data block, or the data file; and

a memory coupled to the processor and configured to provide the processor with instructions.

2. A system as in claim 1, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

3. A system as in claim 1, wherein the one or more compressed segments were compressed and decompressed with one of the following:

a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.

4. A system as in claim 1, wherein for a first compressed segment and a second compressed segment that is identical to the first compressed segment, a decompressed first compressed segment is identical to a decompressed second compressed segment.

5. A method for reading compressed data, comprising:

receiving a request for a data stream, a data block, or a data file to be retrieved;

retrieving one or more compressed segments and corresponding metadata information for each compressed segment,

wherein the corresponding metadata information comprises a compression type and a compression state from a deduplicated storage device using mapping information,

wherein the compression state comprises a state used for compressing the compressed segment, wherein the compression state comprises a 128 bit key, a 256 bit key, or an alphanumeric key,

wherein the one or more compressed segments were determined by breaking the data stream, the data block, or the data file into one or more segments and compressing each of the one or more segments,

wherein the deduplicated storage device stores compressed segments that have not been previously stored and references to compressed segments that have been determined to be identical to a previously stored compressed segment,

wherein the deduplicated storage device stores compressed segments that have not been previously stored with an index entry generated from the compressed segments,

wherein the index entry comprises one or more of the following;

a digital fingerprint, SHA-1 hash, or Rabin Hash of each unique compressed segment,

wherein the deduplicated storage device further retrieves the one or more compressed segments indicated by the mapping information using index entries of compressed segments in the deduplicated storage;

decompressing, using a processor, the one or more compressed segments using the segment compression information retrieved from the deduplicated storage device,

wherein the segment compression information comprises a segment identifier and a compression type; and

assembling the one or more decompressed segments to reconstruct the data stream, the data block, or the data file using the mapping information,

wherein the mapping information comprises a mapping of segment identifiers and the data stream, the data block, or the data file.

6. A method as in claim 5, wherein the breaking the data stream, the data block, or the data file is based at least in part on one of the following:

a content-based technique or a non-content-based technique.

7. A method as in claim 5, wherein the one or more compressed segments were compressed and decompressed with one of the following:

a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.

8. A method as in claim 5, wherein for a first compressed segment and a second compressed segment that is identical to the first compressed segment, a decompressed first compressed segment is identical to a decompressed second compressed segment.

9. A system as in claim 5, wherein the deduplicating storage device stores metadata information associated with the segment.

10. A system as in claim 9, wherein the metadata information comprises one or more of the following:

the segment identifier, a source identifier, a user identifier, a host identifier, a session identifier, a format type, an associated file, an associated data stream, or an associated data block.

11. A system as in claim 5, wherein the processor is further configured to provide the data stream, the data block, or the data file to a user.

12. A computer program product for reading compressed data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving a request for a data stream, a data block, or a data file to be retrieved;

retrieving one or more compressed segments and corresponding metadata information for each compressed segment,

wherein the corresponding metadata information comprises a compression type and a compression state from a deduplicated storage device using mapping information,

wherein the compression state comprises a state used for compressing the compressed segment, wherein the compression state comprises a 128 bit key, a 256 bit key, or an alphanumeric key,

wherein the one or more compressed segments were determined by breaking the data stream, the data block, or the data file into one or more segments and compressing each of the one or more segments,

wherein the deduplicated storage device stores compressed segments that have not been previously stored and references to compressed segments that have been determined to be identical to a previously stored compressed segment,

wherein the deduplicated storage device stores compressed segments that have not been previously stored with an index entry generated from the compressed segments,

wherein the index entry comprises one or more of the following;

a digital fingerprint, SHA-1 hash, or Rabin Hash of each unique compressed segment,

wherein the deduplicated storage device further retrieves the one or more compressed segments indicated by the mapping information using index entries of compressed segments in the deduplicated storage;

decompressing, using a processor, the one or more compressed segments using the segment compression information retrieved from the deduplicated storage device,

wherein the segment compression information comprises a segment identifier and a compression type; and

assembling the one or more decompressed segments to reconstruct the data stream, the data block, or the data file using the mapping information,

wherein the mapping information comprises a mapping of segment identifiers and the data stream, the data block, or the data file.