Data deduplication with adaptive erasure code redundancy

• US 10,484,016 B2
• Filed: 05/23/2017
• Issued: 11/19/2019
• Est. Priority Date: 07/09/2014
• Status: Active Grant

• Current Assignee: Quantum Corporation (Chi Ko Investment Co., Ltd.)
• Original Assignee: Quantum Corporation (Chi Ko Investment Co., Ltd.)
• Inventors: Wideman, Roderick B., Arslan, Suayb Sefik, Lee, Jaewook, Goker, Turguy
• Primary Examiner(s): Kerveros, James C
• Application Number:
  US15/602,726
• Publication Number:
  US 20170257119A1
• Time in Patent Office: 910 Days
• Field of Search: 714766
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0673 Single storage device
• CPC Class Codes: H03M 13/154 Error and erasure correction, e.g. by using the error and erasure locator or Forney polynomial
• CPC Class Codes: H03M 13/3707 Adaptive decoding and hybrid decoding, e.g. decoding methods or techniques providing more than one decoding algorithm for one code
• CPC Class Codes: H03M 13/373 with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes

Abstract

Example apparatus and methods combine erasure coding with data deduplication to simultaneously reduce the overall redundancy in data while increasing the redundancy of unique data. In one embodiment, an efficient representation of a data set is produced by deduplication. The efficient representation reduces duplicate data in the data set. Redundancy is then added back into the data set using erasure coding. The redundancy that is added back in adds protection to the unique data associated with the efficient representation. How much redundancy is added back in and what type of redundancy is added back in may be controlled based on an attribute (e.g., value, reference count, symbol size, number of symbols) of the unique data. Decisions concerning how much and what type of redundancy to add back in may be adapted over time based, for example, on observations of the efficiency of the overall system.

Claims

1. A non-transitory computer-readable storage medium storing computer-executable instructions that when executed by a computer control the computer to perform a method, the method comprising:

generating, by a processor, a characterization of segments of a plurality of segments with failure probabilities associated with the segments that indicate a likelihood of failure of the segments;

parsing, by a processor, the plurality of segments to generate a plurality of original chunks;

deduplicating, by a processor, the plurality of original chunks to generate a plurality of unique chunks;

grouping, by a processor, the plurality of unique chunks to form grouped chunks;

encoding, by a processor, the grouped chunks to generate a desired number of erasure code symbols that satisfy the failure probabilities associated with the segments; and

selectively storing members of the desired number of erasure code symbols on a number of storage devices.

2. The non-transitory computer-readable storage medium of claim 1, further comprising:

calculating, by a processor, attributes of unique chunks of the plurality of unique chunks based, at least in part, on the failure probabilities; and

where the encoding is based, at least in part, on the attributes being constrained by the failure probabilities.

3. The non-transitory computer-readable storage medium of claim 2, where importance is an attribute of the attributes, and where importance is calculated based on a reference count to the data in a unique chunk.

4. The non-transitory computer-readable storage medium of claim 2, where the attributes are variable, and where the desired number of erasure code symbols changes when an attribute changes.

5. The non-transitory computer-readable storage medium of claim 2, where the number of storage devices is based, at least in part, on the attributes of the unique chunks.

6. The non-transitory computer-readable storage medium of claim 1, where an erasure code symbol represents a number of connections between at least one segment and an erasure code symbol.

7. The non-transitory computer-readable storage medium of claim 1, where the encoding is based on information stored in a generator matrix, and the method further comprising:

identifying, by a processor, attributes of unique chunks of the plurality of unique chunks; and

manipulating, by a processor, the generator matrix to control, at least in part, a size or composition of an erasure code symbol of the desired number of erasure code symbols.

8. A method comprising:

generating, by a processor, a characterization of segments of a plurality of segments with failure probabilities associated with the segments that indicate a likelihood of failure of the segments;

parsing, by a processor, the plurality of segments to generate a plurality of original chunks,

deduplicating, by a processor, the plurality of original chunks to generate a plurality of unique chunks;

calculating, by a processor, attributes of unique chunks of the plurality of unique chunks based, at least in part, on the failure probabilities associated with corresponding segments;

grouping, by a processor, the plurality of unique chunks to form a grouped chunk;

encoding, by a processor, the grouped chunk to generate a desired number of erasure code symbols based, at least in part, on the attributes being constrained by the failure probabilities; and

selectively storing members of the desired number of erasure code symbols on a number of storage devices, where the number of storage devices is based, at least in part, on the attributes for the unique chunks.

9. The method of claim 8, where the attributes include an importance of a unique chunk, a size of the unique chunk, or an age of the unique chunk.

10. The method of claim 8, where importance is an attribute, and where importance is calculated, by a processor, based on a reference count to the data in a unique chunk.

11. The method of claim 8, where the attributes are variable, and where the desired number of erasure code symbols changes when an attribute changes.

12. The method of claim 8, where the desired number of erasure code symbols is greater than or equal to a minimum threshold of erasure code symbols but less than the total possible erasure codes.

13. The method of claim 12, where the desired number of erasure code symbols, the minimum threshold of erasure code symbols, and the total possible erasure codes are directly proportional to a variation of the attributes.

14. The method of claim 8, where the encoding is based on information stored in a generator matrix, the method further comprising:

identifying, by a processor, attributes of unique chunks of the plurality unique chunks; and

manipulating, by a processor, the generator matrix to control, at least in part, a size or composition of an erasure code symbol of the desired number of erasure code symbols.

15. An apparatus, comprising:

a number of storage devices; and

a processor configured to;

generate a characterization of segments of a plurality of segments with failure probabilities associated with the segments that indicate a likelihood of failure of segments;

parse the plurality of segments to generate a plurality of original chunks,

deduplicate the plurality of original chunks to generate a plurality of unique chunks;

calculate attributes of unique chunks of the plurality of unique chunks based, at least in part, on the failure probabilities;

group the plurality of unique chunks to form at least one grouped chunk;

encode the at least one grouped chunk to generate a desired number of erasure code symbols that satisfy the failure probabilities associated with the segments, where the encoding is based, at least in part, on the attributes being constrained by the failure probabilities; and

selectively store members of the desired number of erasure code symbols on the number of storage devices, where the number of storage devices is based, at least in part, on the attributes for the unique chunks.

16. The apparatus of claim 15, where importance is an attribute of the attributes, and where importance is calculated based on a reference count to the data in a unique chunk.

17. The apparatus of claim 15, where the attributes are variable, and where the desired number of erasure code symbols changes when an attribute changes.

18. The apparatus of claim 15, where an erasure code symbol represents a number of connections between at least one segment and an erasure code symbol.

19. The apparatus of claim 15, where the encoding is based on information stored in a generator matrix, and where the processor is further configured to:

identify attributes of unique chunks of the plurality unique chunks; and

manipulate the generator matrix to control, at least in part, a size or composition of an erasure code symbol of the desired number of erasure code symbols.

20. The apparatus of claim 19, where a non-zero entry in the generator matrix represents a node/edge probability distribution.