Meta chunks

• US 10,817,374 B2
• Filed: 04/12/2018
• Issued: 10/27/2020
• Est. Priority Date: 04/12/2018
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Danilov, Mikhail, Gusakov, Kirill
• Primary Examiner(s): Nguyen, Thien
• Application Number:
  US15/952,179
• Publication Number:
  US 20190317859A1
• Time in Patent Office: 929 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/154 Error and erasure correction, e.g. by using the error and erasure locator or Forney polynomial

Abstract

Data protection with meta chunks increases capacity use efficiency without verification and data copying. In one aspect, a meta chunk is a data protection unit, which combines two or more source chunks that are determined to have a reduced sets of data fragments. The meta chunk can be encoded to generate a set of coding fragments, which can be stored and utilized to recover data fragments of any of the two or more source chunks. Further, the source chunks can be linked to the meta chunk. Furthermore, the sets of coding fragments, that were previously generated by individually encoding each source chunk, can be deleted.

Claims

1. A system, comprising:

a processor; and

a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising;

determining source chunks stored within a storage system, wherein the source chunks are determined to have a collective number of data fragments that is less than a number of data fragments that a chunk is configured to be able to store;

based on combining the source chunks, generating a meta chunk; and

encoding the meta chunk to generate coding fragments that are employable to recover at least a portion of the source chunks,

wherein the encoding comprises erasure coding the meta chunk.

2. The system of claim 1, wherein the operations further comprise:

linking the source chunks to the meta chunk.

3. The system of claim 2, wherein the linking comprises updating metadata associated with a source chunk of the source chunks to comprise reference data indicative of the meta chunk.

4. The system of claim 1, wherein the coding fragments are meta chunk coding fragments and the operations further comprise:

subsequent to the encoding, deleting source chunk coding fragments that have been generated based on individually encoding the source chunks.

5. The system of claim 1, wherein the collective number of data fragments is a first collective number of data fragments, wherein the meta chunk is a first meta chunk, the source chunks are first source chunks, the first source chunks comprise a second meta chunk that has been generated based on a combination of second source chunks, and wherein the second source chunks are determined to have a second collective number of data fragments that is less than the number of data fragments.

6. The system of claim 1, wherein the generating the meta chunk comprises selecting ones of the source chunks such that the collective number of data fragments of the selected ones of the source chunks is closest to the number of data fragments.

7. The system of claim 1, wherein the erasure coding the meta chunk comprises generating, for k fragments of the meta chunk, m redundant coding fragments, k and m being positive integers and m being less than k.

8. The system of claim 7, further comprising:

performing recovery of an unavailable data fragment by utilizing mapping information that maps the source chunks to the meta chunk.

9. The system of claim 1, wherein physical capacity is not allocated for the meta chunk.

10. The system of claim 8, wherein the performing the recovery comprises employing a decoding matrix that corresponds to a coding matrix that was utilized in the erasure coding.

11. The system of claim 1, wherein the generating the meta chunk comprises:

creating a layout within the meta chunk that maps data fragments of the source chunks to the meta chunk independently of modifying a storage location within the source chunks of the data fragments.

12. The system of claim 1, wherein the generating the meta chunk comprises:

maintaining locations of the data chunks within the source chunks, the meta chunk referring to the locations of the data chunks.

13. A method, comprising:

selecting, by a system comprising a processor, source chunks from chunks of an object storage system, wherein the source chunks are determined to combine to have fewer data fragments than a first number of data fragments that are able to be stored in a chunk;

combining, by the system, the source chunks to generate a meta chunk; and

based on erasure coding the meta chunk, determining, by the system, coding fragments for the source chunks at a meta chunk level, wherein the coding fragments are to be employed to recover at least a portion of the source chunks during a failure condition.

14. The method of claim 13, further comprising:

storing, by the system, reference data that links the source chunks to the meta chunk.

15. The method of claim 14, further comprising:

based on the reference data, determining, by the system, the meta chunk linked to at least the portion of the source chunks.

16. The method of claim 15, further comprising:

recovering, by the system, at least the portion of the source chunks based on performing a decoding operation at the meta chunk level.

17. The method of claim 13, wherein the coding fragments are first coding fragments, and the method further comprises:

subsequent to the determining, deleting, by the system, source second fragments that have been generated based on individually erasure coding the source chunks.

18. A non-transitory computer-readable storage medium comprising instructions that, in response to execution, cause a server device comprising a processor to perform operations, comprising:

combining source chunks stored within an object storage system to generate a meta chunk, wherein the source chunks are determined not to have more than a defined number of data fragments equal to a number of data fragments configured to be stored in a chunk; and

based on erasure coding the meta chunk, determining coding fragments for the source chunks at a meta chunk level, wherein the coding fragments are to be employed to recover at least a portion of the source chunks during a failure condition.

19. The non-transitory computer-readable storage medium of claim 18, wherein the operations further comprise:

determining reference data that links the source chunks to the meta chunk.

20. The non-transitory computer-readable storage medium of claim 19, wherein the operations further comprise:

updating metadata associated with the source chunks the reference data.