Distributed storage of data

• US 9,575,846 B2
• Filed: 07/24/2014
• Issued: 02/21/2017
• Est. Priority Date: 07/24/2014
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Tian, Chao
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US14/339,757
• Publication Number:
  US 20160026543A1
• Time in Patent Office: 943 Days
• Field of Search: 714/6.24, 714/6.1, 714/6.2, 714/15, 714/42, 714/49, 714/751, 714/752, 714/784
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/154 Error and erasure correction, e.g. by using the error and erasure locator or Forney polynomial
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/63 Joint error correction and other techniques H03M13/31 and H03M13/33 take precedence
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Multi-reliability regenerating (MRR) erasure codes are disclosed. The erasure codes can be used to encode and regenerate data. In particular, the regenerating erasure codes can be used to encode data included in at least one of two or more data messages to satisfy respective reliability requirements for the data. Encoded portions of data from one data message can be mixed with encoded or unencoded portions of data from a second data message and stored at a distributed storage system. This approach can be used to improve efficiency and performance of data storage and recovery in the event of failures of one or more nodes of a distributed storage system.

Claims

1. A system, comprising:

a processor; and

a memory that stores instructions that, when executed by the processor, cause the processor to perform operations comprising

obtaining a first set of data and a second set of data,

encoding the first set of data using an erasure code to generate a first set of coded symbols that represent the first set of data,

mixing the first set of coded symbols with the second set of data to generate a plurality of combined data values, wherein the second set of data is unencoded, and

transmitting the plurality of combined data values, the transmitting directed to a plurality of nodes of a distributed storage system.

2. The system of claim 1, wherein the instructions, when executed by the processor, cause the processor to perform operations further comprising:

detecting a failure of one of the plurality of nodes;

retrieving two of the plurality of combined data values from two of the plurality of nodes;

determining, based on the two of the plurality of combined data values, two of the first set of coded symbols, and

recovering the first set of data by decoding the two of the first set of coded symbols using the erasure code.

3. The system of claim 1, wherein the instructions, when executed by the processor, cause the processor to perform operations further comprising storing, in a database, a mapping of the plurality of combined data values to a plurality of node addresses that correspond to the plurality of nodes.

4. The system of claim 1, wherein the plurality of nodes comprises four nodes, and wherein the first set of data is recovered using three of the plurality of combined data values obtained from three of the four nodes.

5. The system of claim 1, wherein transmitting the plurality of combined data values comprises sending, to the plurality of nodes, a plurality of data messages that includes the plurality of combined data values.

6. The system of claim 1, wherein one of the plurality of combined data values comprises a sum of one of the first set of coded symbols and one of the second set of data, and wherein another of the plurality of combined data values comprises the one of the second set of data.

7. The system of claim 6, wherein the instructions, when executed by the processor, cause the processor to perform operations further comprising:

determining the one of the first set of coded symbols based upon the one of the plurality of combined data values and the other of the plurality of combined data values; and

decoding the one of the first set of coded symbols using the erasure code.

8. A method, comprising:

obtaining, at a storage server comprising a processor and a memory, a first set of data and a second set of data;

encoding, by the storage server and using an erasure code, the first set of data to generate a first set of coded symbols that represent the first set of data;

mixing, by the storage server, the first set of coded symbols with the second set of data to generate a plurality of combined data values, wherein the second set of data is unencoded; and

transmitting, by the storage server, the plurality of combined data values to a plurality of nodes of a distributed storage system.

9. The method of claim 8, further comprising:

detecting a failure of one of the plurality of nodes;

retrieving two of the plurality of combined data values from two of the plurality of nodes;

determining, based on the two of the plurality of combined data values, two of the first set of coded symbols, and

recovering the first set of data by decoding the two of the first set of coded symbols using the erasure code.

10. The method of claim 8, wherein the plurality of nodes comprises four nodes, and wherein the first set of data is recovered using any three of the plurality of combined data values obtained from three of the four nodes.

11. The method of claim 8, further comprising:

storing, in a database, a mapping of the plurality of combined data values to a plurality of node addresses that correspond to the plurality of nodes.

12. The method of claim 8, wherein transmitting the plurality of combined data values comprises transmitting, to the plurality of nodes, a plurality of data messages that includes the plurality of combined data values.

13. The method of claim 8, wherein one of the plurality of combined data values comprises a sum of one of the first set of coded symbols and one of the second set of data, and wherein another of the plurality of combined data values comprises the one of the second set of data.

14. The method of claim 8, further comprising:

determining the one of the first set of coded symbols based upon the one of the plurality of combined data values and the other of the plurality of combined data values; and

decoding the one of the first set of coded symbols using the erasure code.

19. The method of claim 8, wherein each of the combined data values comprises a coded symbol and an unencoded data value included in the second set of data.

15. A memory that stores instructions that, when executed by a processor, cause the processor to perform operations comprising:

obtaining a first set of data and a second set of data;

encoding the first set of data using an erasure code to generate a first set of coded symbols that represent the first set of data;

mixing the first set of coded symbols with the second set of data to generate a plurality of combined data values, wherein the second set of data is unencoded; and

transmitting the plurality of combined data values at a plurality of nodes of a distributed storage system.

16. The memory of claim 15, wherein the instructions, when executed by the processor, cause the processor to perform operations further comprising:

detecting a failure of one of the plurality of nodes;

retrieving two of the plurality of combined data values from two of the plurality of nodes;

determining, based on the two of the plurality of combined data values, two of the first set of coded symbols, and

recovering the first set of data by decoding the two of the first set of coded symbols using the erasure code.

17. The memory of claim 15, wherein one of the plurality of combined data values comprises a sum of one of the first set of coded symbols and one of the second set of data, and wherein another of the plurality of combined data values comprises the one of the second set of data.

18. The memory of claim 15, wherein transmitting the plurality of combined data values comprises sending, to the plurality of nodes, a plurality of data messages that includes the plurality of combined data values.

20. The memory of claim 15, wherein the instructions, when executed by the processor, cause the processor to perform operations further comprising storing, in a database, a mapping of the plurality of combined data values to a plurality of node addresses that correspond to the plurality of nodes.