Distributed storage of data

• US 9,952,952 B2
• Filed: 02/13/2017
• Issued: 04/24/2018
• Est. Priority Date: 07/24/2014
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Tian, Chao
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US15/430,792
• Publication Number:
  US 20170153956A1
• Time in Patent Office: 435 Days
• Field of Search: 714 624, 714 61, 714 62, 714 15, 714 42, 714 49, 714751, 714752, 714784
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/154 Error and erasure correction, e.g. by using the error and erasure locator or Forney polynomial
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/63 Joint error correction and other techniques H03M13/31 and H03M13/33 take precedence
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Multi-reliability regenerating (MRR) erasure codes are disclosed. The erasure codes can be used to encode and regenerate data. In particular, the regenerating erasure codes can be used to encode data included in at least one of two or more data messages to satisfy respective reliability requirements for the data. Encoded portions of data from one data message can be mixed with encoded or unencoded portions of data from a second data message and stored at a distributed storage system. This approach can be used to improve efficiency and performance of data storage and recovery in the event of failures of one or more nodes of a distributed storage system.

Claims

1. A system, comprising:

a processor; and

a memory that stores instructions that, when executed by the processor, cause the processor to perform operations comprising

obtaining, in a data message, a first plurality of information symbols and a second plurality of information symbols,

encoding the first plurality of information symbols using an erasure code to generate a plurality of coded symbols,

combining a portion of the plurality of coded symbols with a portion of the second plurality of information symbols to generate a plurality of combined data values, wherein one of the plurality of combined data values comprises a linear combination of one of the plurality of coded symbols and one of the second plurality of information symbols, and wherein the one of the second plurality of information symbols is unencoded, and

initiating storage of the plurality of combined data values by a plurality of nodes of a distributed storage system.

2. The system of claim 1, wherein the instructions, when executed by the processor, cause the processor to perform operations further comprising:

detecting a failure of one of the plurality of nodes;

retrieving a first combined data value of the plurality of combined data values from a first node of the plurality of nodes;

retrieving a second combined data value of the plurality of combined data values from a second node of the plurality of nodes;

determining, based on the first combined data value and the second combined data value, two of the plurality of coded symbols; and

recovering the first plurality of information symbols by decoding, using the erasure code, the two of the plurality of coded symbols.

3. The system of claim 1, wherein the instructions, when executed by the processor, cause the processor to perform operations further comprising:

initiating storage of a mapping of the plurality of combined data values to a plurality of node addresses that correspond to the plurality of nodes.

4. The system of claim 1, wherein initiating storage of the plurality of combined data values comprises sending a plurality of data messages that includes the plurality of combined data values.

5. The system of claim 4, wherein the plurality of data messages is directed to the plurality of nodes.

6. The system of claim 1, wherein the linear combination of one of the plurality of coded symbols and one of the second plurality of information symbols comprises a sum of the one of the plurality of coded symbols and the one of the second plurality of information symbols.

7. The system of claim 6, wherein another of the plurality of combined data values comprises the one of the second plurality of information symbols.

8. The system of claim 7, wherein the instructions, when executed by the processor, cause the processor to perform operations further comprising:

determining the one of the plurality of coded symbols based on the one of the plurality of combined data values and the other of the plurality of combined data values; and

decoding the one of the plurality of coded symbols using the erasure code.

9. A method, comprising:

obtaining, at a storage server that comprises processor and a memory and in a data message, a first plurality of information symbols and a second plurality of information symbols;

encoding, by the storage server and using an erasure code, the first plurality of information symbols to generate a plurality of coded symbols;

combining, by the storage server, a portion of the plurality of coded symbols with a portion of the second plurality of information symbols to generate a plurality of combined data values, wherein one of the plurality of combined data values comprises a linear combination of one of the plurality of coded symbols and one of the second plurality of information symbols, and wherein the one of the second plurality of information symbols is unencoded; and

initiating, by the storage server, storage of the plurality of combined data values by a plurality of nodes of a distributed storage system.

10. The method of claim 9, further comprising:

detecting a failure of one of the plurality of nodes;

retrieving a first combined data value of the plurality of combined data values from a first node of the plurality of nodes;

retrieving a second combined data value of the plurality of combined data values from a second node of the plurality of nodes;

determining, based on the first combined data value and the second combined data value, two of the plurality of coded symbols; and

recovering the first plurality of information symbols by decoding, using the erasure code, the two of the plurality of coded symbols.

11. The method of claim 9, further comprising:

storing a mapping of the plurality of combined data values to a plurality of node addresses that correspond to the plurality of nodes.

12. The method of claim 9, wherein storing the plurality of combined data values comprises transmitting a plurality of data messages that includes the plurality of combined data values.

13. The method of claim 12, wherein the plurality of data messages is directed to the plurality of nodes.

14. The method of claim 9, wherein the linear combination of one of the plurality of coded symbols and one of the second plurality of information symbols comprises a sum of the one of the plurality of coded symbols and the one of the second plurality of information symbols.

15. The method of claim 14, wherein another of the plurality of combined data values comprises the one of the second plurality of information symbols.

16. The method of claim 15, further comprising:

determining the one of the plurality of coded symbols based on the one of the plurality of combined data values and the other of the plurality of combined data values; and

decoding the one of the plurality of coded symbols using the erasure code.

17. A memory that stores instructions that, when executed by a processor, cause the processor to perform operations comprising:

obtaining, in a data message, a first plurality of information symbols and a second plurality of information symbols;

encoding the first plurality of information symbols using an erasure code to generate a plurality of coded symbols;

combining a portion of the plurality of coded symbols with a portion of the second plurality of information symbols to generate a plurality of combined data values, wherein one of the plurality of combined data values comprises a linear combination of one of the plurality of coded symbols and one of the second plurality of information symbols, and wherein the one of the second plurality of information symbols is unencoded; and

initiating storage of the plurality of combined data values by a plurality of nodes of a distributed storage system.

18. The memory of claim 17, wherein the instructions, when executed by the processor, cause the processor to perform operations further comprising:

detecting a failure of one of the plurality of nodes;

retrieving a first combined data value of the plurality of combined data values from a first node of the plurality of nodes;

retrieving a second combined data value of the plurality of combined data values from a second node of the plurality of nodes;

determining, based on the first combined data value and the second combined data value, two of the plurality of coded symbols; and

recovering the first plurality of information symbols by decoding, using the erasure code, the two of the plurality of coded symbols.

19. The memory of claim 17, wherein the linear combination of one of the plurality of coded symbols and one of the second plurality of information symbols comprises a sum of the one of the plurality of coded symbols and the one of the second plurality of information symbols.

20. The memory of claim 17, wherein storing the plurality of combined data values comprises sending a plurality of data messages that includes the plurality of combined data values, wherein the plurality of data messages are directed to the plurality of nodes.