Storing data across groups of storage nodes

• US 8,856,619 B1
• Filed: 03/09/2012
• Issued: 10/07/2014
• Est. Priority Date: 03/09/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Cypher, Robert
• Primary Examiner(s): NGUYEN, STEVE N
• Application Number:
  US13/416,119
• Time in Patent Office: 942 Days
• Field of Search: 714/6.2, 714/770
• US Class Current: 714/770
• CPC Class Codes: G06F 11/1044 with specific ECC/EDC distribution
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for storing data reliably across groups of storage nodes. In one aspect, a method includes receiving (n−f) data chunks for storage across n groups of storage nodes and generating (f−1) error-correcting code chunks using an error-correcting code and the (n−f) data chunks. The (n−f) data chunks are stored at a first group of storage nodes. Each data chunk of the (n−f) data chunks is stored at a respective second group of storage nodes. Each code chunk of the (f−1) code chunks is stored at a respective third group of storage nodes. Each second group of storage nodes and each third group of storage nodes is distinct from each other and from the first group of storage nodes.

Claims

1. A method performed by one or more computers, the method comprising:

receiving (n−

f) data chunks for storage across n groups of storage nodes, wherein n and f are greater than or equal to two, and wherein (n−

f) is greater than or equal to one;

generating (f−

1) error-correcting code chunks using an error-correcting code and the (n−

f) data chunks;

storing the (n−

f) data chunks at a first group of storage nodes; and

storing each data chunk of the (n−

f) data chunks at a respective second group of storage nodes of a plurality of second groups of storage nodes and storing each code chunk of the (f−

1) code chunks at a respective third group of storage nodes of a plurality of third groups of storage nodes, wherein each second group of storage nodes and each third group of storage nodes is distinct from each other and from the first group of storage nodes;

wherein;

each of the n groups of storage nodes corresponds to one of n vertices in a complete block design of size n, the complete block design including

2. The method of claim 1, wherein storing the (n−

f) data chunks at the first group of storage nodes includes determining that the data chunks are more likely to be accessed at the first group of storage nodes than any of the other groups of storage nodes.

3. The method of claim 1, wherein the error-correcting code is a systematic maximal-distance separable (MDS) code.

4. The method of claim 1, further comprising:

receiving a request for the (n−

f) data chunks;

determining that the first group of storage nodes is not accessible; and

responding to the request using the group of storage nodes different from the first group of storage nodes.

5. The method of claim 4, wherein responding to the request includes reading the data chunks stored at the different groups of storage nodes storing the data chunks.

6. The method of claim 4, wherein responding to the request includes:

determining that at least one of the data chunks stored at the different groups of storage nodes is not accessible; and

reconstructing the at least one data chunk using one or more of the code chunks.

7. The method of claim 1, wherein:

each data chunk corresponds to a respective one of the blocks of vertices having the home vertex in a first instance of the block design; and

each data chunk corresponds to a respective one of the blocks of vertices having a vertex corresponding to the respective second group of storage nodes in a second instance of the block design, wherein each data chunk corresponds to a respective first vertex and a respective block in the first instance of the block design and a respective second vertex other than the first vertex in the respective block in the second instance of the block design.

8. A system of one or more computers configured to perform operations comprising:

receiving (n−

f) data chunks for storage across n groups of storage nodes, wherein n and f are greater than or equal to two, and wherein (n−

f) is greater than or equal to one;

generating (f−

1) error-correcting code chunks using an error-correcting code and the (n−

f) data chunks;

storing the (n−

f) data chunks at a first group of storage nodes; and

storing each data chunk of the (n−

f) data chunks at a respective second group of storage nodes of a plurality of second groups of storage nodes and storing each code chunk of the (f−

1) code chunks at a respective third group of storage nodes of a plurality of third groups of storage nodes, wherein each second group of storage nodes and each third group of storage nodes is distinct from each other and from the first group of storage nodes;

wherein;

each of the n groups of storage nodes corresponds to one of n vertices in a complete block design of size n, the complete block design including

9. The system of claim 8, wherein storing the (n−

f) data chunks at the first group of storage nodes includes determining that the data chunks are more likely to be accessed at the first group of storage nodes than any of the other groups of storage nodes.

10. The system of claim 8, wherein the error-correcting code is a systematic maximal-distance separable (MDS) code.

11. The system of claim 8, the operations further comprising:

receiving a request for the (n−

f) data chunks;

determining that the first group of storage nodes is not accessible; and

responding to the request using the group of storage nodes different from the first group of storage nodes.

12. The system of claim 11, wherein responding to the request includes reading the data chunks stored at the different groups of storage nodes storing the data chunks.

13. The system of claim 11, wherein responding to the request includes:

determining that at least one of the data chunks stored at the different groups of storage nodes is not accessible; and

reconstructing the at least one data chunk using one or more of the code chunks.

14. The system of claim 8, wherein:

each data chunk corresponds to a respective one of the blocks of vertices having the home vertex in a first instance of the block design; and

each data chunk corresponds to a respective one of the blocks of vertices having a vertex corresponding to the respective second group of storage nodes in a second instance of the block design, wherein each data chunk corresponds to a respective first vertex and a respective block in the first instance of the block design and a respective second vertex other than the first vertex in the respective block in the second instance of the block design.

15. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving (n−

f) data chunks for storage across n groups of storage nodes, wherein n and f are greater than or equal to two, and wherein (n−

f) is greater than or equal to one;

generating (f−

1) error-correcting code chunks using an error-correcting code and the (n−

f) data chunks;

storing the (n−

f) data chunks at a first group of storage nodes; and

storing each data chunk of the (n−

f) data chunks at a respective second group of storage nodes of a plurality of second groups of storage nodes and storing each code chunk of the (f−

1) code chunks at a respective third group of storage nodes of a plurality of third groups of storage nodes, wherein each second group of storage nodes and each third group of storage nodes is distinct from each other and from the first group of storage nodes;

wherein;

each of the n groups of storage nodes corresponds to one of n vertices in a complete block design of size n, the complete block design including

16. The computer storage medium of claim 15, wherein storing the (n−

f) data chunks at the first group of storage nodes includes determining that the data chunks are more likely to be accessed at the first group of storage nodes than any of the other groups of storage nodes.

17. The computer storage medium of claim 15, wherein the error-correcting code is a systematic maximal-distance separable (MDS) code.

18. The computer storage medium of claim 15, the operations further comprising:

receiving a request for the (n−

f) data chunks;

determining that the first group of storage nodes is not accessible; and

responding to the request using the group of storage nodes different from the first group of storage nodes.

19. The computer storage medium of claim 18, wherein responding to the request includes reading the data chunks stored at the different groups of storage nodes storing the data chunks.

20. The computer storage medium of claim 18, wherein responding to the request includes:

determining that at least one of the data chunks stored at the different groups of storage nodes is not accessible; and

reconstructing the at least one data chunk using one or more of the code chunks.

21. The computer storage medium of claim 15, wherein:

each data chunk corresponds to a respective one of the blocks of vertices having the home vertex in a first instance of the block design; and

each data chunk corresponds to a respective one of the blocks of vertices having a vertex corresponding to the respective second group of storage nodes in a second instance of the block design, wherein each data chunk corresponds to a respective first vertex and a respective block in the first instance of the block design and a respective second vertex other than the first vertex in the respective block in the second instance of the block design.