Disaster recovery for high-bandwidth distributed archives

• US 10,733,053 B1
• Filed: 01/31/2018
• Issued: 08/04/2020
• Est. Priority Date: 01/31/2018
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Miller, Ethan L., Karr, Ronald
• Primary Examiner(s): Kudirka, Joseph R
• Application Number:
  US15/885,670
• Time in Patent Office: 916 Days
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks

Abstract

A method for disaster recovery for distributed storage is provided. The method includes distributing erasure coded data to N storage systems at N sites so that the data is recoverable from N−1 storage systems at N−1 sites in case of failure of one of the N storage systems. The method includes coordinating movement of one or more storage devices of a replacement storage system to each of the N−1 sites and transferring data for rebuilding the erasure coded data, from each of the N−1 storage systems to the one or more storage devices of the replacement storage system, responsive to the failure of the one of the N storage systems. The method includes coordinating movement of the one or more storage devices of the replacement storage system to the one of the N sites or a further replacement site, to restore recoverability of the distributed archival storage.

Claims

1. A method, comprising:

distributing erasure coded data to N storage systems at N sites so that the data is recoverable from N−

1 storage systems at N−

1 sites in case of failure of one of the N storage systems at one of the N sites, for N finite and greater than two;

coordinating physical movement of one or more storage devices of a replacement storage system to each of the N−

1 sites;

transferring data for rebuilding the erasure coded data, from each of the N−

1 storage systems to the one or more storage devices of the replacement storage system, at the N−

1 sites, responsive to the failure of the one of the N storage systems;

coordinating physical movement of the one or more storage devices of the replacement storage system to the one of the N sites or a further replacement site, to restore recoverability of the distributed archival storage in case of failure of one of the N storage systems; and

constructing, in the one or more storage devices of the replacement storage system, at the N−

1 sites, an index of where the erasure coded data is stored, based on the transferred data for the rebuilding the erasure coded data.

2. The method of claim 1, wherein the coordinating physical movement to each of the N−

1 sites comprises coordinating physical movement of all of the one or more storage devices of the replacement storage system together to each of the N−

1 sites in succession.

3. The method of claim 1, wherein the coordinating physical movement to each of the N−

1 sites comprises;

coordinating physical movement of subsets of a plurality of storage devices of the replacement storage system to the N−

1 sites in parallel; and

coordinating physical movement of each of the subsets of the plurality of storage devices of the replacement storage system among the N−

1 sites in rotation.

4. The method of claim 1, further comprising:

generating a snapshot of the erasure coded data in the N−

1 storage systems, responsive to the failure of the one of the N storage systems;

writing further data to the N−

1 storage systems; and

redistributing imbalances in the further data, among the N−

1 storage systems and the one or more storage devices of the replacement storage system, based on the snapshot.

5. The method of claim 1, wherein the transferring the data for the rebuilding the erasure coded data comprises:

writing a subset of data from each of the N−

1 storage systems to a subset of the one or more storage devices of the replacement storage system at each of the N−

1 sites, as a plurality of subsets of the one or more storage devices of the replacement storage system is rotated around the N−

1 sites.

6. The method of claim 1, further comprising:

writing further data into the N−

1 storage systems, after the failure of the one of the N storage systems;

determining parity data for the further data;

distributing the parity data among the N−

1 storage systems and the one or more storage devices of the replacement storage system, during the transferring the data for rebuilding; and

redistributing the further data among the N−

1 storage systems and the one or more storage devices of the replacement storage system, during the transferring the data for rebuilding.

7. A method, comprising:

distributing erasure coded data to N storage systems at N sites so that the data is recoverable from N−

1 storage systems at N−

1 sites in case of failure of one of the N storage systems at one of the N sites, with N finite and greater than or equal to three;

coordinating physical movement of a replacement storage system to each of the N−

1 sites, in sequence, responsive to the failure of the one of the N storage systems;

transferring data to rebuild the erasure coded data, from each of the N−

1 storage systems to the replacement storage system, at each of the N−

1 sites in the sequence;

coordinating physical movement of the replacement storage system to the one of the N sites or a further replacement site, to restore recoverability of the distributed storage in case of failure of one of the N storage systems as comprising the N−

1 storage systems and the replacement storage system; and

constructing, in the replacement storage system or storage memory thereof, at the N−

1 sites in the sequence, an index of where the erasure coded data is stored, based on the transferred data.

8. The method of claim 7, further comprising:

writing further data to the N−

1 storage systems, after the failure of the one of the N storage systems; and

redistributing imbalances in the further data, among the N−

1 storage systems and the replacement storage system, based on a snapshot.

9. The method of claim 7, further comprising:

writing further data into the N−

1 storage systems, after the failure of the one of the N storage systems;

determining parity data for the further data; and

distributing the parity data and the further data among the N−

1 storage systems and the replacement storage system, during the transferring the data.

10. The method of claim 7, further comprising:

reconstructing data of the failed one of the N storage systems, in the replacement storage system at the one of the N sites or the further replacement site.

11. The method of claim 7, further comprising:

reconstructing data of the failed one of the N storage systems, into the replacement storage system, at each of the N−

1 sites in the sequence.

12. A method, comprising:

distributing erasure coded data to N storage systems at N sites so that the data is recoverable from N−

1 storage systems at N−

1 sites in case of failure of one of the N storage systems at one of the N sites, for N finite and greater than two;

coordinating physical movement of distinct subsets of a plurality of storage devices, of a replacement storage system among the N−

1 sites in a sequence, responsive to the failure of the one of the N storage systems;

transferring data to rebuild the erasure coded data, from each of the N−

1 storage systems to the distinct subsets of the plurality of storage devices of the replacement storage system, at the N−

1 sites in the sequence;

coordinating physical movement of the plurality of storage devices of the replacement storage system to the one of the N sites or a further replacement site, to restore recoverability of the distributed storage in case of failure of one of the N storage systems henceforth comprising the N−

1 storage systems and the replacement storage system; and

constructing, in the plurality of storage devices of the replacement storage system, at the N−

1 sites, an index of where the erasure coded data is stored, based on the transferred data.

13. The method of claim 12, further comprising:

making a snapshot of the erasure coded data in the N−

1 storage systems, responsive to the failure of the one of the N storage systems;

writing further data to the N−

1 storage systems; and

redistributing imbalances in the further data, among the N−

1 storage systems and the plurality of storage devices of the replacement storage system, at the N−

1 sites, based on the snapshot.

14. The method of claim 12, wherein the transferring the data comprises:

writing a differing subset of data from each of the N−

1 storage systems to each of the distinct subsets of the plurality of storage devices of the replacement storage system, at each of the N−

1 sites, each time the distinct subsets are physically moved around the N−

1 sites.

15. The method of claim 12, further comprising:

writing further data into the N−

1 storage systems, following the failure of the one of the N storage systems; and

distributing the further data and parity data for the further data among the N−

1 storage systems and the distinct subsets of the plurality of storage devices of the replacement storage system, during the transferring the data.

16. The method of claim 12, wherein the transferring the data to rebuild the erasure coded data comprises reconstructing data of the failed one of the N storage systems, at the N−

1 sites.

17. The method of claim 12, further comprising:

reconstructing data of the failed one of the N storage systems, based on the transferred data, at the one of the N sites or the further replacement site.