Handling failing memory devices in a dispersed storage network

• US 10,372,562 B2
• Filed: 06/06/2018
• Issued: 08/06/2019
• Est. Priority Date: 04/30/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Iljazi, Ilir, Resch, Jason K.
• Primary Examiner(s): Alphonse, Fritz
• Application Number:
  US16/001,671
• Publication Number:
  US 20180293141A1
• Time in Patent Office: 426 Days
• Field of Search: 714763, 714766, 714768, 714770
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method for execution by a dispersed storage and task (DST) execution unit identifying a failing memory device based on memory device diagnostic data. A cloning task is executed by designating memory of a replacement memory device to store encoded slices stored in the failing memory device, where the cloning task is executed over a cloning duration time period. A write request is received via a network at a receiving time during the cloning duration time period that includes a new encoded slice, and the new encoded slice is assigned to a temporary memory device for storage based on an identifier of the new encoded. The new encoded slice is transferred from the temporary memory device to the replacement memory device in response to an elapsing of the cloning duration time period corresponding to completion of the execution of the cloning task.

Claims

1. A method for execution by a dispersed storage and task (DST) execution unit that includes a processor, the method comprises:

identifying a failing memory device based on memory device diagnostic data;

designate memory of a replacement memory device to store encoded slices stored in the failing memory device in response to initiating a cloning duration time period;

receiving a write request via a network at a receiving time during the cloning duration time period that includes a new encoded slice, wherein the new encoded slice is assigned to a temporary memory device based on an identifier of the new encoded slice;

storing the new encoded slice in the temporary memory device; and

transferring the new encoded slice from the temporary memory device to the replacement memory device in response to an elapsing of the cloning duration time period.

2. The method of claim 1, wherein the memory device diagnostic data includes at least one of:

a test result, an error message, or a request received via the network.

3. The method of claim 1, wherein the failing memory device is identified by detecting that a number of storage errors of the failing memory device exceeds a storage error threshold level.

4. The method of claim 1, further comprising transferring a stored encoded slice to the replacement memory device during the cloning duration time period.

5. The method of claim 4, wherein the failing memory device stores a plurality of encoded slices, further comprising:

identifying a salvageable slice set that is a proper subset of the plurality of encoded slices,

transferring only the salvageable slice set to the replacement memory device during the cloning duration time period.

6. The method of claim 1, further comprising:

generating a rebuilding set that includes a plurality of slice identifiers corresponding to encoded slices of the failing memory device;

generating at least one recovered encoded slice corresponding to at least one of the plurality of slice identifiers by executing a rebuilding task based a corresponding at least one slice identifier of the plurality of slice identifiers in response to the elapsing of the cloning duration time period; and

storing the at least one recovered encoded slice in the replacement memory device.

7. The method of claim 6, wherein the rebuilding set is generated by identifying a plurality of encoded slices of the failing memory device that are at least one of:

missing or corrupted, and wherein the plurality of slice identifiers of the rebuilding set correspond to the identified plurality of encoded slices.

8. The method of claim 1, further comprising:

generating temporary redirection data by generating updated location weight values of the plurality of memory devices in response to identifying the failing memory device;

wherein the new encoded slice is stored in the temporary memory device based on the updated location weight values.

9. The method of claim 8, wherein the updated location weight values replace original location weight values, and wherein the new encoded slice has a corresponding storage assignment in the failing memory device based on the original location weight values.

10. The method of claim 8, further comprising:

generating final redirection data by generating final location weight values of the plurality of memory devices in response to the elapsing of the cloning duration time period

receiving a second new encoded slice via the network at a second time after the elapsing of the cloning duration time period; and

storing the second new encoded slice in the replacement memory device based on the final redirection data.

11. The method of claim 1, further comprising:

selecting the temporary memory device based on a decentralized agreement protocol.

12. A processing system of a dispersed storage and task (DST) execution unit comprises:

at least one processor;

a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to;

identify a failing memory device based on memory device diagnostic data;

designate memory of a replacement memory device to store encoded slices stored in the failing memory device, in response to initiating a cloning duration time period;

receive a write request via a network at a receiving time during the cloning duration time period that includes a new encoded slice, wherein the new encoded slice is assigned to a temporary memory device based on an identifier of the new encoded slice;

store the new encoded slice in the temporary memory device; and

transfer the new encoded slice from the temporary memory device to the replacement memory device in response to an elapsing of the cloning duration time period.

13. The processing system of claim 12, wherein the memory device diagnostic data includes at least one of:

a test result, an error message, or a request received via the network.

14. The processing system of claim 12, wherein the failing memory device is identified by detecting that a number of storage errors of the failing memory device exceeds a storage error threshold level.

15. The processing system of claim 12, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to transfer a stored encoded slice to the replacement memory device.

16. The processing system of claim 15, wherein the failing memory device stores a plurality of encoded slices, wherein the operation instructions, when executed by the at least one processor, further cause the processing system to:

identify a salvageable slice set that is a proper subset of the plurality of encoded slices;

transfer only the salvageable slice set to the replacement memory device during the cloning duration time period.

17. The processing system of claim 12, wherein the operation instructions, when executed by the at least one processor, further cause the processing system to:

generate a rebuilding set that includes a plurality of slice identifiers corresponding to encoded slices of the failing memory device;

generate at least one recovered encoded slice corresponding to at least one of the plurality of slice identifiers by executing a rebuilding task based a corresponding at least one slice identifier of the plurality of slice identifiers in response to the elapsing of the cloning duration time period; and

store the at least one recovered encoded slice in the replacement memory device.

18. The processing system of claim 12, wherein the operation instructions, when executed by the at least one processor, further cause the processing system to:

generate temporary redirection data by generating updated location weight values of the plurality of memory devices in response to identifying the failing memory device;

wherein the new encoded slice is stored in the temporary memory device based on the updated location weight values.

19. The processing system of claim 18, wherein the operation instructions, when executed by the at least one processor, further cause the processing system to:

generate final redirection data by generating final location weight values of the plurality of memory devices in response to the elapsing of the cloning duration time period;

receive a second new encoded slice via the network at a second time after the elapsing of the cloning duration time period; and

store the second new encoded slice in the replacement memory device based on the final redirection data.

20. A computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to;

identify a failing memory device based on memory device diagnostic data;

designate memory of a replacement memory device to store encoded slices stored in the failing memory device in response to initiating a cloning duration time period;

receive a write request via a network at a receiving time during the cloning duration time period that includes a new encoded slice, wherein the new encoded slice is assigned to a temporary memory device based on an identifier of the new encoded slice;

store the new encoded slice in the temporary memory device; and

transfer the new encoded slice from the temporary memory device to the replacement memory device in response to an elapsing of the cloning duration time period.