Selecting a storage error abatement alternative in a dispersed storage network

• US 10,534,661 B2
• Filed: 12/15/2017
• Issued: 01/14/2020
• Est. Priority Date: 03/31/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Resch, Jason K.
• Primary Examiner(s): Chu, Gabriel
• Application Number:
  US15/843,390
• Publication Number:
  US 20180107544A1
• Time in Patent Office: 760 Days
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method for execution by an integrity processing unit includes detecting a storage error associated with storage of at least one encoded data slice in at least one memory of an associated storage unit of a set of storage units. Error parameters associated with the storage error are determined and a plurality of recovery alternatives to abate the storage error are identified based on the error parameters. A plurality of cost-benefit factors that correspond the plurality of recovery alternatives are determined. One of the plurality of recovery alternatives is selected based on the plurality of cost-benefit factors, and execution of the one of the plurality of recovery alternatives is facilitated to abate the storage error.

Claims

1. A method for execution by an integrity processing unit that includes a processor, the method comprises:

detecting a storage error associated with storage of at least one encoded data slice in at least one memory of an associated storage unit of a set of storage units;

determining error parameters associated with the storage error;

identifying a plurality of recovery alternatives to abate the storage error based on the error parameters;

determining a plurality of cost-benefit factors that correspond to the plurality of recovery alternatives;

selecting one of the plurality of recovery alternatives based on the plurality of cost-benefit factors; and

facilitating execution of the one of the plurality of recovery alternatives to abate the storage error.

2. The method of claim 1, wherein a set of encoded data slices that includes the at least one encoded data slice is associated with a data segment, and wherein the data segment was dispersed storage error encoded to produce the set of encoded data slices for storage in the set of storage units.

3. The method of claim 1, wherein the at least one encoded data slice is one of:

missing or corrupted.

4. The method of claim 1, wherein detecting the storage error includes at least one of:

scanning for a block error, receiving a file system error message, or identifying a bin file associated with the at least one encoded data slice.

5. The method of claim 1, wherein identifying the plurality of recovery alternatives includes interpreting a recovery alternative list based on the error parameters.

6. The method of claim 1, wherein determining the each of the plurality of cost-benefit factors is based on maximum network bandwidth utilization and maximum required downtime.

7. The method of claim 1, wherein determining the each of the plurality of cost-benefit factors is based on estimated recovery effectiveness, estimated data loss, and an estimated resource utilization level.

8. The method of claim 1, wherein selecting the one of the plurality of recovery alternatives includes ranking the plurality of recovery alternatives based on the plurality of cost-benefit factors and selecting a most favorably ranked recovery alternative.

9. The method of claim 1, wherein facilitating execution of the one of the plurality of recovery alternatives includes:

executing the one of the plurality of recovery alternatives;

measuring an effectiveness of executing the one of the plurality of recovery alternatives; and

selecting a next most favorable recovery alternative for execution when the measured effectiveness indicates that a level of abatement of the storage error has not been reached.

10. The method of claim 9, wherein facilitating execution of the one of the plurality of recovery alternatives further includes:

measuring a resource utilization level of executing the one of the plurality of recovery alternatives; and

saving the measured effectiveness and the measured resource utilization level in a historical record;

wherein determining the plurality of cost-benefit factors is based on at least one previously measured effectiveness and at least one previously measured resource utilization level of at least one previous execution of at least one of the plurality of recovery alternatives, wherein the at least one previously measured effectiveness and the at least one previously measured resource utilization level are retrieved from the historical record.

11. A processing system of an integrity processing unit comprises:

at least one processor;

a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to;

detect a storage error associated with storage of at least one encoded data slice in at least one memory of an associated storage unit of a set of storage units;

determine error parameters associated with the storage error;

identify a plurality of recovery alternatives to abate the storage error based on the error parameters;

determine a plurality of cost-benefit factors that correspond to the plurality of recovery alternatives;

select one of the plurality of recovery alternatives based on the plurality of cost-benefit factors; and

facilitate execution of the one of the plurality of recovery alternatives to abate the storage error.

12. The processing system of claim 11, wherein a set of encoded data slices that includes the at least one encoded data slice is associated with a data segment, and wherein the data segment was dispersed storage error encoded to produce the set of encoded data slices for storage in the set of storage units.

13. The processing system of claim 11, wherein the at least one encoded data slice is one of:

missing or corrupted.

14. The processing system of claim 11, wherein detecting the storage error includes at least one of:

scanning for a block error, receiving a file system error message, or identifying a bin file associated with the at least one encoded data slice.

15. The processing system of claim 11, wherein determining the each of the plurality of cost-benefit factors is based on maximum network bandwidth utilization and maximum required downtime.

16. The processing system of claim 11, wherein determining the each of the plurality of cost-benefit factors is based on estimated recovery effectiveness, estimated data loss, and an estimated resource utilization level.

17. The processing system of claim 11, wherein selecting the one of the plurality of recovery alternatives includes ranking the plurality of recovery alternatives based on the plurality of cost-benefit factors and selecting a most favorably ranked recovery alternative.

18. The processing system of claim 11, wherein facilitating execution of the one of the plurality of recovery alternatives includes:

executing the one of the plurality of recovery alternatives;

measuring an effectiveness of executing the one of the plurality of recovery alternatives; and

selecting a next most favorable recovery alternative for execution when the measured effectiveness indicates that a level of abatement of the storage error has not been reached.

19. The processing system of claim 18, wherein facilitating execution of the one of the plurality of recovery alternatives further includes:

measuring a resource utilization level of executing the one of the plurality of recovery alternatives; and

saving the measured effectiveness and the measured resource utilization level in a historical record;

wherein determining the plurality of cost-benefit factors is based on at least one previously measured effectiveness and at least one previously measured resource utilization level of at least one previous execution of at least one of the plurality of recovery alternatives, wherein the at least one previously measured effectiveness and the at least one previously measured resource utilization level are retrieved from the historical record.

20. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to;

detect a storage error associated with storage of at least one encoded data slice in at least one memory of an associated storage unit of a set of storage units;

determine error parameters associated with the storage error;

identify a plurality of recovery alternatives to abate the storage error based on the error parameters;

determine a plurality of cost-benefit factors that correspond to the plurality of recovery alternatives;

select one of the plurality of recovery alternatives based on the plurality of cost-benefit factors; and

facilitate execution of the one of the plurality of recovery alternatives to abate the storage error.