Method and apparatus for storage integrity processing based on error types in a dispersed storage network

• US 8,489,915 B2
• Filed: 04/26/2010
• Issued: 07/16/2013
• Est. Priority Date: 07/30/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Dhuse, Greg, Baptist, Andrew, Mark, Zachary J., Resch, Jason K., Volvovski, Ilya
• Primary Examiner(s): Lottich, Joshua P
• Assistant Examiner(s): Arcos, Jeison C
• Application Number:
  US12/767,404
• Publication Number:
  US 20110029836A1
• Time in Patent Office: 1,177 Days
• Field of Search: 714/6.12, 714/112, 714/712, 714/6.1, 714/42, 714/752, 712/112, 711/153
• US Class Current: 714/6.22
• CPC Class Codes: G06F 11/08 Error detection or correction by redundancy in data representation, e.g. by using checking codes
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 2211/104 Metadata, i.e. metadata associated with RAID systems with parity
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A storage integrity system in a dispersed storage network scans an address range of data slices to identify errors in one of a plurality of encoded data slices, wherein the plurality of encoded data slices are generated from a data segment using an error encoding dispersal function. When the storage integrity system detects an error, it identifies one of the encoded data slices for rebuilding. The identified data slice is rebuilt in response to the type of error. For example, when the type of the error includes a temporary error, the storage integrity system waits a predetermined time period to determine whether the error still exists prior to rebuilding the identified data slice.

Claims

1. A method for execution by one or more processing modules, the method comprises:

selecting an encoded data slice with an error for rebuilding to produce an identified data slice, wherein the identified data is one of a plurality of encoded data slices generated from a data segment using an error encoding dispersal function;

determining a type of the error;

when a type of the error includes a temporary error, scanning the plurality of encoded data slices to determine a number E of the plurality of encoded data slices with an associated error;

when the number E of the plurality of encoded data slices with an associated error compares unfavorably to a threshold, rebuilding the encoded data slice; and

when the number E of the plurality of encoded data slices with an associated error compares favorably to the threshold, waiting a predetermined time period before rebuilding the identified encoded data slice.

2. The method of claim 1, further comprising:

when the type of the error includes a temporary error, waiting a predetermined time period;

after the predetermined time period, determining whether the error still exists; and

when the error still exists after the predetermined time period, rebuilding the identified data slice when the number E of the plurality of encoded data slices with an associated error compares unfavorably to a threshold.

3. The method of claim 1, wherein the threshold equals a number X of the plurality of encoded data slices less a number T of the plurality of encoded data slices needed to reconstruct the data segment based on the error coding dispersal function.

4. The method of claim 1, wherein the rebuilding the identified data slice comprises:

retrieving at least a number T of the plurality of encoded data slices, wherein the number T corresponds to a number of the plurality of encoded data slices needed to reconstruct the data segment based on the error coding dispersal function;

reconstructing the data segment from at least the number T of the plurality of encoded data slices to produce a reconstructed data segment; and

generating a second plurality of encoded data slices from the reconstructed data segment based on the error coding dispersal function, wherein the second plurality of encoded data slices includes a rebuilt encoded data slice corresponding to the identified data slice.

5. The method of claim 1 further comprising:

determining the type of the error to be at least one of a slice failure, a storage unit failure, and a site failure.

6. The method of claim 1, further comprising:

determining the type of the error to be a slice failure;

when the slice failure includes a data corruption error, rebuilding the identified data slice;

when the slice failure includes a version update error, rebuilding the identified data slice; and

when the slice failure includes a missing slice error, waiting a predetermined time period to determine whether the missing slice error is corrected during the predetermined time period without rebuilding.

7. The method of claim 1, further comprising:

determining the type of the error to be a storage unit failure, wherein the identified data slice is stored on a storage unit with the storage unit failure; and

when the storage unit failure is a temporary outage, waiting a predetermined time period to determine whether the storage unit failure is corrected during the predetermined time period without rebuilding.

8. The method of claim 1, further comprising:

determining the type of the error to be a site failure, wherein the identified data slice is stored on a storage unit at a site with the site failure; and

when the site failure is a temporary outage, waiting a predetermined time period to determine whether the site failure is corrected during the predetermined time period without rebuilding.

9. The method of claim 1, further comprising:

periodically scanning an address range of encoded data slices stored in one or more storage units, wherein the plurality of encoded data slices has addresses within the address range; and

determining whether one of the encoded data slices within the address range has an associated error.

10. The method of claim 1, further comprising:

scanning the plurality of encoded data slices, wherein the plurality of encoded data slices is stored in one or more storage units; and

determining whether one of the plurality of encoded data slices has an associated error.

11. A storage integrity system in a dispersed storage network, comprising:

at least one network interface to the dispersed storage network; and

at least one processing module that is operable to;

identify an encoded data slice with an error for rebuilding to produce an identified data slice, wherein the identified data is one of a plurality of encoded data slices generated from a data segment using an error encoding dispersal function;

determine a type of the error;

when the type of the error includes a temporary error, scan the plurality of encoded data slices to determine a number E of the plurality of encoded data slices with an associated error;

when the number E of the plurality of encoded data slices with an associated error compares unfavorably to a threshold, rebuild the encoded data slice; and

when the number E of the plurality of encoded data slices with an associated error compares favorably to the threshold, wait a predetermined time period before rebuilding the identified encoded data slice.

12. The system of claim 11 wherein the at least one processing module is further operable to:

when the type of the error includes a temporary error, wait a predetermined time period;

after the predetermined time period, determine whether the error still exists; and

when the error still exists after the predetermined time period and when the number E of the plurality of encoded data slices with an associated error compares unfavorably to a threshold, build rebuilding the identified data slice.

13. The system of claim 11, wherein the threshold equals a number X of the plurality of encoded date slices less a number T of the plurality of encoded data slices needed to reconstruct the data segment based on the error coding dispersal function.

14. The system of claim 12, wherein the at least one processing module is further operable to:

determine the type of the error to be a slice failure;

when the slice failure includes a data corruption error, rebuild the identified data slice;

when the slice failure includes a version update error, rebuild the identified data slice; and

when the slice failure includes a missing slice error, determine whether the missing slice error is corrected during a predetermined time period before rebuilding.

15. The system of claim 11, wherein the at least one processing module is further operable to:

determine the type of the detected error to be a storage unit failure, wherein the identified data slice is stored on a storage unit with the storage unit failure; and

when the storage unit failure is a temporary outage, determine whether the storage unit failure is corrected during a predetermined time period before rebuilding.

16. The system of claim 11, wherein the at least one control unit is further operable to:

determine the type of the detected error to be a site failure, wherein the identified data slice is stored on a storage unit at a site with the site failure; and

when the site failure is a temporary outage, determine whether the site failure is corrected during a predetermined time period before rebuilding.

17. The system of claim 11, further comprising:

at least one scanning agent operable to;

periodically scan an address range of encoded data slices stored in one or more storage units, wherein the plurality of encoded data slices has addresses within the address range; and

determine whether one of the encoded data slices within the address range has an associated error.

18. The system of claim 11, further comprising:

at least one scanning agent operable to;

scan the plurality of encoded data slices, wherein the plurality of encoded data slices is stored in one or more storage units; and

determine whether one of the plurality of encoded data slices has an associated error.