Detecting storage errors in a dispersed storage network

• US 9,292,212 B2
• Filed: 05/09/2013
• Issued: 03/22/2016
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Kazi, Asimuddin, Resch, Jason K.
• Primary Examiner(s): KO, CHAE M
• Application Number:
  US13/890,438
• Publication Number:
  US 20130346809A1
• Time in Patent Office: 1,048 Days
• Field of Search: 714/6.22
• US Class Current: 1/1
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/08 Error detection or correction by redundancy in data representation, e.g. by using checking codes
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1068 in sector programmable memories, e.g. flash disk G06F11/1072 takes precedence
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/109 Sector level checksum or ECC, i.e. sector or stripe level checksum or ECC in addition to the RAID parity calculation
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/09 Error detection only, e.g. using cyclic redundancy check [CRC] codes or single parity bit
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/611 Specific encoding aspects, e.g. encoding by means of decoding

Abstract

A method begins by a dispersed storage (DS) processing module detecting, in accordance with a rebuilding process, a storage error of an encoded data slice stored in a storage node of a dispersed storage network (DSN) and identifying the encoded data slice for rebuilding. The method continues with the DS processing module identifying one or more storage traits associated with the encoded data slice and identifying encoded data slices having at least one storage trait in common with the one or more storage traits of the encoded data slice to produce identified encoded data slices. The method continues with the DS processing module prioritizing storage error detection analysis of the identified encoded data slices over other encoded data slices stored in the DSN and when a storage error is detected for one of the identified encoded data slices, identifying the one of the identified encoded data slices for rebuilding.

Claims

1. A method for execution by a processing module of one or more computing devices, the method comprises:

detecting, in accordance with a rebuilding process, a storage error of an encoded data slice stored in a storage node of a dispersed storage network (DSN);

identifying the encoded data slice for rebuilding;

identifying one or more storage traits associated with the encoded data slice;

identifying encoded data slices having at least one storage trait in common with the one or more storage traits of the encoded data slice to produce identified encoded data slices;

prioritizing, within the rebuilding process, storage error detection analysis of the identified encoded data slices over other encoded data slices stored in the DSN; and

when a storage error is detected for one of the identified encoded data slices, identifying the one of the identified encoded data slices for rebuilding.

2. The method of claim 1, wherein the rebuilding process comprises:

scanning addresses of the DSN;

identifying the storage node in accordance with the scanning of the addresses of the DSN;

sending a rebuilding feedback request to the storage node;

receiving a rebuilding feedback response from the storage node, wherein the rebuilding feedback response includes information that identifies encoded data slices stored in the storage node;

testing validity of the encoded data slices identified by the storage node based on the rebuilding feedback response and comparative encoded data slice information; and

when the comparative encoded data slice information for the encoded data slice indicates a validity issue, identifying the encoded data slice as having the storage error.

3. The method of claim 1, wherein the prioritizing the storage error detection analysis of the identified encoded data slices comprises:

pausing scanning addresses of the DSN;

identifying addresses of the encoded data slices having at least one storage trait in common with the one or more storage traits of the encoded data slice; and

sending rebuilding feedback request to storage nodes of the DSN storing the encoded data slices based on the addresses of the encoded data slices.

4. The method of claim 1, wherein a storage trait of the storage traits comprises one or more of:

a write timeframe;

a writing entity identifier;

a data owner identifier;

a vault identifier;

previous rebuilding operations;

a common network connection;

a common network path; and

an address being in an address range.

5. The method of claim 1, wherein the identifying the encoded data slices having at the least one storage trait in common with the one or more storage traits of the encoded data slice comprises:

identifying first priority encoded data slices of the identified encoded data slices that have more than one storage trait in common with the one or more storage traits of the encoded data slice; and

identifying second priority encoded data slices of the identified encoded data slices that have one storage trait in common with the one or more storage traits of the encoded data slice, wherein the first priority encoded data slices have a higher priority than the second priority encoded data slices.

6. The method of claim 1 further comprises:

when the storage error is detected for the one of the identified encoded data slices, determining storage traits for the one of the identified encoded data slices;

generating updated storage traits based on the storage traits for the one of the identified encoded data slices and the one or more storage traits of the encoded data slice;

adjusting the identifying the encoded data slices based on the updated storage traits to produce updated identified encoded data slices; and

prioritizing, within the rebuilding process, storage error detection analysis of the updated identified encoded data slices.

7. The method of claim 1 further comprises:

facilitating rebuilding of the encoded data slice; and

facilitating rebuilding of the one of the identified encoded data slices.

8. A method for execution by a processing module of one or more computing devices, the method comprises:

determining common storage traits of encoded data slices stored in a dispersed storage network (DSN) to produce groups of common storage trait encoded data slices;

executing storage error detection analysis on a sampling of encoded data slices of one of the groups of common storage trait encoded data slices;

when the sampling of the encoded data slices passes the storage error detection analysis, executing the storage error detection analysis on another sampling of encoded data slices of another one of the groups of common storage trait encoded data slices; and

when an encoded data slice of the sampling of encoded data slices has a detected storage error, executing the storage error detection analysis on a substantial number of the encoded data slices of the one of the groups of common storage trait encoded data slices.

9. The method of claim 8, wherein a storage trait of the storage traits comprises one or more of:

a write timeframe;

a writing entity identifier;

a data owner identifier;

a vault identifier;

previous rebuilding operations;

a common network connection;

a common network path; and

an address being in an address range.

10. The method of claim 8, wherein the executing the storage error detection analysis on the sampling of encoded data slices comprises:

sending a rebuilding feedback request to one or more storage nodes of the DSN storing the sampling of the encoded data slices;

receiving rebuilding feedback responses from the one or more storage nodes, wherein one of the rebuilding feedback responses includes storage information regarding the encoded data slice;

testing validity of the sampling of the encoded data slices based on the one of the rebuilding feedback responses and comparative encoded data slice information; and

when the comparative encoded data slice information for the encoded data slice indicates a validity issue, identifying the encoded data slice as having the detected storage error.

11. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

detect, in accordance with a rebuilding process, a storage error of an encoded data slice stored in a storage node of a dispersed storage network (DSN); and

identify the encoded data slice for rebuilding;

a second module, when operable within the computing device, causes the computing device to;

identify one or more storage traits associated with the encoded data slice; and

identify encoded data slices having at least one storage trait in common with the one or more storage traits of the encoded data slice to produce identified encoded data slices; and

a third module, when operable within the computing device, causes the computing device to;

prioritize, within the rebuilding process, storage error detection analysis of the identified encoded data slices over other encoded data slices stored in the DSN; and

when a storage error is detected for one of the identified encoded data slices, identify the one of the identified encoded data slices for rebuilding.

12. The DS module of claim 11, wherein the rebuilding process comprises:

the first module further functions to;

scan addresses of the DSN;

identify the storage node in accordance with the scanning of the addresses of the DSN;

sending a rebuilding feedback request to the storage node;

receive a rebuilding feedback response from the storage node, wherein the rebuilding feedback response includes information that identifies encoded data slices stored in the storage node;

test validity of the encoded data slices identified by the storage node based on the rebuilding feedback response and comparative encoded data slice information; and

when the comparative encoded data slice information for the encoded data slice indicates a validity issue, identify the encoded data slice as having the storage error.

13. The DS module of claim 11, wherein the third module functions to prioritize the storage error detection analysis of the identified encoded data slices by:

pausing scanning addresses of the DSN;

identifying addresses of the encoded data slices having at least one storage trait in common with the one or more storage traits of the encoded data slice; and

sending rebuilding feedback request to storage nodes of the DSN storing the encoded data slices based on the addresses of the encoded data slices.

14. The DS module of claim 11, wherein a storage trait of the storage traits comprises one or more of:

a write timeframe;

a writing entity identifier;

a data owner identifier;

a vault identifier;

previous rebuilding operations;

a common network connection;

a common network path; and

an address being in an address range.

15. The DS module of claim 11, wherein the second module functions to identify the encoded data slices having at the least one storage trait in common with the one or more storage traits of the encoded data slice by:

identifying first priority encoded data slices of the identified encoded data slices that have more than one storage trait in common with the one or more storage traits of the encoded data slice; and

identifying second priority encoded data slices of the identified encoded data slices that have one storage trait in common with the one or more storage traits of the encoded data slice, wherein the first priority encoded data slices have a higher priority than the second priority encoded data slices.

16. The DS module of claim 11 further comprises:

the second module further functions to;

when the storage error is detected for the one of the identified encoded data slices, determine storage traits for the one of the identified encoded data slices;

generate updated storage traits based on the storage traits for the one of the identified encoded data slices and the one or more storage traits of the encoded data slice; and

adjust the identifying the encoded data slices based on the updated storage traits to produce updated identified encoded data slices; and

the third module further functions to;

prioritize, within the rebuilding process, storage error detection analysis of the updated identified encoded data slices.

17. The DS module of claim 11 further comprises:

the third module further functions to;

facilitate rebuilding of the encoded data slice; and

facilitate rebuilding of the one of the identified encoded data slices.

18. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

determine common storage traits of encoded data slices stored in a dispersed storage network (DSN) to produce groups of common storage trait encoded data slices; and

a second module, when operable within the computing device, causes the computing device to;

execute storage error detection analysis on a sampling of encoded data slices of one of the groups of common storage trait encoded data slices;

when the sampling of the encoded data slices passes the storage error detection analysis, execute the storage error detection analysis on another sampling of encoded data slices of another one of the groups of common storage trait encoded data slices; and

when an encoded data slice of the sampling of encoded data slices has a detected storage error, execute the storage error detection analysis on a substantial number of the encoded data slices of the one of the groups of common storage trait encoded data slices.

19. The DS module of claim 18, wherein a storage trait of the storage traits comprises one or more of:

a write timeframe;

a writing entity identifier;

a data owner identifier;

a vault identifier;

previous rebuilding operations;

a common network connection;

a common network path; and

an address being in an address range.

20. The DS module of claim 18, wherein the second module executes the storage error detection analysis on the sampling of encoded data slices by:

sending a rebuilding feedback request to one or more storage nodes of the DSN storing the sampling of the encoded data slices;

receiving rebuilding feedback responses from the one or more storage nodes, wherein one of the rebuilding feedback responses includes storage information regarding the encoded data slice;

testing validity of the sampling of the encoded data slices based on the one of the rebuilding feedback responses and comparative encoded data slice information; and

when the comparative encoded data slice information for the encoded data slice indicates a validity issue, identifying the encoded data slice as having the detected storage error.