Identifying a slice error in a dispersed storage network

• US 8,914,667 B2
• Filed: 07/12/2012
• Issued: 12/16/2014
• Est. Priority Date: 07/27/2011
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Dhuse, Greg, Vedpathak, Yogesh Ramesh
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US13/547,887
• Publication Number:
  US 20130031407A1
• Time in Patent Office: 887 Days
• Field of Search: 714/6.2, 714/20, 714/37
• US Class Current: 714/6.2
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3086 where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats
• CPC Class Codes: G06F 11/321 Display for diagnostics, e.g. diagnostic result display, self-test user interface
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3485 for I/O devices

Abstract

A method begins by a dispersed storage (DS) processing module identifying a set of collections of records corresponding to a data segment that is stored in a dispersed storage network (DSN) as a set of encoded data slices, wherein a collection of records includes an event record including information regarding an event, a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, and a plurality of records including information regarding a set of DS units processing the plurality of sub-event requests. The method continues with the DS processing module determining whether an error exists for one of the set of encoded data slices based on at least some of the set of collections of records and when the error exists, flagging the one of the set of encoded data slices for potential rebuilding.

Claims

1. A method for correcting a storage error within a dispersed storage network (DSN), the method comprises:

identifying a set of collections of records corresponding to a data segment that is stored in the DSN as a set of encoded data slices, wherein a collection of records of the set of collections of records includes;

an event record including information regarding an event, wherein the event is a user access operation or a system administrative operation initiated by a device affiliated with the DSN;

a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, wherein the event request is regarding the event; and

a plurality of records including information regarding a set of DS units processing the plurality of sub-event requests;

determining whether an error exists for one of the set of encoded data slices based on at least some of the set of collections of records; and

when the error exists, flagging the one of the set of encoded data slices for potential rebuilding.

2. The method of claim 1, wherein the identifying the set of collections of records further comprises:

selecting the data segment based on one or more of;

an error message regarding the one of the set of encoded data slices;

an error of another data segment of a data object in common with the data segment;

a data segment analysis list;

a random selection process; and

a request to analyze the data segment.

3. The method of claim 1, wherein the identifying the set of collections of records comprises:

identifying events associated with the data segment; and

for one of the events;

identifying the one of the events corresponding to a user access operation regarding the data segment;

determining a parent event identifier for the first event; and

identifying one of the set of the collections of records based on the parent event identifier.

4. The method of claim 1, wherein the determining whether the error exists comprises at least one of:

identifying a record of the set of collections of records in which one of the set of DS units failed to perform an expected function;

identifying another record of the set of collections of records in which the DS processing module failed to perform another expected function;

identifying a group of records of the set of collections of records from which an out-of-order processing of a set of expected functions is detected; and

identifying yet another record of the set of collections of records in which the one of the set of DS units failed to perform the expected function within an expected time frame.

5. The method of claim 1 further comprises:

determining that errors exist for two or more encoded data slices of the set of encoded data slices based on the at least some of the set of collections of records; and

determining whether to rebuild each of the two or more encoded data slices; and

when a determination is made to rebuild one of the two or more encoded data slices;

initiating rebuilding of the one of the two or more encoded data slices; and

unflagging remaining ones of the two or more of encoded data slices for potential rebuilding.

6. The method of claim 1, wherein the first record comprises:

identity of the DS processing module;

an event identifier (ID) associated with the processing of the event request;

a parent event ID associated with the event;

information regarding initiation of the processing of the event request; and

information regarding completion of the processing of the event request.

7. The method of claim 1, wherein one of the plurality of records comprises:

identity of one of the set of DS units;

an event identifier (ID) associated with the processing of a corresponding one of the plurality of sub-event requests by the one of the set of DS units;

a parent event ID associated with the event request;

information regarding initiation of the processing of the corresponding one of the plurality of sub-event requests; and

information regarding completion of the processing of the corresponding one of the plurality of sub-event requests.

8. The method of claim 1 further comprises:

generating a decoding coded matrix of coded values of at least a decode threshold number of encoded data slices of the set of encoded data slices in accordance with a dispersed storage error coding function;

generating a data matrix from the decoding coded matrix and a decoding matrix in accordance with the dispersed storage error coding function, wherein the data matrix represents a rebuilding of the data segment;

generating a rebuilt coded matrix from the data matrix and an encoding matrix; and

generating a rebuilt version of the one of the set of encoded data slices from the rebuilt coded matrix.

9. A non-transitory computer readable storage medium having accessible therefrom a set of instructions interpretable by a processing module, the set of instructions being configured to cause the processor to carry out operations to:

identify a set of collections of records corresponding to a data segment that is stored in a dispersed storage network (DSN) as a set of encoded data slices, wherein a collection of records of the set of collections of records includes;

an event record including information regarding an event, wherein the event is a user access operation or a system administrative operation initiated by a device affiliated with the DSN;

a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, wherein the event request is regarding the event; and

a plurality of records including information regarding a set of DS units processing the plurality of sub-event requests;

determine whether an error exists for one of the set of encoded data slices based on at least some of the set of collections of records; and

when the error exists, flag the one of the set of encoded data slices for potential rebuilding.

10. The storage medium of claim 9, wherein the set of instructions further causes the processing module to carry out operations to identify the set of collections of records by:

selecting the data segment based on one or more of;

an error message regarding the one of the set of encoded data slices;

an error of another data segment of a data object in common with the data segment;

a data segment analysis list;

a random selection process; and

a request to analyze the data segment.

11. The storage medium of claim 9, wherein the set of instructions further causes the processing module to identify the set of collections of records by:

identifying events associated with the data segment; and

for one of the events;

identifying the one of the events corresponding to a user access operation regarding the data segment;

determining a parent event identifier for the first event; and

identifying one of the set of the collections of records based on the parent event identifier.

12. The storage medium of claim 9, wherein the set of instructions further causes the processing module to determine whether the error exists by at least one of:

identifying a record of the set of collections of records in which one of the set of DS units failed to perform an expected function;

identifying another record of the set of collections of records in which the DS processing module failed to perform another expected function;

identifying a group of records of the set of collections of records from which an out-of-order processing of a set of expected functions is detected; and

identifying yet another record of the set of collections of records in which the one of the set of DS units failed to perform the expected function within an expected time frame.

13. The storage medium of claim 9, wherein the set of instructions further causes the processing module to:

determine that errors exist for two or more encoded data slices of the set of encoded data slices based on the at least some of the set of collections of records; and

determine whether to rebuild each of the two or more encoded data slices; and

when a determination is made to rebuild one of the two or more encoded data slices;

initiate rebuilding of the one of the two or more encoded data slices; and

unflag remaining ones of the two or more of encoded data slices for potential rebuilding.

14. The storage medium of claim 9, wherein the first record comprises:

identity of the DS processing module;

an event identifier (ID) associated with the processing of the event request;

a parent event ID associated with the event;

information regarding initiation of the processing of the event request; and

information regarding completion of the processing of the event request.

15. The storage medium of claim 9, wherein one of the plurality of records comprises:

identity of one of the set of DS units;

an event identifier (ID) associated with the processing of a corresponding one of the plurality of sub-event requests by the one of the set of DS units;

a parent event ID associated with the event request;

information regarding initiation of the processing of the corresponding one of the plurality of sub-event requests; and

information regarding completion of the processing of the corresponding one of the plurality of sub-event requests.

16. The storage medium of claim 9, wherein the set of instructions further causes the processing module to carry out operations to:

generate a decoding coded matrix of coded values of at least a decode threshold number of encoded data slices of the set of encoded data slices in accordance with a dispersed storage error coding function;

generate a data matrix from the decoding coded matrix and a decoding matrix in accordance with the dispersed storage error coding function, wherein the data matrix represents a rebuilding of the data segment;

generate a rebuilt coded matrix from the data matrix and an encoding matrix; and

generate a rebuilt version of the one of the set of encoded data slices from the rebuilt coded matrix.

17. A computing device for use within a dispersed storage network, comprising:

an interface;

memory; and

a processing module operably coupled to the memory and the interface, wherein the processing module is operable to;

identify a set of collections of records corresponding to a data segment that is stored in the DSN as a set of encoded data slices, wherein a collection of records of the set of collections of records includes;

an event record including information regarding an event, wherein the event is a user access operation or a system administrative operation initiated by a device affiliated with the DSN;

a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, wherein the event request is regarding the event; and

a plurality of records including information regarding a set of DS units processing the plurality of sub-event requests;

determine whether an error exists for one of the set of encoded data slices based on at least some of the set of collections of records; and

when the error exists, flag the one of the set of encoded data slices for potential rebuilding.

18. The computing device of claim 17, wherein the processing module is further operable to:

select the data segment based on one or more of;

an error message regarding the one of the set of encoded data slices;

an error of another data segment of a data object in common with the data segment;

a data segment analysis list;

a random selection process; and

a request to analyze the data segment.

19. The computing device of claim 17, wherein the processing module is further operable to:

identify events associated with the data segment; and

for one of the events;

identify the one of the events corresponding to a user access operation regarding the data segment;

determine a parent event identifier for the first event; and

identify one of the set of the collections of records based on the parent event identifier.

20. The computing device of claim 17, wherein the processing module is further operable to:

identify a record of the set of collections of records in which one of the set of DS units failed to perform an expected function;

identify another record of the set of collections of records in which the DS processing module failed to perform another expected function;

identify a group of records of the set of collections of records from which an out-of-order processing of a set of expected functions is detected; and

identify yet another record of the set of collections of records in which the one of the set of DS units failed to perform the expected function within an expected time frame.