Identifying an error cause within a dispersed storage network

• US 9,170,868 B2
• Filed: 07/12/2012
• Issued: 10/27/2015
• Est. Priority Date: 07/27/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Dhuse, Greg, Vedpathak, Yogesh Ramesh
• Primary Examiner(s): MASKULINSKI, MICHAEL C
• Application Number:
  US13/547,941
• Publication Number:
  US 20130031414A1
• Time in Patent Office: 1,202 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3086 where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats
• CPC Class Codes: G06F 11/321 Display for diagnostics, e.g. diagnostic result display, self-test user interface
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3485 for I/O devices

Abstract

A method begins by a dispersed storage (DS) processing module selecting two or more correlated error messages of a plurality of error messages. The method continues with the DS processing module identifying two or more collection of records corresponding to the selected two or more correlated error messages such that a collection of records includes an event record including information regarding an event, a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, and a plurality of records including information regarding a plurality of DS units processing the plurality of sub-event requests. The method continues with the DS processing module identifying a correlation cause of one or more errors corresponding to the two or more correlated error messages based on the two or more collections of records.

Claims

1. A method for determining a cause of an error within a dispersed storage network (DSN), the method comprises:

selecting two or more correlated error messages of a plurality of error messages;

identifying two or more collection of records corresponding to the selected two or more correlated error messages, wherein a collection of records of the two or more collection of records includes;

an event record including information regarding an event, wherein the event is a user access operation or a system administrative operation initiated by a device affiliated with the DSN;

a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, wherein the event request is regarding the event; and

a plurality of records including information regarding a plurality of DS units processing the plurality of sub-event requests; and

identifying a correlation cause of one or more errors corresponding to the two or more correlated error messages based on the two or more collections of records.

2. The method of claim 1, wherein the selecting the two or more correlated error messages comprises:

selecting a first one of the two or more correlated error messages based on an error selection scheme; and

one or more of;

identifying remaining ones of the two or more correlated error messages that includes an error type in common with the first one of the two or more error messages;

identifying remaining ones of the two or more correlated error messages that includes a reporting entity in common with the first one of the two or more correlated error messages;

identifying remaining ones of the two or more correlated error messages that includes a timestamp value correlating with a timestamp of the first one of the two or more correlated error messages; and

identifying remaining ones of the two or more correlated error messages that includes an operation type in common with the first one of the two or more correlated error messages; and

identifying remaining ones of the two or more correlated error messages that includes DSN addressing information in common with the first one of the two or more correlated error messages.

3. The method of claim 1, wherein the identifying the one of the two or more collection of records comprises:

identifying a first event corresponding to a first one of the two or more correlated error messages;

determining a parent event identifier for the first event; and

identifying a first one of the two or more collection of records based on the parent event identifier.

4. The method of claim 1, wherein the event record comprises:

identity of a device that initiated the event;

an event identifier (ID) associated with the event;

information regarding initiation of the event; and

information regarding completion of the event.

5. The method of claim 1, wherein the first record comprises:

identity of the DS processing module;

an event identifier (ID) associated with the processing of the event request;

a parent event ID associated with the event;

information regarding initiation of the processing of the event request; and

information regarding completion of the processing of the event request.

6. The method of claim 1, wherein one of the plurality of records comprises:

identity of one of the plurality of DS units;

an event identifier (ID) associated with the processing of a corresponding one of the plurality of sub-event requests by the one of the plurality of DS units;

a parent event ID associated with the event request;

information regarding initiation of the processing of the corresponding one of the plurality of sub-event requests; and

information regarding completion of the processing of the corresponding one of the plurality of sub-event requests.

7. The method of claim 1, wherein the identifying the correlation cause comprises at least one of:

identifying the correlation cause as one of the plurality of DS units failing to perform a function corresponding to the processing of one of the plurality of sub-event requests;

identifying the correlation cause as the DS processing unit failing to perform a function corresponding to the processing of the event request;

identifying the correlation cause as one of the plurality of DS units performing an additional function in excess of the processing of the one of the plurality of sub-event requests;

identifying the correlation cause as the DS processing module performing an additional function in excess of the processing of the event request;

identifying the correlation cause as one of the plurality of DS units or the DS processing unit performing a function of the respective processing out of order of an expected function processing sequence; and

identifying the correlation cause as one of the plurality of DS units failing to perform the function corresponding to the processing of one of the plurality of sub-event requests within an expected time frame.

8. The method of claim 1 further comprises:

generating a graphical representation of at least one of the two or more collection of records to illustrate the correlation cause.

9. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

select two or more correlated error messages of a plurality of error messages;

a second module, when operable within the computing device, causes the computing device to;

identify two or more collection of records corresponding to the selected two or more correlated error messages, wherein a collection of records of the two or more collection of records includes;

an event record including information regarding an event, wherein the event is a user access operation or a system administrative operation initiated by a device affiliated with a dispersed storage network (DSN);

a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, wherein the event request is regarding the event; and

a plurality of records including information regarding a plurality of DS units processing the plurality of sub-event requests; and

a third module, when operable within the computing device, causes the computing device to;

identify a correlation cause of one or more errors corresponding to the two or more correlated error messages based on the two or more collections of records.

10. The DS module of claim 9, wherein the first module functions to select the two or more correlated error messages by:

selecting a first one of the two or more correlated error messages based on an error selection scheme; and

one or more of;

identifying remaining ones of the two or more correlated error messages that includes an error type in common with the first one of the two or more error messages;

identifying remaining ones of the two or more correlated error messages that includes a reporting entity in common with the first one of the two or more correlated error messages;

identifying remaining ones of the two or more correlated error messages that includes a timestamp value correlating with a timestamp of the first one of the two or more correlated error messages; and

identifying remaining ones of the two or more correlated error messages that includes an operation type in common with the first one of the two or more correlated error messages; and

identifying remaining ones of the two or more correlated error messages that includes DSN addressing information in common with the first one of the two or more correlated error messages.

11. The DS module of claim 9, wherein the second module functions to identify the one of the two or more collection of records by:

identifying a first event corresponding to a first one of the two or more correlated error messages;

determining a parent event identifier for the first event; and

identifying a first one of the two or more collection of records based on the parent event identifier.

12. The DS module of claim 9, wherein the event record comprises:

identity of a device that initiated the event;

an event identifier (ID) associated with the event;

information regarding initiation of the event; and

information regarding completion of the event.

13. The DS module of claim 9, wherein the first record comprises:

identity of the DS processing module;

an event identifier (ID) associated with the processing of the event request;

a parent event ID associated with the event;

information regarding initiation of the processing of the event request; and

information regarding completion of the processing of the event request.

14. The DS module of claim 9, wherein one of the plurality of records comprises:

identity of one of the plurality of DS units;

an event identifier (ID) associated with the processing of a corresponding one of the plurality of sub-event requests by the one of the plurality of DS units;

a parent event ID associated with the event request;

information regarding initiation of the processing of the corresponding one of the plurality of sub-event requests; and

information regarding completion of the processing of the corresponding one of the plurality of sub-event requests.

15. The DS module of claim 9, wherein the third module functions to identify the correlation cause by at least one of:

identifying the correlation cause as one of the plurality of DS units failing to perform a function corresponding to the processing of one of the plurality of sub-event requests;

identifying the correlation cause as the DS processing module failing to perform a function corresponding to the processing of the event request;

identifying the correlation cause as one of the plurality of DS units performing an additional function in excess of the processing of the one of the plurality of sub-event requests;

identifying the correlation cause as the DS processing unit performing an additional function in excess of the processing of the event request;

identifying the correlation cause as one of the plurality of DS units or the DS processing unit performing a function of the respective processing out of order of an expected function processing sequence; and

identifying the correlation cause as one of the plurality of DS units failing to perform the function corresponding to the processing of one of the plurality of sub-event requests within an expected time frame.

16. The DS module of claim 9 further comprises:

a fourth module, when operable within the computing device, causes the computing device to;

generate a graphical representation of at least one of the two or more collection of records to illustrate the correlation cause.