Modifying dispersed storage network event records

• US 9,135,098 B2
• Filed: 07/12/2012
• Issued: 09/15/2015
• Est. Priority Date: 07/27/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Dhuse, Greg, Vedpathak, Yogesh Ramesh
• Primary Examiner(s): MCCARTHY, CHRISTOPHER S
• Application Number:
  US13/548,033
• Publication Number:
  US 20130031402A1
• Time in Patent Office: 1,160 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3086 where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats
• CPC Class Codes: G06F 11/321 Display for diagnostics, e.g. diagnostic result display, self-test user interface
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3485 for I/O devices

Abstract

A method begins by a dispersed storage (DS) processing module identifying a performance anomaly within a dispersed storage network (DSN). The method continues with the DS processing module identifying a set of collections of records corresponding to the performance anomaly, wherein one of the set of collections of records includes an event record including information regarding an event, a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, and a plurality of records including information regarding a set of DS units processing the plurality of sub-event requests. The method continues with the DS processing module determining whether a reliable significance indication of the performance anomaly is determinable, and when the reliable significance indication of the performance anomaly is not determinable, modifying data collection criteria for one or more of the sets collections of records.

Claims

1. A method comprises:

identifying a performance anomaly within a dispersed storage network (DSN);

identifying a set of collections of records corresponding to the performance anomaly, wherein a collection of records of the set of collections of records includes;

an event record including information regarding an event, wherein the event is a user access operation or a system administrative operation initiated by a device affiliated with the DSN;

a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, wherein the event request is regarding the event; and

a plurality of records including information regarding a set of DS units processing the plurality of sub-event requests wherein the identifying the set of collections of records includes;

identifying events associated with the performance anomaly; and

for one of the events;

identifying the one of the events corresponding to a user access operation regarding the performance anomaly;

determining a parent event identifier for the first event; and

identifying one of the collection of records of the set of the collections of records based on the parent event identifier;

determining, based on at least some of the collection of records of the set of collections of records, whether the performance anomaly is indicative of an undesired performance degradation of the DSN, a de-minimis performance degradation of the DSN, or an undeterminable performance degradation of the DSN; and

when the performance anomaly is indicative of the undeterminable performance degradation of the DSN, modifying data collection criteria for one or more of the collection of records of the set of the collections of records such that a determination of the undesired performance degradation of the DSN or the de-minimis performance degradation of the DSN can be rendered regarding the performance anomaly or for subsequent performance anomalies.

2. The method of claim 1, wherein the performance anomaly comprises one or more of:

an encoded data slice error;

a DS module processing error;

a deviation from expected performance of a DS unit of the set of DS units;

a deviation from expected performance of the DS processing module;

a deviation from an expected ordering of performance of functions;

a delay in performance of an expected function; and

an error in performance of the expected function.

3. The method of claim 1, wherein the identifying the performance anomaly comprises at least one of:

identifying a deviation of performance of one or more of the set of DS units from an expected DS unit performance trend; and

identifying another deviation of the DS processing module from an expected DS processing module performance trend.

4. The method of claim 1, wherein the modifying data collection criteria for the one or more of the collection of records of the set of collections of records comprises at least one of:

adding a log record to one or more records of the collection of records of the set of collections of records, wherein a log record includes one or more of a reporting entity identifier, a state identifier associated with a state of processing, a timestamp corresponding to the state, a state descriptor corresponding to the state, and state parameters corresponding to the state;

adding a statistics record to the one or more records of the collection of records of the set of collections of records, wherein the statistics record includes one or more of a reporting entity identifier, a step identifier associated with a step of processing, a timestamp corresponding to the step, and one or more descriptors corresponding to the step; and

modifying persistence of the one or more records of the set of collections of records.

5. The method of claim 1, wherein the first record comprises:

identity of the DS processing module;

an event identifier (ID) associated with the processing of the event request;

a parent event ID associated with the event;

information regarding initiation of the processing of the event request; and

information regarding completion of the processing of the event request.

6. The method of claim 1, wherein one of the plurality of records comprises:

identity of one of the set of DS units;

an event identifier (ID) associated with the processing of a corresponding one of the plurality of sub-event requests by the one of the set of DS units;

a parent event ID associated with the event request;

information regarding initiation of the processing of the corresponding one of the plurality of sub-event requests; and

information regarding completion of the processing of the corresponding one of the plurality of sub-event requests.

7. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

identify a performance anomaly within a dispersed storage network (DSN);

a second module, when operable within the computing device, causes the computing device to;

identify a set of collections of records corresponding to the performance anomaly,

wherein one of the set of collections of records includes;

an event record including information regarding an event, wherein the event is a user access operation or a system administrative operation initiated by a device affiliated with the DSN;

a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests,

wherein the event request is regarding the event; and

a plurality of records including information regarding a set of DS units processing the plurality of sub-event requests wherein the second module functions to identify the set of collections of records by;

identifying events associated with the performance anomaly; and

for one of the events;

identifying the one of the events corresponding to a user access operation regarding the performance anomaly;

determining a parent event identifier for the first event; and

identifying one of the collection of records of the set of the collections of records based on the parent event identifier;

a third module, when operable within the computing device, causes the computing device to;

determine, based on at least some of the collection of records of the set of collections of records, whether the performance anomaly is indicative of an undesired performance degradation of the DSN, a de-minimis performance degradation of the DSN, or an undeterminable performance degradation of the DSN; and

a fourth module, when operable within the computing device, causes the computing device to;

when the performance anomaly is indicative of the undeterminable performance degradation of the DSN, modify data collection criteria for one or more of the collection of records of the set of the collections of records such that a determination of the undesired performance degradation of the DSN or the de-minimis performance degradation of the DSN can be rendered regarding the performance anomaly or for subsequent performance anomalies.

8. The DS module of claim 7, wherein the performance anomaly comprises one or more of:

an encoded data slice error;

a DS module processing error;

a deviation from expected performance of a DS unit of the set of DS units;

a deviation from expected performance of the DS processing module;

a deviation from an expected ordering of performance of functions;

a delay in performance of an expected function; and

an error in performance of the expected function.

9. The DS module of claim 7, wherein the first module functions to identify the performance anomaly by at least one of:

identifying a deviation of performance of one or more of the set of DS units from an expected DS unit performance trend; and

identifying another deviation of the DS processing module from an expected DS processing module performance trend.

10. The DS module of claim 7, wherein the fourth module functions to modify data collection criteria for the one or more of the collection of records of the set of collections of records by at least one of:

adding a log record to one or more records of the collection of records of the set of collections of records, wherein a log record includes one or more of a reporting entity identifier, a state identifier associated with a state of processing, a timestamp corresponding to the state, a state descriptor corresponding to the state, and state parameters corresponding to the state;

adding a statistics record to the one or more records of the collection of records of the set of collections of records, wherein the statistics record includes one or more of a reporting entity identifier, a step identifier associated with a step of processing, a timestamp corresponding to the step, and one or more descriptors corresponding to the step; and

modifying persistence of the one or more records of the set of collections of records.

11. The DS module of claim 7, wherein the first record comprises:

identity of the DS processing module;

an event identifier (ID) associated with the processing of the event request;

a parent event ID associated with the event;

information regarding initiation of the processing of the event request; and

information regarding completion of the processing of the event request.

12. The DS module of claim 7, wherein one of the plurality of records comprises:

identity of one of the set of DS units;

an event identifier (ID) associated with the processing of a corresponding one of the plurality of sub-event requests by the one of the set of DS units;

a parent event ID associated with the event request;

information regarding initiation of the processing of the corresponding one of the plurality of sub-event requests; and

information regarding completion of the processing of the corresponding one of the plurality of sub-event requests.