Issue escalation by management unit

• US 10,248,505 B2
• Filed: 01/10/2017
• Issued: 04/02/2019
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Iljazi, Ilir
• Primary Examiner(s): Bonzo, Bryce P
• Assistant Examiner(s): Lin, Katherine
• Application Number:
  US15/402,378
• Publication Number:
  US 20170249205A1
• Time in Patent Office: 812 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/327 Alarm or error message display
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 13/4282 on a serial bus, e.g. I2C bus, SPI bus on daisy chain buses G06F13/4247
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0623 in relation to content
• CPC Class Codes: G06F 3/0629 : Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/064 : Management of blocks
• CPC Class Codes: G06F 3/0644 : Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0653 : Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 : Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 30/20 : Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06N 3/04 : Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/10 : Interfaces, programming languages or software development kits, e.g. for simulating neural networks
• CPC Class Codes: G06Q 10/063116 : Schedule adjustment for a person or group
• CPC Class Codes: G06Q 10/06316 : Sequencing of tasks or work
• CPC Class Codes: G06Q 10/20 : Administration of product repair or maintenance
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H03M 13/2909 : Product codes
• CPC Class Codes: H03M 13/3761 : using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/616 : Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 63/0428 : wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/061 : for key exchange, e.g. in peer-to-peer networks cryptographic mechanisms or cryptographic arrangements for key agreement H04L9/0838
• CPC Class Codes: H04L 63/101 : Access control lists [ACL]
• CPC Class Codes: H04L 9/0869 : involving random numbers or seeds
• CPC Class Codes: H04L 9/0894 : Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage
• CPC Class Codes: H04L 9/14 : using a plurality of keys or algorithms
• CPC Class Codes: H04L 9/3242 : involving keyed hash functions, e.g. message authentication codes [MACs], CBC-MAC or HMAC

Abstract

Monitoring the health status of a distributed storage network (DSN) can include obtaining first status information indicating a first operational status of at least one DSN memory at a first point in time, the first operational status indicating one or more first operational issues. Second status information indicating a second operational status of the at least one DSN memory at a later point in time is obtained. The first operational status is compared to the second operational status to identify outstanding operational issues, e.g. issues that are indicated in both the first status information and the second status information. Each outstanding operational issue is mapped to an impact category, and an escalation level is determined based, at least in part, on the impact category. Notifications for a particular issue can be sent based on the escalation level for that issue.

Claims

1. A method for use in a distributed storage network (DSN) including at least one DSN memory including a plurality of storage units storing encoded data slices, the method comprising:

monitoring a health status of the DSN using at least one processor and associated memory included in the DSN, wherein monitoring the health status includes;

obtaining first status information indicating a first operational status of the at least one DSN memory at a first point in time, the first operational status indicating one or more first operational issues;

obtaining second status information indicating a second operational status of the at least one DSN memory at a second point in time, the second point in time being later than the first point in time, the second operational status indicating one or more second operational issues;

comparing the first operational status to the second operational status to identify outstanding operational issues, wherein an outstanding operational issue is an operational issue indicated in both the first status information and the second status information;

mapping each of the outstanding operational issues to an impact category;

monitoring a health status of one or more management units included in the DSN;

determining a number of remaining management units capable of monitoring the health status of the DSN;

determining an escalation level for particular outstanding operational issues based, at least in part, on the impact category mapped to the particular outstanding operation issues;

in response to determining that there is at least one remaining management unit capable of monitoring the health status of the DSN, issuing one or more notifications for the particular outstanding operational issues from the at least one remaining management unit, the one or more notifications based, at least in part, on the escalation level determined for the particular outstanding operational issues; and

in response to determining that there are no remaining management units capable of monitoring the health status of the DSN, issuing one or more notifications for the particular outstanding operational issues from the at least one DSN memory.

2. The method of claim 1, wherein obtaining status information includes:

interpreting a received error message.

3. The method of claim 1, wherein comparing the first operational status to the second operational status to identify outstanding operational issues includes:

determining an age of an outstanding operational issue by calculating a difference between a time at which a first operational issue was initially discovered, and a current time.

4. The method of claim 1, wherein determining the escalation level is based on one or more of a template mapping impact category to escalation level, interpreting system registry information, or previous resolution information.

5. The method of claim 1, wherein monitoring a health status of the DSN is performed by a management unit function applied across multiple different storage units included in the DSN.

6. A distributed storage network (DSN) including:

at least one DSN memory, the DSN memory including a plurality of storage units storing encoded data slice;

at least one management unit implemented using at least one processor and associated memory, the management unit configured to monitor a health status of the DSN by;

obtaining first status information indicating a first operational status of the at least one DSN memory at a first point in time, the first operational status indicating one or more first operational issues;

obtaining second status information indicating a second operational status of the at least one DSN memory at a second point in time, the second point in time being later than the first point in time, the second operational status indicating one or more second operational issues;

comparing the first operational status to the second operational status to identify outstanding operational issues, wherein an outstanding operational issue is an operational issue indicated in both the first status information and the second status information;

mapping each of the outstanding operational issues to an impact category;

determining an escalation level for particular outstanding operational issues based, at least in part, on the impact category mapped to the particular outstanding operation issues;

issuing one or more notifications for the particular outstanding operational issues based, at least in part, on the escalation level determined for the particular outstanding operational issues;

the at least one DSN memory configured to

monitor a health status of one or more management units included in the DSN;

determine a number of remaining management units capable of monitoring the health status of the DSN; and

in response to determining that there are no remaining management units capable of monitoring the health status of the DSN, issue one or more notifications for the particular outstanding operational issues from the at least one DSN memory.

7. The distributed storage network (DSN) of claim 6, wherein obtaining status information includes:

interpreting a received error message.

8. The distributed storage network (DSN) of claim 6, wherein comparing the first operational status to the second operational status to identify outstanding operational issues includes:

determining an age of an outstanding operational issue by calculating a difference between a time at which a first operational issue was initially discovered, and a current time.

9. The distributed storage network (DSN) of claim 6, wherein determining the escalation level is based on one or more of a template mapping impact category to escalation level, an interpretation of system registry information, or historical resolution information.

10. The distributed storage network (DSN) of claim 6, wherein monitoring a health status of the DSN is performed by a management unit function applied across multiple different storage units included in the DSN.