Monitoring storage cluster elements

• US 10,437,510 B2
• Filed: 02/03/2015
• Issued: 10/08/2019
• Est. Priority Date: 02/03/2015
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Batra, Ankit, Jolad, Amarnath, Nishime, Kazunobu
• Primary Examiner(s): Bayard, Djenane M
• Application Number:
  US14/613,085
• Publication Number:
  US 20160224277A1
• Time in Patent Office: 1,708 Days
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 41/065 involving logical or physical relationship, e.g. grouping and hierarchies
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/0894 Policy-based network configuration management
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Monitoring health of associated, but separated storage clusters can be done at both a node scope and a cluster scope. Monitoring the storage clusters at the cluster scope includes monitoring the network elements that support the storage clusters and connect the storage clusters. Initially, a fabric monitor in each cluster discovers cluster topology. This cluster topology is communicated and maintained throughout the managing storage elements of the storage clusters. After the storage cluster topologies have been discovered, the fabric monitors of each cluster can periodically determine status of network elements of the storage clusters. This allows the storage clusters to maintain awareness of interconnect status, and react to changes in status. In addition, each managing storage element monitors its own health. This information is aggregated to determine when to trigger corrective actions, alerts, and/or storage features in accordance with rules defined at the managing storage elements.

Claims

1. A method comprising:

monitoring clusters using rules within a synchronized database to determine whether conditions of the rules are violated by operational information of devices within the clusters, wherein a policy specifies that a notification is to be generated when a combination of a first rule being violated a first threshold number of times and a second rule being violated a second threshold number of times occurs within a threshold timespan;

performing a database update to the database using the operational information to trigger execution of the rules to evaluate the operational information against the conditions;

tracking a number of times each rule is violated; and

generating and sending the notification over a network to a computing device based upon the first rule being violated the first threshold number of times and the second rule being violated the second threshold number of times within the threshold timespan.

2. The method of claim 1, comprising:

transmitting the notification as a computer interrupt to the computing device based upon the computing device having a registered interest in an operational aspect of the clusters.

3. The method of claim 1, wherein the first rule is violated when a number of redundant paths to storage is below a threshold subsystem.

4. The method of claim 1, comprising:

performing in-band communication with a first storage cluster upon completion of a cluster establishment and cluster relationship configuration phase, wherein the in-band communication occurs over a communication channel used for data exchange.

5. The method of claim 4, comprising:

transitioning between out-of-band communication and in-band communication with the first storage cluster according to defined intervals.

6. The method of claim 4, comprising:

alternating between out-of-band communication and in-band communication with the first storage cluster during a back-hoe event until the back-hoe event is resolved.

7. The method of claim 1, wherein the first rule is violated if a logical interface fails.

8. The method of claim 1, wherein a format of the notification is agreed upon between the computing device and a device performing the monitoring.

9. A non-transitory machine readable medium comprising instructions for performing a method, which when executed by a machine, causes the machine to:

monitor clusters using rules within a synchronized database to determine whether conditions of the rules are violated by operational information of devices within the clusters, wherein a policy specifies that a notification is to be generated when a combination of a first rule being violated a first threshold number of times and a second rule being violated a second threshold number of times occurs within a threshold timespan;

perform a database update to the database using the operational information to trigger execution of the rules to evaluate the operational information against the conditions;

track number of times each rule is violated; and

generate and send the notification over a network to a computing device based upon the first rule being violated the first threshold number of times and the second rule being violated the second threshold number of times within the threshold timespan.

10. The non-transitory machine readable medium of claim 9, wherein the first rule is violated if a switch is not configured for a designated virtual storage area network (VSAN).

11. The non-transitory machine readable medium of claim 9, wherein the instructions cause the machine to:

utilize an application programming interface (API) to access a computer subsystem for retrieving the operational information.

12. The non-transitory machine readable medium of claim 9, wherein the instructions cause the machine to:

perform in-band communication with a first storage cluster upon completion of a cluster establishment and cluster relationship configuration phase, wherein the in-band communication occurs over a communication channel used for data exchange.

13. The non-transitory machine readable medium of claim 12, wherein the instructions cause the machine to:

transition between out-of-band communication and in-band communication with the first storage cluster according to defined intervals.

14. The non-transitory machine readable medium of claim 12, wherein the instructions cause the machine to:

alternate between out-of-band communication and in-band communication with the first storage cluster during a back-hoe event until the back-hoe event is resolved.

15. The non-transitory machine readable medium of claim 9, wherein the first rule is violated based upon a number of managing storage elements being below a threshold.

16. The non-transitory machine readable medium of claim 9, wherein the first rule is violated based upon a number of switches being below a threshold.

17. A computing device comprising:

a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method; and

a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to;

monitor clusters using rules within a synchronized database to determine whether conditions of the rules are violated by operational information of devices within the clusters, wherein a policy specifies that a notification is to be generated when a combination of a first rule being violated a first threshold number of times and a second rule being violated a second threshold number of times occurs within a threshold timespan;

perform a database update to the database using the operational information to trigger execution of the rules to evaluate the operational information against the conditions;

track number of times each rule is violated; and

generate and send the notification over a network to a second computing device based upon the first rule being violated the first threshold number of times and the second rule being violated the second threshold number of times within the threshold timespan.

18. The computing device of claim 17, wherein the machine executable code causes the processor to:

transmit the notification as a computer interrupt to the second computing device based upon the second computing device having a registered interest in an operational aspect of the clusters.

19. The computing device of claim 17, wherein the machine executable code causes the processor to:

utilize an application programming interface (API) to access a computer subsystem for retrieving the operational information.

20. The computing device of claim 17, wherein the machine executable code causes the processor to:

perform in-band communication with a first storage cluster upon completion of a cluster establishment and cluster relationship configuration phase, wherein the in-band communication occurs over a communication channel used for data exchange.