MONITORING STORAGE CLUSTER ELEMENTS

• US 20160224277A1
• Filed: 02/03/2015
• Published: 08/04/2016
• Est. Priority Date: 02/03/2015
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Jolad, Amarnath, Batra, Ankit, Nishime, Kazunobu
• Granted Patent: US 10,437,510 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 41/065 involving logical or physical relationship, e.g. grouping and hierarchies
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/0894 Policy-based network configuration management
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Monitoring health of associated, but separated storage clusters can be done at both a node scope and a cluster scope. Monitoring the storage clusters at the cluster scope includes monitoring the network elements that support the storage clusters and connect the storage clusters. Initially, a fabric monitor in each cluster discovers cluster topology. This cluster topology is communicated and maintained throughout the managing storage elements of the storage clusters. After the storage cluster topologies have been discovered, the fabric monitors of each cluster can periodically determine status of network elements of the storage clusters. This allows the storage clusters to maintain awareness of interconnect status, and react to changes in status. In addition, each managing storage element monitors its own health. This information is aggregated to determine when to trigger corrective actions, alerts, and/or storage features in accordance with rules defined at the managing storage elements.

Claims

1. A method comprising:

determining topology of a backend of a first storage cluster and a backend of a second storage cluster associated with the first storage cluster;

periodically obtaining first operational information from network elements of the first storage cluster backend and from network elements of the second storage cluster backend;

after obtaining the first operational information from any one of the network elements,

updating storage cluster information that is communicated among the first storage cluster based, at least in part, on the operational information;

determining whether the operational information violates a first set of one or more rules;

determining whether a notification should be generated in response to a determination that the operational information violates the first set of rules;

generating the notification in response to determining that the notification should be generated;

obtaining second operational information from a local subsystem;

determining whether the second operational information in combination with the first operational information violates a second set of one or more rules; and

generating a notification if the combination of the first operational information and the second operational information violates the second set of rules.

2. The method of claim 1, wherein determining topology of the backend of the first storage cluster and the backend of the second storage cluster associated with the first storage cluster comprises collecting topological information from the second storage cluster out-of-band with respect to a communication channel between the first storage cluster and the second storage cluster.

3. The method of claim 2 further comprising determining topology of a front end of both the first storage cluster and the second storage cluster via in-band communications.

4. The method of claim 2, wherein collecting the topological information out-of-band comprises collecting the topological information in accordance with the simple network management protocol.

5. The method of claim 1 further comprising validating the determined topology against topology validation rules.

6. The method of claim 1, wherein periodically obtaining the first operational information comprises obtaining the first operational information via a first communication channel in a first time period and obtaining the first operational information via a second communication channel in a second time period.

7. The method of claim 1, wherein the operational information comprises at least one of configuration information and current operating information.

8. The method of claim 1 further comprising:

determining that the second operational information indicates a loss of connectivity between the first storage cluster and the second storage cluster; and

determining whether the first operational information also indicates a loss of connectivity between the first storage cluster and the second storage cluster in response to determining that the second operational information indicates the loss of connectivity.

9. One or more non-transitory machine-readable media having program code stored therein, the program code to:

determine topology of a backend of a first storage cluster and a backend of a second storage cluster associated with the first storage cluster;

periodically obtain operational information from network elements of the first storage cluster backend and from network elements of the second storage cluster backend;

in response to receipt of operational information from a network element,

update storage cluster information that is communicated among the first storage cluster based, at least in part, on the operational information from the network element;

determine whether the operational information violates a first set of one or more rules;

determine whether a notification should be generated in response to a determination that the operational information violates the first set of rules;

generate the notification in response to determining that the notification should be generated;

obtain operational information from a local subsystem;

determine whether the operational information from the local subsystem in combination with the periodically obtained operational information violates a second set of one or more rules; and

generate a notification if the combination of operational information violates the second set of rules.

10. The machine-readable media of claim 9, wherein the program code to determine topology of the backend of the first storage cluster and the backend of the second storage cluster associated with the first storage cluster comprises the program code to collect topological information from the second storage cluster out-of-band with respect to a communication channel between the first storage cluster and the second storage cluster.

11. The machine-readable media of claim 10, further comprising program code to determine topology of a front end of the first storage cluster and a front end of the second storage cluster via in-band communications.

12. The machine-readable media of claim 10, wherein the program code to collect the topological information out-of-band comprises program code to collect the topological information in accordance with the simple network management protocol.

13. The machine-readable media of claim 9, further comprising program code to validate the determined topology against topology validation rules.

14. The machine-readable media of claim 9, wherein the program code to periodically obtain the operational information from the network elements comprises program code to obtain the operational information from the network elements via a first communication channel in a first time period and to obtain the operational information from the network elements via a second communication channel in a second time period.

15. The machine-readable media of claim 9, wherein the operational information comprises at least one of configuration information and current operating information.

16. The machine-readable media of claim 9 further comprising program code to:

determine that the operational information from the local subsystem indicates a loss of connectivity between the first storage cluster and the second storage cluster; and

determine whether the operational information from the network elements also indicates a loss of connectivity between the first storage cluster and the second storage cluster in response to a determination that the operational information from the local subsystem indicates the loss of connectivity.

17. An apparatus comprising:

a processor; and

a machine-readable storage medium having program code stored therein, the program code executable by the processor to cause the apparatus to,

determine topology of a backend of a first storage cluster and a backend of a second storage cluster associated with the first storage cluster;

periodically obtain operational information from network elements of the first storage cluster backend and from network elements of the second storage cluster backend;

in response to receipt of operational information from a network element,

update storage cluster information that is communicated among the first storage cluster based, at least in part, on the operational information from the network element;

determine whether the operational information violates a first set of one or more rules;

determine whether a notification should be generated in response to a determination that the operational information violates the first set of rules;

generate the notification in response to determining that the notification should be generated;

obtain operational information from a local subsystem;

determine whether the operational information from the local subsystem in combination with the periodically obtained operational information violates a second set of one or more rules; and

generate a notification if the combination of operational information violates the second set of rules.

18. The apparatus of claim 17, wherein the program code that causes the apparatus to determine topology of the backend of the first storage cluster and the backend of the second storage cluster associated with the first storage cluster comprises the program code executable by the processor to cause the apparatus to collect topological information from the second storage cluster out-of-band with respect to a communication channel between the first storage cluster and the second storage cluster.

19. The apparatus of claim 18, wherein the program code that causes the apparatus to collect the topological information out-of-band comprises program code executable by the processor to cause the apparatus to collect the topological information in accordance with the simple network management protocol.

20. The apparatus of claim 17 further comprising program code executable by the processor to cause the apparatus to:

determine that the operational information from the local subsystem indicates a loss of connectivity between the first storage cluster and the second storage cluster; and

determine whether the operational information from the network elements also indicates a loss of connectivity between the first storage cluster and the second storage cluster in response to a determination that the operational information from the local subsystem indicates the loss of connectivity.