Cluster-wide virtual machine health monitoring

• US 10,133,619 B1
• Filed: 06/06/2016
• Issued: 11/20/2018
• Est. Priority Date: 06/08/2015
• Status: Active Grant

• Current Assignee: Nutanix Incorporated
• Original Assignee: Nutanix Incorporated
• Inventors: Nagpal, Abhinay, Kaufmann, Alexander J., Shukla, Himanshu, Sims, Jason, Arora, Varun Kumar, Kothuri, Venkata Vamsi Krishna
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US15/174,977
• Time in Patent Office: 897 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0712 in a virtual computing platform, e.g. logically partitioned systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 2009/45575 Starting, stopping, suspending or resuming virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

Systems for self-configuring health monitoring instrumentation for clustered storage platforms. Master and slave health modules implement a health monitoring system in a clustered virtualization environment comprising a plurality of nodes of the cluster with an installed health module instance running on the nodes. The health module system may gather and analyze data on a node level and at a cluster level to manage the cluster. The cluster health module system observes I/O commands issued to, and I/O command responses returned from, a common storage pool. Health data is stored in the storage pool.

Claims

1. A computer-implemented method for managing a clustered virtualization environment, comprising:

identifying a cluster of computing nodes that are interconnected by at least one communication path, wherein at least two of the computing nodes are further interconnected to a storage pool comprising at least one node-local storage device and at least one networked storage device;

invoking, on a first node, a first health module comprising at least one first node data collection unit that accesses the storage pool;

invoking, on a second node, a second health module comprising at least one second data collection unit that accesses the storage pool;

storing, in the storage pool, a first set of collected data generated by the first health module, the first set of collected data tracking a health status for the first node or the cluster of computing nodes by receiving observations taken at the first node; and

storing, in the storage pool, a second set of collected data generated by the second health module, the second set of collected data tracking a health status for the second node or the cluster of computing nodes by receiving observations taken at the second node.

2. The method of claim 1, further comprising performing an analysis, by the second node, of the first set of collected data to determine a health status of the first node.

3. The method of claim 1, further comprising detecting, by the second node, an interruption of service of the first node and invoking an instance of a new health module.

4. The method of claim 3, wherein the new health module accesses the storage pool and reads at least a portion of the first set of collected data gathered by the first health module.

5. The method of claim 1, further comprising issuing one or more alerts based on results of an analysis of at least a portion of the collected data.

6. The method of claim 1, further comprising logging an occurrence of an alert based at least in part on one or more observed events.

7. The method of claim 6, further comprising updating an alert database based at least in part on the one or more observed events.

8. The method of claim 6, further comprising initiating a corrective action based at least in part on the one or more observed events.

9. The method of claim 6, further comprising modifying a data collection schedule.

10. The method of claim 1, further comprising determining a version identifier of the first node health module and replacing the first node health module with a new instance of a health module having a newer version.

11. A computer readable medium, embodied in a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by a processor causes the processor to perform a set of acts for managing a clustered virtualization environment, the acts comprising:

identifying a cluster of computing nodes that are interconnected by at least one communication path, wherein at least two of the computing nodes are further interconnected to a storage pool comprising at least one node-local storage device and at least one networked storage device;

invoking, on a first node, a first health module comprising at least one first node data collection unit that accesses the storage pool;

invoking, on a second node, a second health module comprising at least one second data collection unit that accesses the storage pool;

storing, in the storage pool, a first set of collected data generated by the first health module, the first set of collected data tracking a health status for the first node or the cluster of computing nodes by receiving observations taken at the first node; and

storing, in the storage pool, a second set of collected data generated by the second health module, the second set of collected data tracking a health status for the second node or the cluster of computing nodes by receiving observations taken at the second node.

12. The computer readable medium of claim 11, further comprising instructions which, when stored in memory and executed by the processor causes the processor to perform acts of performing an analysis, by the second node, of the first set of collected data to determine a health status of the first node.

13. The computer readable medium of claim 11, further comprising instructions which, when stored in memory and executed by the processor causes the processor to perform acts of detecting, by the second node, an interruption of service of the first node and invoking an instance of a new health module.

14. The computer readable medium of claim 13, wherein the new health module accesses the storage pool and reads at least a portion of the first set of collected data gathered by the first health module.

15. The computer readable medium of claim 11, further comprising instructions which, when stored in memory and executed by the processor causes the processor to perform acts of issuing one or more alerts based on results of an analysis of at least a portion of the collected data.

16. The computer readable medium of claim 11, further comprising instructions which, when stored in memory and executed by the processor causes the processor to perform acts of logging an occurrence of an alert based at least in part on one or more observed events.

17. The computer readable medium of claim 16, further comprising instructions which, when stored in memory and executed by the processor causes the processor to perform acts of updating an alert database based at least in part on the one or more observed events.

18. The computer readable medium of claim 11, further comprising instructions which, when stored in memory and executed by the processor causes the processor to perform acts of determining a version identifier of the first node health module and replacing the first node health module with a new instance of a health module having a newer version.

19. A system for managing a clustered virtualization environment comprising:

a storage medium having stored thereon a sequence of instructions; and

a processor or processors that execute the instructions to cause the processor or processors to perform a set of acts, the acts comprising,

identifying a cluster of computing nodes that are interconnected by at least one communication path, wherein at least two of the computing nodes are further interconnected to a storage pool comprising at least one node-local storage device and at least one networked storage device;

invoking, on a first node, a first health module comprising at least one first node data collection unit that accesses the storage pool;

invoking, on a second node, a second health module comprising at least one second data collection unit that accesses the storage pool;

storing, in the storage pool, a first set of collected data generated by the first health module, the first set of collected data tracking a health status for the first node or the cluster of computing nodes by receiving observations taken at the first node; and

storing, in the storage pool, a second set of collected data generated by the second health module, the second set of collected data tracking a health status for the second node or the cluster of computing nodes by receiving observations taken at the second node.

20. The system of claim 19, wherein a new health module accesses the storage pool and reads at least a portion of the first set of collected data gathered by the first health module.