Monitoring and resolving deadlocks, contention, runaway CPU and other virtual machine production issues

• US 8,904,240 B2
• Filed: 04/20/2012
• Issued: 12/02/2014
• Est. Priority Date: 04/20/2012
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bradley, Lisa M. W., Grigorenko, Kevin, Kelapure, Rohit D., Price, Dana L.
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US13/451,937
• Publication Number:
  US 20130283086A1
• Time in Patent Office: 956 Days
• Field of Search: 714/25, 714/26, 714/39, 714/45, 714/46, 714/47.1, 714/47.2, 714/47.3
• US Class Current: 714/47.1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0712 in a virtual computing platform, e.g. logically partitioned systems
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/301 where the computing system is a virtual computing platform, e.g. logically partitioned systems virtual machines G06F9/45533; logical partitioning of resources G06F9/5077
• CPC Class Codes: G06F 11/3089 Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents software debugging using additional hardware using a specific debug interface G06F11/3656; performance evaluation by tracing or monitoring G06F11/3466

Abstract

Resolving virtual machine (VM) issues, by executing VM and operating system (OS) diagnostic monitors, including, monitoring a set of VM and OS health status metrics of a system at a first level, analyzing data of the monitored health status metrics to determine that an instability has occurred when the data exceeds defined bounds for the health status metrics, responding to the instability by monitoring additional VM and OS health status metrics, whereby a level of monitoring of the system is increased from the first level to a second level, greater than the first level, identifying the instability, repairing the system by taking corrective action based on the identified instability; and removing at least one of the set of monitoring and profiling tools to reduce the level of monitoring to a third level once the instability has been resolved, wherein the third level is less than the second level.

Claims

1. A computer program product, comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising;

computer-readable program code configured to monitor a set of health status metrics of a virtual machine at a first level;

computer-readable program code configured to analyze data of the monitored health status metrics to determine that an instability has occurred when the data exceeds defined bounds for the computing system health status metrics;

computer-readable program code configured to respond to the instability by monitoring additional health status metrics, whereby a level of monitoring of the system is increased from the first level to a second level, greater than the first level;

computer-readable program code configured to repair the system by taking corrective action based on the instability; and

computer-readable program code configured to stop monitoring at least one of the set of monitored health status metrics to reduce the level of monitoring to a third level once the instability has been resolved, wherein the third level is less than the second level.

2. The computer program product of claim 1, wherein a user defines the sets of health status metrics to be monitored and the bounds used to determine that an instability has occurred.

3. The computer program product of claim 1, wherein the health status metrics monitored at the first level of monitoring comprise health status metrics of a central processing unit (CPU), a system memory, a system network activity, and a system input/output interface.

4. The computer program product of claim 1, wherein monitoring further comprises:

monitoring no health status metrics during a period of system stability.

5. The computer program product of claim 1, further comprising computer-readable program code configured to write to a system log the additional health status metrics monitored, the instability, the corrective action taken, and an associated effect on system stability for future use, wherein the system log is referenced to identify the health status metrics monitored and corrective actions taken to repair similar instabilities such that effective actions are repeated, and ineffective actions are not repeated.

6. The computer program product of claim 1, further comprising:

prior to repairing the system, receiving user authorization to take the corrective action.

7. The computer program product of claim 1, wherein the computing system is one of:

(i) a virtual machine, or (ii) an operating system.

8. The computer program product of claim 7, wherein the computing system is a virtual machine, wherein the computer-readable program code to analyze data of the monitored computing system health status metrics to determine that the instability has occurred further comprises:

upon determining that a resource has been exhausted;

supporting an early return from a method during execution of the method; and

notifying the virtual machine that the resource has been exhausted.

9. A system, comprising:

one or more physical computer processors; and

a memory containing a program, which when executed by the one or more physical computer processors is configured to perform an operation comprising;

monitoring a set of health status metrics of a system at a first level;

analyzing data of the monitored health status metrics to determine that an instability has occurred when the data exceeds defined bounds for the computing system health status metrics;

responding to the instability by monitoring additional health status metrics, whereby a level of monitoring of the system is increased from the first level to a second level, greater than the first level;

repairing the system by taking corrective action based on the instability; and

refraining from monitoring at least one of the set of monitored health status metrics to reduce the level of monitoring to a third level once the instability has been resolved, wherein the third level is less than the second level.

10. The system of claim 9, wherein a user defines the sets of health status metrics to be monitored and the bounds used to determine that an instability has occurred.

11. The system of claim 9, wherein the health status metrics monitored at the first level of monitoring comprise health status metrics of a central processing unit (CPU), a system memory, a system network activity, and a system input/output interface.

12. The system of claim 9, wherein monitoring further comprises:

monitoring no health status metrics during a period of virtual machine stability.

13. The system of claim 9, further comprising writing to a system log the additional health status metrics monitored, the instability, the corrective action taken, and an associated effect on virtual machine stability for future use, wherein the system log is referenced to identify the health status metrics monitored and corrective actions taken to repair similar instabilities such that effective actions are repeated, and ineffective actions are not repeated.

14. The system of claim 9, further comprising:

prior to repairing the virtual machine, receiving user authorization to take the corrective action.

15. The system of claim 9, wherein the computing system is one of:

(i) a virtual machine, or (ii) an operating system.

16. The system of claim 15, wherein the computing system is a virtual machine, wherein analyzing data of the monitored health status metrics to determine that the instability has occurred further comprises:

upon determining that a resource has been exhausted;

supporting an early return from a method during execution of the method; and

notifying the virtual machine that the resource has been exhausted.

17. The system of claim 9, wherein the computing system is a Java virtual machine.