Always-on monitoring in the cloud

• US 10,530,837 B2
• Filed: 04/10/2014
• Issued: 01/07/2020
• Est. Priority Date: 04/10/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bala, Vasanth, Isci, Canturk, Mummert, Todd W., Suneja, Sahil
• Primary Examiner(s): Madamba, Glenford J
• Application Number:
  US14/249,487
• Publication Number:
  US 20150295800A1
• Time in Patent Office: 2,098 Days
• Field of Search: 709224
• CPC Class Codes: G06F 2009/45591 Monitoring or debugging support
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 43/14 using software, i.e. software packages network security related monitoring H04L63/1408
• CPC Class Codes: H04L 43/20 the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/562 Brokering proxy services
• CPC Class Codes: H04L 67/564 Enhancement of application control based on intercepted application data

Abstract

A mechanism is provided in a data processing system for monitoring a virtual machine in a distributed computing system. An out-of-virtual machine monitor receives, via an out-of-band path, disk and memory state data of the virtual machine exposed in a hypervisor hosting the virtual machine. The out-of-virtual machine monitor interprets logical information from the virtual machine disk and memory state data. The out-of-virtual machine monitor parses the logical information to extract related system or log information for a monitoring endpoint. The out-of-virtual machine monitor translates the system or log information to a standard format expected by the monitoring endpoint. The out-of-virtual machine monitor forwards a monitoring data stream to the monitoring endpoint.

Claims

1. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to:

receive, by a monitoring broker executing on at least one processor of the data processing system, exposed disk and memory state data of the virtual machine as an out-of-band data stream via an out-of-band path comprising attach logic within a hypervisor executing on the at least one processor within the distributed computing system and hosting the virtual machine;

receive, by the monitoring broker, an in-band data stream from the virtual machine via an in-band path comprising a software agent executing within the virtual machine on the at least one processor within the distributed computing system;

normalize, by the monitoring broker, the in-band data stream and the out-of-band data stream to a common format;

determine, by the monitoring broker, the monitoring data stream, selected between the in-band data stream and the out-of-band data stream, to forward to a monitoring endpoint within the distributed computing system;

forward the monitoring data stream from the monitoring broker to the monitoring endpoint;

use the out-of-band data stream to monitor a footprint of the in-band path; and

responsive to determining a failure of the in-band path based on the footprint of the in-band path, select the out-of-band data stream to forward to the monitoring endpoint and alert the monitoring endpoint to revive the in-band path.

2. The computer program product of claim 1, wherein the computer readable program further causes the data processing system to:

responsive to determining the in-band data stream and the out-of-band data stream substantially deviate, alert a security solution of a potential compromise of the in-band path and quarantine the virtual machine.

3. The computer program product of claim 1, wherein determining the monitoring data stream comprises:

responsive to the in-band-data stream and the out-of-band data stream being substantially equivalent, selecting an earliest received data stream between the in-band data stream and the out-of-band data stream.

4. The computer program product of claim 1, wherein determining the monitoring data stream comprises:

responsive to a given data stream between the in-band data stream and the out-of-band data stream being more feature rich than the other, selecting the given data stream.

5. The computer program product of claim 1, wherein the computer readable program further causes the data processing system to interpret logical information from the exposed disk and memory state data, parse the logical information to extract related system or log information, and translate the system or log information to a standard format expected by the monitoring endpoint.

6. The computer program product of claim 5, wherein interpreting logical information from the exposed disk and memory state data comprises mounting a filesystem of the disk, inspecting blocks of the disk to discover file-level information, mapping well-known operating data structures on an address space of the memory, or using heuristics to discover data structures in the memory.

7. An apparatus, comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive, by a monitoring broker executing on at least one processor of a data processing system, exposed disk and memory state data of the virtual machine as an out-of-band data stream via an out-of-band path comprising attach logic within a hypervisor executing on the at least one processor within the distributed computing system and hosting the virtual machine;

receive, by the monitoring broker, an in-band data stream from the virtual machine via an in-band path comprising a software agent executing within the virtual machine on the at least one processor within the distributed computing system;

normalize, by the monitoring broker, the in-band data stream and the out-of-band data stream to a common format;

determine, by the monitoring broker, the monitoring data stream, selected between the in-band data stream and the out-of-band data stream, to forward to a monitoring endpoint within the distributed computing system a monitoring endpoint within the distributed computing system;

forward the monitoring data stream from the monitoring broker to the monitoring endpoint;

use the out-of-band data stream to monitor a footprint of the in-band path; and

responsive to determining a failure of the in-band path based on the footprint of the in-band path, select the out-of-band data stream to forward to the monitoring endpoint and alert the monitoring endpoint to revive the in-band path.

8. The apparatus of claim 7, wherein the instructions further cause the processor to:

responsive to determining the in-band data stream and the out-of-band data stream substantially deviate, alert a security solution of a potential compromise of the in-band path and quarantine the virtual machine.

9. The apparatus of claim 7, wherein determining the monitoring data stream comprises:

responsive to the in-band-data stream and the out-of-band data stream being substantially equivalent, selecting an earliest received data stream between the in-band data stream and the out-of-band data stream.

10. The apparatus of claim 7, wherein determining the monitoring data stream comprises:

responsive to a given data stream between the in-band data stream and the out-of-band data stream being more feature rich than the other, selecting the given data stream.

11. The apparatus of claim 7, wherein the instructions further cause the processor to interpret logical information from the exposed disk and memory state data, parse the logical information to extract related system or log information, and translate the system or log information to a standard format expected by the monitoring endpoint.

12. The apparatus of claim 11, wherein interpreting logical information from the exposed disk and memory state data comprises mounting a filesystem of the disk, inspecting blocks of the disk to discover file-level information, mapping well-known operating data structures on an address space of the memory, or using heuristics to discover data structures in the memory.

13. The apparatus of claim 7, wherein the disk state data are represented as a file on the hypervisor filesystem, a direct-mapped local or remote block device, or a virtual disk format.

14. The apparatus of claim 7, wherein the attach logic creates a local block device wrapper or network attached device wrapper around the disk.

15. The apparatus of claim 7, wherein the memory state data are exposed in a hypervisor hosting the virtual machine by dumping the memory, migrating the memory to file, snapshotting the memory, or remote direct memory access.

16. A method, in a data processing system, for monitoring a virtual machine (VM) in a distributed computing system, the method comprising:

receiving, by a monitoring broker executing on at least one processor of the data processing system, exposed disk and memory state data of the virtual machine as an out-of-band data stream via an out-of-band path comprising attach logic within a hypervisor executing on the at least one processor within the distributed computing system and hosting the virtual machine;

receiving, by the monitoring broker, an in-band data stream from the virtual machine via an in-band path comprising a software agent executing within the virtual machine on the at least one processor within the distributed computing system;

normalizing, by the monitoring broker, the in-band data stream and the out-of-band data stream to a common format;

determining, by the monitoring broker, the monitoring data stream, selected between the in-band data stream and the out-of-band data stream, to forward to a monitoring endpoint within the distributed computing system;

forwarding the monitoring data stream from the monitoring broker to the monitoring endpoint;

using the out-of-band data stream to monitor a footprint of the in-band path; and

responsive to determining a failure of the in-band path based on the footprint of the in-band path, selecting the out-of-band data stream to forward to the monitoring endpoint and alerting the monitoring endpoint to revive the in-band path.

17. The method of claim 16, wherein the disk state data are represented as a file on the hypervisor filesystem, a direct-mapped local or remote block device, or a virtual disk format.

18. The method of claim 16, wherein the attach logic creates a local block device wrapper or network attached device wrapper around the disk.

19. The method of claim 16, wherein the memory state data are exposed in a hypervisor hosting the virtual machine by dumping the memory, migrating the memory to file, snapshotting the memory, or remote direct memory access.

20. The method of claim 16, further comprising interpreting logical information from the exposed disk and memory state data, parsing the logical information to extract related system or log information, and translating the system or log information to a standard format expected by the monitoring endpoint.

21. The method of claim 20, wherein interpreting logical information from the exposed disk and memory state data comprises mounting a filesystem of the disk, inspecting blocks of the disk to discover file-level information, mapping well-known operating data structures on an address space of the memory, or using heuristics to discover data structures in the memory.

22. The method of claim 16, wherein determining the monitoring data stream comprises:

responsive to the in-band-data stream and the out-of-band data stream being substantially equivalent, selecting an earliest received data stream between the in-band data stream and the out-of-band data stream.

23. The method of claim 16, wherein determining the monitoring data stream comprises:

responsive to a given data stream between the in-band data stream and the out-of-band data stream being more feature rich than the other, selecting the given data stream.

24. The method of claim 16, further comprising:

responsive to determining the in-band data stream and the out-of-band data stream substantially deviate, alerting a security solution of a potential compromise of the in-band path and quarantining the virtual machine.