System and method for graph based monitoring and management of distributed systems

• US 10,353,800 B2
• Filed: 10/18/2017
• Issued: 07/16/2019
• Est. Priority Date: 10/18/2017
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Chen, Mingye, Huang, Xinyuan, Dutta, Debojyoti
• Primary Examiner(s): Vo, Ted T.
• Application Number:
  US15/786,790
• Publication Number:
  US 20190114247A1
• Time in Patent Office: 636 Days
• Field of Search: 717101-105, 717131-133
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 11/3048 where the topology of the computing system or computing system component explicitly influences the monitoring activity, e.g. serial, hierarchical systems
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data

Abstract

Systems, methods, and computer-readable media are disclosed for graph based monitoring and management of network components of a distributed streaming system. In one aspect, a method includes generating, by a processor, a first metrics and a second metrics based on data collected on a system; generating, by the processor, a topology graph representing data flow within the system; generating, by the processor, at least one first metrics graph corresponding to the first metrics based in part on the topology graph; generating, by the processor, at least one second metrics graph corresponding to the second metrics based in part on the topology graph; identifying, by the processor, a malfunction within the system based on a change in at least one of the first metrics graph and the second metrics graph; and sending, by the processor, a feedback on the malfunction to an operational management component of the system.

Claims

1. A method comprising:

generating, by one or more processors, for each instance of a plurality of instances of a moving window, first metrics indicative of at least one of central processing unit (CPU) utilization, memory utilization, or disk utilization by each of a plurality of servers of a distributed streaming system and second metrics indicative of at least one of throughput or latency of each of the plurality of servers of the distributed streaming system;

generating, by the one or more processors, a topology graph including a plurality of vertices representing the plurality of servers and a plurality of edges representing data flow among the plurality of servers;

generating, by the one or more processors, at least one first metrics graph corresponding to the first metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window based in part on the topology graph, wherein each vertex of the first metrics graph represents one of the servers of the plurality of servers and each edge between each pair of vertices of the first metrics graph is indicative of the first metrics of a first server represented by a first vertex of the pair of vertices of the first metrics graph being within a predetermined threshold of the first metrics of a second server represented by a second vertex of the pair of vertices of the first metrics graph;

generating, by the one or more processors, at least one second metrics graph corresponding to the second metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window based in part on the topology graph wherein each vertex of the second metrics graph represents one of the servers of the plurality of servers and each edge between each pair of vertices of the second metrics graph is indicative of the second metrics of a first server represented by a first vertex of the pair of vertices of the second metrics graph being within a predetermined threshold of the second metrics of a second server represented by a second vertex of the pair of vertices of the second metrics graph;

identifying, by the one or more processors, one or more differences between at least one of the first metrics graph at a first instance of the plurality of instances of the moving window and the first metrics graph at a second instance of the plurality of instances of the moving window or the second metrics graph at the first instance and the second metrics graph at the second instance; and

displaying, by the one or more processors, the one or more differences as indicative of a malfunction of the distributed streaming system.

2. The method of claim 1, wherein generating at least one first metrics graph corresponding to the first metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window includes:

determining a correlation matrix for each pair of vertices in the topology graph using the first metrics of a first server represented by a first vertex of the pair of vertices of the topology graph and the first metrics of a second server represented by a second vertex of the pair of vertices of the topology graph;

performing a Laplacian transformation on the correlation matrix to yield a transformed matrix;

generating a system metrics graph by performing a spectral clustering on the transformed matrix; and

generating the first metrics graph by trimming each edge in the system metrics graph not having a corresponding edge in the topology graph.

3. The method of claim 1, wherein generating at least one second metrics graph corresponding to the second metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window includes:

determining a correlation between for each pair of vertices in the topology graph using the second metrics of a first server represented by a first vertex of the pair of vertices of the topology graph and the second metrics of a second server represented by a second vertex of the pair of vertices of the topology graph;

performing a Laplacian transformation on the correlation matrix to yield a transformed matrix;

generating a performance metrics graph by performing a spectral clustering on the transformed matrix; and

generating the second metrics graph by trimming each edge in the performance metrics graph not having a corresponding edge in the topology graph.

4. The method of claim 1, wherein the topology graph is a Direct Acyclic Graph (DAG).

5. The method of claim 1, further comprising:

comparing the one or more differences between at least one of the first metrics graph at the first instance and the first metrics graph at the second instance or the second metrics graph at the first instance and the second metrics graph at the second instance to a historical pattern of differences between metrics graphs to identify a previous action for fixing the malfunction.

6. The method of claim 1, further comprising:

presenting a recommendation on fixing the malfunction.

7. A system comprising:

a memory having computer-readable instructions stored therein; and

one or more processors configured to execute the computer-readable instructions to perform operations including;

generating, for each instance of a plurality of instances of a moving window, system metrics indicative of at least one of central processing unit (CPU) utilization, memory utilization, or disk utilization by each of a plurality of servers of a distributed streaming system and performance metrics indicative of at least one of throughput or latency of each of the plurality of servers of the distributed streaming system;

generating a topology graph including a plurality of vertices representing the plurality of servers and a plurality of edges representing data flow among the plurality of servers;

generating at least one system metrics graph corresponding to the system metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window based in part on the topology graph, wherein each vertex of the system metrics graph represents one of the servers of the plurality of servers and each edge between each pair of vertices of the system metrics graph is indicative of the system metrics of a first server represented by a first vertex of the pair of vertices of the system metrics graph being within a predetermined threshold of the system metrics of a second server represented by a second vertex of the pair of vertices of the system metrics graph;

generating at least one performance metrics graph corresponding to the performance metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window based in part on the topology graph, wherein each vertex of the performance metrics graph represents one of the servers of the plurality of servers and each edge between each pair of vertices of the performance metrics graph is indicative of the performance metrics of a first server represented by a first vertex of the pair of vertices of the performance metrics graph being within a predetermined threshold of the performance metrics of a second server represented by a second vertex of the pair of vertices of the performance metrics graph;

identifying one or more differences between at least one of the system metrics graph at a first instance of the plurality of instances of the moving window and the system metrics graph at a second instance of the plurality of instances of the moving window or the performance metrics graph at the first instance and the performance metrics graph at the second instance; and

displaying the one or more differences as indicative of a malfunction of the distributed streaming system.

8. The system of claim 7, wherein generating at least one systems metrics graph corresponding to the system metrics for each server plurality of servers and for each instance of the plurality of instances of the moving window includes:

determining a correlation matrix for each pair of vertices in the topology graph using the system metrics of a first server represented by a first vertex of the pair of vertices of the topology graph and the systems metrics of a second server represented by a second vertex of the pair of vertices of the topology graph;

performing a Laplacian transformation on the correlation matrix to yield a transformed matrix;

generating an untrimmed system metrics graph by performing a spectral clustering on the transformed matrix; and

generating the system metrics graph by trimming each edge in the untrimmed system metrics graph not having a corresponding edge in the topology graph.

9. The system of claim 7, wherein generating at least one performance metrics graph corresponding to the performance metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window includes:

determine a correlation matrix for each pair of vertices in the topology graph using the performance metrics of a first server represented by a first vertex of the pair of vertices of the topology graph and the performance metrics of a second server represented by a second vertex of the pair of vertices of the topology graph;

performing a Laplacian transformation on the correlation matrix to yield a transformed matrix;

generating an untrimmed performance metrics graph by performing a spectral clustering on the transformed matrix; and

generating the performance metrics graph by trimming each edge in the untrimmed performance metrics graph not having a corresponding edge in the topology graph.

10. The system of claim 7, wherein the topology graph is a Direct Acyclic Graph (DAG).

11. The system of claim 7, wherein the one or more processors are further configured to execute the computer-readable instructions to perform additional operations including:

comparing the one or more differences between at least one of the system metrics graph at the first instance and the system metrics graph at the second instance or the performance metrics graph at the first instance and the performance metrics graph at the second instance to a historical pattern of differences between metrics graphs to identify a previous action for fixing the malfunction, within the system.

12. The system of claim 7, wherein the one or more processors are further configured to execute the computer-readable instructions to perform additional operations including:

presenting a recommendation on fixing the malfunction.

13. A non-transitory computer-readable medium having computer-readable instructions, which when executed by one or more processors, cause the one or more processors to perform operations comprising:

generating, for each instance of a plurality of instances of a moving window, system metrics indicative of at least one of central processing unit (CPU) utilization, memory utilization, or disk utilization by each of a plurality of servers of a distributed streaming system and performance metrics indicative of at least one of throughput or latency of each of the plurality of servers of the distributed streaming system;

generating a topology graph including a plurality of vertices representing the plurality of servers and a plurality of edges representing data flow among the plurality of servers;

generating at least one system metrics graph corresponding to the system metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window based in part on the topology graph, wherein each vertex of the system metrics graph represents one of the servers of the plurality of servers and each edge between each pair of vertices of the system metrics graph is indicative of the system metrics of a first server represented by a first vertex of the pair of vertices of the system metrics graph being within a predetermined threshold of the system metrics of a second server represented by a second vertex of the pair of vertices of the system metrics graph;

generating at least one performance metrics graph corresponding to the performance metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window based in part on the topology graph, wherein each vertex of the performance metrics graph represents one of the servers of the plurality of servers and each edge between each pair of vertices of the performance metrics graph is indicative of the performance metrics of a first server represented by a first vertex of the pair of vertices of the performance metrics graph being within a predetermined threshold of the performance metrics of a second server represented by a second vertex of the pair of vertices of the performance metrics graph;

identifying one or more differences between at least one of the system metrics graph at a first instance of the plurality of instances of the moving window and the system metrics graph at a second instance of the plurality of instances of the moving window or the performance metrics graph at the first instance and the performance metrics graph at the second instance; and

displaying the one or more differences as indicative of a malfunction of the distributed streaming system.

14. The non-transitory computer-readable medium of claim 13, wherein generating at least one systems metrics graph corresponding to the system metrics for each server plurality of servers and for each instance of a plurality of instances of the moving window includes:

determining a correlation matrix for each pair of vertices in the topology graph using the system metrics of a first server represented by a first vertex of the pair of vertices of the topology graph and the systems metrics of a second server represented by a second vertex of the pair of vertices of the topology graph;

perform a Laplacian transformation on the correlation matrix to yield a transformed matrix;

generating an untrimmed system metrics graph by performing a spectral clustering on the transformed matrix; and

generating the system metrics graph by trimming each edge in the untrimmed system metrics graph not having a corresponding edge in the topology graph.

15. The non-transitory computer-readable medium of claim 14, wherein generating at least one performance metrics graph corresponding to the performance metrics for each server of the plurality of servers and for each instance of the plurality of instances of the moving window includes:

determining a correlation matrix for each pair of vertices in the topology graph using the performance metrics of a first server represented by a first vertex of the pair of vertices of the topology graph and the performance metrics of a second server represented by a second vertex of the pair of vertices of the topology graph;

performing a Laplacian transformation on the correlation matrix to yield a transformed matrix;

generating an untrimmed performance metrics graph by performing a spectral clustering on the transformed matrix; and

generating the performance metrics graph by trimming each edge in the untrimmed performance metrics graph not having a corresponding edge the topology graph.

16. The non-transitory computer-readable medium of claim 13, wherein the topology graph is a Direct Acyclic Graph (DAG).

17. The non-transitory computer-readable medium of claim 13, wherein the one or more processors are further configured to execute the computer-readable instructions to perform additional operations including:

comparing the one or more differences between at least one of the system metrics graph at the first instance and the system metrics graph at the second instance or the performance metrics graph at the first instance and the performance metrics graph at the second instance to a historical pattern of differences between metrics graphs to identify a previous action for fixing the malfunction.

18. The non-transitory computer-readable medium of claim 13, wherein the one or more processors are further configured to execute the computer-readable instructions to perform additional operations including:

presenting a recommendation on fixing the malfunction.