Large-scale distributed correlation

• US 10,291,463 B2
• Filed: 11/25/2015
• Issued: 05/14/2019
• Est. Priority Date: 10/07/2015
• Status: Active Grant

• Current Assignee: Riverbed Technology, LLC (Riverbed Technology Incorporated)
• Original Assignee: Riverbed Technology Incorporated
• Inventors: Sykes, Edward A.
• Primary Examiner(s): Nguyen, Thu Ha T
• Application Number:
  US14/952,313
• Publication Number:
  US 20170104658A1
• Time in Patent Office: 1,266 Days
• Field of Search: 709223, 709224
• CPC Class Codes: H04L 41/044 comprising hierarchical management structures
• CPC Class Codes: H04L 41/064 involving time analysis

Abstract

Disclosed herein are system, method, and computer program product embodiments for performing distributed correlation to determine a probable cause for a performance problem detected in an application. An embodiment operates by triggering an alert for a performance metric of an application executing on a local-level node. The alert may be sent to a higher-level node. Upon receiving the alert, the higher-level node may send a distributed correlation request, used to determine a root cause of the alert, to the lower-level node. Upon receiving the distributed correlation request, the lower-level node may produce and send a correlation result to the higher-level node. Upon receiving the correlation result, the higher-level node may select the probable cause of triggering the alert based on the correlation result. The probable cause may then be presented to the user.

Claims

1. A system, comprising:

one or more computing devices; and

a local-level node, implemented on the one or more computing devices, configured to;

trigger an alert for a performance metric of an application executing on the local-level node, wherein the alert indicates anomalous behavior for the performance metric, and

send the alert to a higher-level node, implemented on the one or more computing devices;

receive a distributed correlation request from the higher-level node, wherein the distributed correlation request is initiated to determine a root cause of the alert;

construct a correlation graph, the correlation graph including a root node representing the performance metric, a plurality of leaf nodes representing other performance metrics correlated with the performance metric, and a plurality of edges connecting the root node and the plurality of leaf nodes, each edge representing a dependent relationship between two performance metrics;

assign a correlation strength to each of the plurality of edges;

select one or more of the plurality of leaf nodes to be included in a correlation result based on the correlation strength assigned to each of the plurality of edges connected to the plurality of leaf nodes; and

send the correlation result to the higher-level node;

wherein the higher-level node is configured to;

select a probable cause of triggering the alert based on the performance metrics represented by the one or more leaf nodes included in the correlation result; and

present the probable cause to a user.

2. The system of claim 1, wherein to trigger the local-level node is further configured to:

detect that the performance metric contains metric values that are anomalous by calculating deviations of the metric values from a dynamic baseline.

3. The system of claim 1, wherein the local-level node is further configured to:

collect performance metric values for the performance metric across a period of time;

consolidate the performance metric values, collected across the period of time, into a bucketized histogram time slice; and

send the consolidated performance metric values to the higher-level node.

4. The system of claim 3, wherein the higher-level node has a parent higher-level node, and wherein the higher-level node is further configured to:

receive the distributed correlation request from the parent higher-level node, wherein the distributed correlation request includes the performance metric;

perform correlation between the performance metric and other metrics collected or received at the higher-level node, wherein the correlation is performed using the consolidated performance metric values; and

select the probable cause based on the performed correlation and the correlation results.

5. The system of claim 3, wherein the local-level node is further configured to:

collect second performance metric values for a second performance metric across the period of time;

consolidate the second performance metric values into a second bucketized histogram time slice; and

correlate the bucketized histogram time slice against the second bucketized histogram time slice in order to produce the correlation result.

6. The system of claim 1, wherein the local-level node is further configured to:

track application-dependency-mapping information for the application; and

send the application-dependency-mapping information to the higher-level node.

7. The system of claim 6, wherein the higher-level node is further configured to:

receive the application-dependency-mapping information from the local-level node; and

send the distributed correlation request to the lower-level node depending on whether the received application-dependency-mapping information indicates a relationship between the higher-level node and the local-level node.

8. The system of claim 6, wherein the local-level node is further configured to:

prune leaf nodes of the correlation graph depending on whether the application-dependency-mapping information conflicts with the plurality of edges connecting the root node and the plurality of leaf nodes.

9. A method, comprising:

triggering, by a local-level node that is implemented on one or more computing devices, an alert for a performance metric of an application executing on the local-level node, wherein the alert indicates anomalous behavior for the performance metric;

sending the alert to a higher-level node that is implemented on the one or more computing devices;

receiving a distributed correlation request from the higher-level node, wherein the distributed correlation request is initiated to determine a root cause of the alert;

constructing a correlation graph, the correlation graph including a root node representing the performance metric, a plurality of leaf nodes representing other performance metrics correlated with the performance metric, and a plurality of edges connecting the root node and the plurality of leaf nodes, each edge representing a dependent relationship between two performance metrics;

assigning a correlation strength to each of the plurality of edges;

selecting one or more of the plurality of leaf nodes to be included in a correlation result based on the correlation strength assigned to each of the plurality of edges connected to the plurality of leaf nodes; and

sending the correlation result to the higher-level node, wherein the higher-level node is configured to select a probable cause of triggering the alert based on the performance metrics represented by the one or more leaf nodes included in the correlation result, and present the probable cause to a user.

10. The method of claim 9, wherein the triggering further comprises:

detecting, at the local-level node, if the performance metric contains metric values that are anomalous by calculating deviations of the metric values from a dynamic baseline.

11. The method of claim 9, further comprising:

collecting, at the local-level node, performance metric values for the performance metric across a period of time;

consolidating the performance metric values, collected across the period of time, into

a bucketized histogram time slice;

and

sending the consolidated performance metric values to the higher-level node.

12. The method of claim 11, further comprising:

collecting, at the local-level node, second performance metric values for a second performance metric across the period of time;

consolidating the second performance metric values into a second bucketized histogram time slice; and

correlating the bucketized histogram time slice against the second bucketized histogram time slice in order to produce the correlation result.

13. The method of claim 9, wherein the higher-level node has a parent higher-level node, the method further comprising:

receiving by the higher level node the distributed correlation request from the parent higher-level node, wherein the distributed correlation request includes the performance metric;

performing by the higher level node correlation between the performance metric and other metrics collected or received at the higher-level node, wherein the correlation is performed using the consolidated performance metric values; and

selecting by the higher level node the probable cause based on the performed correlation and the correlation results.

14. The method of claim 9, further comprising:

tracking application-dependency-mapping information for the application; and

sending the application-dependency-mapping information to the higher-level node.

15. The method of claim 14, further comprising:

receiving, at the lower-level node, the distributed correlation request depending on whether the application dependency mapping information indicates a relationship between the higher-level node and the lower-level node.

16. The method of claim 14, further comprising:

pruning leaf nodes of the correlation graph depending on whether the received application-dependency-mapping information conflicts with the plurality of edges connecting the root node and the plurality of leaf nodes;

selecting a remaining leaf node, associated with a highly correlated metric, to be included in the correlation result.

17. A non-transitory computer readable storage medium having instructions stored thereon that, in response to execution by a computing device, cause the computing device to perform operations for performing distributed correlation in order to diagnose a root cause of anomalous behavior for a performance metric, the operations comprising:

triggering, by a local-level node that is implemented on one or more computing devices, an alert for a performance metric of an application executing on the local-level node, wherein the alert indicates anomalous behavior for the performance metric;

sending the alert to a higher-level node that is implemented on the one or more computing devices;

receiving a distributed correlation request from the higher-level node, wherein the distributed correlation request is initiated to determine a root cause of the alert;

constructing a correlation graph, the correlation graph including a root node representing the performance metric, a plurality of leaf nodes representing other performance metrics correlated with the performance metric, and a plurality of edges connecting the root node and the plurality of leaf nodes, each edge representing a dependent relationship between two performance metrics;

assigning a correlation strength to each of the plurality of edges;

selecting one or more of the plurality of leaf nodes to be included in a correlation result based on the correlation strength assigned to each of the plurality of edges connected to the plurality of leaf nodes; and

sending the correlation result to the higher-level node, wherein the higher-level node is configured to select a probable cause of triggering the alert based on the performance metrics represented by the one or more leaf nodes included in the correlation result, and present the probable cause to a user.