System and method for relationship based root cause recommendation

• US 10,318,366 B2
• Filed: 09/04/2015
• Issued: 06/11/2019
• Est. Priority Date: 09/04/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Qi, Yao Dong, Wang, Lan Jun, Wu, Hai Shan, Xu, Di Dx, Yang, Yi Bj
• Primary Examiner(s): Truong, Loan L. T.
• Application Number:
  US14/845,667
• Publication Number:
  US 20170068581A1
• Time in Patent Office: 1,376 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: H04L 41/00 Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 41/5096 wherein the managed service relates to distributed or central networked applications
• CPC Class Codes: H04L 43/0864 Round trip delays

Abstract

A method of identifying a root cause in a distributed computing environment includes traversing a plurality of nodes in a call graph starting with an end user node. Each node corresponds to an application component. A response time is calculated between connected pairs of neighboring nodes. A weight is calculated for each of a plurality of edges connecting the neighboring nodes. The nodes are traversed starting with the end user node in an order based on the weight of each of the edges. A root cause score is calculated for each node based on traversing all of the nodes in the order based on the weight of each of the edges. A ranked list is generated.

Claims

1. A method of identifying a root cause in a distributed computing environment, comprising:

generating a call graph including a plurality of nodes by merging topology relationship data, transaction tracking relationship data and metric correlation relationship data of the plurality of nodes, wherein the topology relationship data includes data regarding a physical distance between nodes of the plurality of nodes positioned in different geographic locations;

traversing the plurality of nodes in the call graph starting with an end user node, wherein each node corresponds to an application component in the distributed computing environment;

calculating a response time between pairs of neighboring nodes from among the plurality of nodes, wherein the neighboring nodes in each pair are connected to each other in the call graph;

calculating a weight for each of a plurality of edges connecting the neighboring nodes in the pairs based on the calculated response time between pairs of neighboring nodes among the plurality of nodes, wherein the weight of each edge is calculated based on a correlation between (i) the response time between the neighboring nodes in the corresponding pair and (ii) a response time between the neighboring node furthest from the end user node and the end user node;

traversing all of the nodes in the call graph starting with the end user node in an order based on the weight of each of the plurality of edges;

calculating a root cause score for each node in the call graph based on traversing all of the nodes in the call graph in the order based on the weight of each of the plurality of edges;

generating a ranked list comprising all of the nodes in an order based on the root cause score of each node; and

generating a recommendation to repair at least one node of all of the nodes in the ranked list, wherein the at least one node corresponds to an application component that acts as a system bottleneck in the distributed computing environment.

2. The method of claim 1, wherein a node having a highest root cause score is a first node in the ranked list and a node having a lowest root cause score is a last node in the ranked list.

3. The method of claim 1, further comprising:

generating the recommendation comprising the first node in the ranked list, wherein the first node is the root cause identified in the distributed computing environment.

4. The method of claim 1, wherein the application components are computer servers.

5. The method of claim 1, wherein the order in which all of the nodes in the call graph are traversed is a highest weight to lowest weight order.

6. The method of claim 1, wherein the order in which all of the nodes in the call graph are traversed is a lowest weight to highest weight order.

7. The method of claim 1, further comprising:

detecting similar abnormal patterns that occur while traversing all of the nodes in the call graph starting with the end user node in the order based on the weight of each of the plurality of edges.

15. The method of claim 1, wherein the nodes of the plurality of nodes positioned in different geographic locations represent servers located in different countries from each other.

8. A method of identifying a root cause in a distributed computing environment, comprising:

generating a call graph including a plurality of nodes by merging topology relationship data, transaction tracking relationship data and metric correlation relationship data of the plurality of nodes, wherein the topology relationship data includes data regarding a physical distance between nodes of the plurality of nodes positioned in different geographic locations;

traversing the plurality of nodes in the call graph starting with an end user node, wherein each node corresponds to an application component in the distributed computing environment;

calculating a throughput between pairs of neighboring nodes from among the plurality of nodes, wherein the neighboring nodes in each pair are connected to each other in the call graph;

calculating a weight for each of a plurality of edges connecting the neighboring nodes in the pairs based on the throughput between pairs of neighboring nodes from among the plurality of nodes, wherein the weight of each edge is calculated based on a correlation between (i) the response time between the neighboring nodes in the corresponding pair and (ii) a response time between the neighboring node furthest from the end user node and the end user node;

traversing all of the nodes in the call graph starting with the end user node in an order based on the weight of each of the plurality of edges;

calculating a root cause score for each node in the call graph based on traversing all of the nodes in the call graph in the order based on the weight of each of the plurality of edges;

generating a ranked list comprising all of the nodes in an order based on the root cause score of each node; and

generating a recommendation to repair at least one node of all of the nodes in the ranked list, wherein the at least one node corresponds to an application component that acts as a system bottleneck in the distributed computing environment.

9. The method of claim 8, wherein a node having a highest root cause score is a first node in the ranked list and a node having a lowest root cause score is a last node in the ranked list.

10. The method of claim 8, further comprising:

generating the recommendation comprising the first node in the ranked list, wherein the first node is the root cause identified in the distributed computing environment.

11. The method of claim 8, wherein the order in which all of the nodes in the call graph are traversed is a highest weight to lowest weight order.

12. The method of claim 8, wherein the order in which all of the nodes in the call graph are traversed is a lowest weight to highest weight order.

13. A method of identifying a root cause in a distributed computing environment, comprising:

generating a call graph including a plurality of nodes by merging topology relationship data, transaction tracking relationship data and metric correlation relationship data of the plurality of nodes, wherein the topology relationship data includes data regarding a physical distance between nodes of the plurality of nodes positioned in different geographic locations;

traversing the plurality of nodes in the call graph starting with an end user node, wherein each node corresponds to an application component in the distributed computing environment;

calculating a packet loss rate between pairs of neighboring nodes from among the plurality of nodes, wherein the neighboring nodes in each pair are connected to each other in the call graph;

calculating a weight for each of a plurality of edges connecting the neighboring nodes in the pairs based on the packet loss rate between pairs of neighboring nodes from among the plurality of nodes, wherein the weight of each edge is calculated based on a correlation between (i) the response time between the neighboring nodes in the corresponding pair and (ii) a response time between the neighboring node furthest from the end user node and the end user node;

traversing all of the nodes in the call graph starting with the end user node in an order based on the weight of each of the plurality of edges, wherein the order in which all of the nodes in the call graph are traversed is a highest weight to lowest weight order;

calculating a root cause score for each node in the call graph based on traversing all of the nodes in the call graph in the order based on the weight of each of the plurality of edges;

generating a ranked list comprising all of the nodes in an order based on the root cause score of each node, wherein a node having a highest root cause score is a first node in the ranked list and a node having a lowest root cause score is a last node in the ranked list; and

generating a recommendation to repair at least one node of all of the nodes in the ranked list, wherein the at least one node corresponds to an application component that acts as a system bottleneck in the distributed computing environment.

14. The method of claim 13, further comprising:

generating a recommendation comprising the first node in the ranked list, wherein the first node is the root cause identified in the distributed computing environment.