Determination of related failure events in a multi-node system

• US 7,409,604 B2
• Filed: 12/19/2003
• Issued: 08/05/2008
• Est. Priority Date: 12/19/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ganapathi, Archana, Murphy, Brendan
• Primary Examiner(s): Beausoliel, Jr.; Robert W
• Assistant Examiner(s): Ehne; Charles
• Application Number:
  US10/741,094
• Publication Number:
  US 20050188240A1
• Time in Patent Office: 1,691 Days
• Field of Search: 714/4, 714/25, 714/31, 714/39, 714/43, 714/45, 714/47, 714/48, 714/56
• US Class Current: 714/48
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: H04L 41/064 involving time analysis
• CPC Class Codes: H04L 43/091 Measuring contribution of individual network components to actual service level

Abstract

Systems and methods for determining related node failures in a multi-node system use log data obtained from the nodes. This log data is processed in various ways to indicate clusters of nodes that experience related failures.

Claims

1. A method comprising:

obtaining event data from each node in a group of associated nodes;

identifying at least one single-node event burst occurring on each of a plurality of the nodes in the group of nodes based on the obtained event data, including identifying at least one node event that does not have a corresponding opposite node event and creating an opposite node event associated with the at least one node event; and

identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node event bursts.

2. A method as recited in claim 1, wherein identifying a single-node event burst on a node comprises identifying outage events on a node occurring within a specified time frame.

3. A method as recited in claim 1, wherein identifying a single-node event burst on a node comprises identifying at least two outage events on a node that occur within a predetermined time from one another.

4. A method as recited in claim 1, wherein identifying a single-node event burst on a node comprises identifying a plurality of outage events on a node, each of the plurality of outage events occurring within a predetermined time of at least one other of the plurality of outage events.

5. A method as recited in claim 1, wherein identifying the multi-node event burst comprises identifying single-node event bursts occurring on at least two different nodes according to a predetermined timing rule.

6. A method as recited in claim 1, wherein identifying the multi-node event burst comprises identifying single-node event bursts occurring on at least two different nodes within a predetermined time from one another.

7. A method as recited in claim 1, wherein identifying the multi-node event burst comprises identifying a plurality of single-node event bursts, each of the plurality of single-node event bursts occurring within a predetermined time relative to at least one other of the plurality of outage events, wherein at least two of the single-node event bursts occur on different nodes.

8. A method as recited in claim 1, wherein obtaining event data from a node comprises obtaining data from an event log associated with the node.

9. A method as recited in claim 1, wherein obtaining event data from a node comprises obtaining information identifying up events and down events that have occurred on the node.

10. A method as recited in claim 1, wherein obtaining event data from a node comprises obtaining from an event log at the node information identifying up events and down events that have occurred on the node.

11. A method as recited in claim 1, further comprising storing the identified single-node event bursts in a single-node event bursts data structure and wherein the multi-node event burst is identified based on the single-node bursts stored in the single-node event bursts data structure.

12. A method comprising:

obtaining event data from each node in a group of associated nodes;

storing the obtained event data in an event table;

identifying at least one single-node event burst occurring on each of a plurality of the nodes in the group of nodes based on the obtained event data;

identifying at least one up event that does not have a corresponding down event in the data structure;

creating an associated down event for the up event; and

identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node event bursts.

13. A method comprising:

obtaining event data from each node in a group of associated nodes;

storing the obtained event data in an event table;

identifying at least one down event that does not have a corresponding up event in the data structure;

creating an associated up event for the down event; and

identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node event bursts.

14. A method comprising:

obtaining event data from each node in a group of associated nodes;

identifying at least one single-node event burst occurring on each of a plurality of the nodes in the group of nodes based on the obtained event data; and

identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node event bursts; and

determining a relationship strength value for two nodes having associated single-node event bursts occurring in the identified multi-node event burst.

15. A method comprising:

obtaining event data from each node in a group of associated nodes;

identifying at least one single-node event burst occurring on each of a plurality of the nodes in the group of nodes based on the obtained event data;

identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node event bursts;

determining relationship strength values for pairs of nodes based on the identified multi-node event burst; and

determining at least one cluster of related nodes based on the determined relationship strength values.

16. A method comprising:

obtaining event data from each node in a group of associated nodes;

identifying at least one single-node event burst occurring on each of a plurality of the nodes in the group of nodes based on the obtained event data;

identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node event bursts;

determining relationship strength values for pairs of nodes based on the identified multi-node event burst;

determining at least one cluster of related nodes based on the determined relationship strength values; and

determining at least one metric for the at least one cluster.

17. A method comprising:

obtaining event data from each node in a group of associated nodes;

identifying at least one single-node event burst occurring on each of a plurality of the nodes in the group of nodes based on the obtained event data;

identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node event bursts;

determining relationship strength values for pairs of nodes based on the identified multi-node event burst;

determining at least one cluster of related nodes based on the determined relationship strength values;

determining at least one metric for the at least one cluster; and

presenting the determined metric to a user.

18. A method comprising:

obtaining event data from each node in a group of associated nodes;

identifying at least one single-node event burst occurring on each of a plurality of the nodes in the group of nodes based on the obtained event data;

identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node event bursts;

determining relationship strength values for pairs of nodes based on the identified multi-node event burst;

determining at least one cluster of related nodes based on the determined relationship strength values;

determining a plurality of metrics for the at least one cluster; and

presenting the determined plurality of metrics to a user.

19. One or more processor-readable storage media having stored thereon processor executable instructions for performing acts comprising:

obtaining event data from each of a plurality of nodes in a node group;

identifying a multi-node event burst occurring in the node group based on the event data;

determining relationship strength values for pairs of nodes based on the identified multi-node event burst; and

determining at least one cluster of related nodes based on the determined relationship strength values.

20. One or more processor-readable storage media as recited in claim 19, further having stored thereon processor executable instructions for performing acts comprising:

determining at least one metric for the at least one cluster; and

presenting the determined metric to a user.

21. One or more processor-readable storage media as recited in claim 19, further having stored thereon processor executable instructions for performing acts comprising:

determining a plurality of metrics for the at least one cluster; and

presenting the determined plurality of metrics to a user.

22. One or more processor-readable storage media as recited in claim 19, wherein identifying a multi-node event burst comprises:

identifying a single-node event bursts on each of a plurality of nodes based on the obtained event data; and

identifying the multi-node event burst based on the identified single-node event bursts.

23. One or more processor-readable storage media as recited in claim 21, wherein identifying a single-node event burst on a node comprises identifying at least two outage events on a node that occur within a predetermined time from one another.

24. One or more processor-readable storage media as recited in claim 21, wherein identifying a single-node event burst comprises identifying a plurality of outage events on a node, each of the plurality of outage events occurring within a predetermined time of at least one other of the plurality of outage events.

25. One or more processor-readable storage media as recited in claim 19, wherein identifying the multi-node event bursts comprises identifying single-node event bursts occurring on at least two different nodes according to a predetermined timing rule.

26. One or more processor-readable storage media as recited in claim 19, wherein identifying the multi-node event burst comprises identifying single-node event bursts occurring on at least two different nodes within a predetermined time relative to one another.

27. One or more processor-readable storage media as recited in claim 19, wherein identifying the multi-node event bursts comprises identifying a plurality of single-node event bursts, each of the plurality of single-node event bursts occurring within a predetermined time relative to at least one other of the plurality of outage events, wherein at least two of the single-node event burst occur on different nodes.

28. One or more processor-readable storage media as recited in claim 19, wherein relationship strength values are determined using a relationship strength rule.

29. One or more processor-readable storage media as recited in claim 19, wherein relationship strength values are determined using a plurality of relationship strength rules.

30. One or more processor-readable storage media as recited in claim 19, wherein relationship strength values are determined using a plurality of relationship strength rules and the similarity of node names.

31. One or more processor-readable storage media as recited in claim 19, wherein relationship strength values are determined using a plurality of relationship strength rules and node name similarity values.

32. One or more processor-readable storage media as recited in claim 19, wherein each node has a name associated therewith and wherein a relationship strength value for a pair of nodes is determined using at least one relationship strength rule and value indicating a relative similarity between the names of the pair of nodes.

53. One or more processor-readable storage media as recited in claim 21, wherein identifying a single-node event burst on a node comprises identifying outage events on a node occurring within a specified time frame of one another.

33. One or more processor-readable storage media having stored thereon processor executable instructions for performing operations comprising:

identifying a multi-node event burst occurring in a node group based on the event data collected from each of a plurality of nodes in a node group;

determining a cluster of related nodes based on the identified multi-node event burst; and

determining at least one metric for cluster.

34. One or more processor-readable storage media as recited in claim 33, wherein the at least one metric is a monitoring time period for the cluster.

35. One or more processor-readable storage media as recited in claim 33, wherein the at least one metric is a time period over which all nodes in the cluster are simultaneously down.

36. One or more processor-readable storage media as recited in claim 33, wherein the at least one metric is a time period over which at least one of the nodes in the cluster down and not all of the nodes in the cluster are down.

37. One or more processor-readable storage media as recited in claim 33, wherein the at least one metric comprises a plurality of metrics.

38. One or more processor-readable storage media as recited in claim 33, further having stored thereon processor executable instructions for performing an operation comprising:

presenting the determined metric to a user.

39. A computer system comprising:

means for obtaining event data from each of a plurality of nodes of a group of associated nodes;

means for identifying single-node event bursts occurring on the nodes based on the obtained event data, including means for identifying at least one node event that does not have a corresponding opposite node event and means for creating an opposite node event associated with the at least one node event; and

means for identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node bursts.

40. A computer system as recited in claim 39, wherein identifying a single-nod event burst on a node comprises identifying outage events on a node occurring within a specified time frame.

41. A computer system as recited in claim 39, wherein identifying a single-nod event burst on a node comprises identifying at least two outage events on a node that occur within a predetermined time from one another.

42. A computer system as recited in claim 39, wherein identifying a single-nod event burst on a node comprises identifying a plurality of outage events on a node, each of the plurality of outage events occurring within a predetermined time of at least one other of the plurality of outage events.

43. A computer system as recited in claim 39, wherein identifying the multi-nod event bursts comprises identifying single-node event bursts occurring on at least two different nodes within a predetermined time.

44. A computer system as recited in claim 39, wherein identifying the multi-node event burst comprises identifying single-node event bursts occurring on at least two different nodes within a predetermined time from one another.

45. A computer system as recited in claim 39, wherein identifying the multi-node event burst comprises identifying a plurality of single-node event bursts, each of the plurality of single-node event bursts occurring within a predetermined time of at least one other of the plurality of the plurality of single node event bursts, wherein at least two of the single-node event burst occur on different nodes.

46. A computer system as recited in claim 39, wherein obtaining event data from a node comprises obtaining data from an event log associated with the node.

47. A computer system as recited in claim 39, wherein obtaining event data from a node comprises obtaining information identifying up events and down events that have occurred on the node.

48. A computer system as recited in claim 39, wherein obtaining event data from a node comprises obtaining from an event log at the node information identifying up events and down events that have occurred on the node.

49. A computer system as recited in claim 39, further comprising means for storing the identified single-node event bursts in a single-node event bursts data structure and wherein the multi-node event burst is identified based on the single-node bursts stored in the single-node event bursts data structure.

50. A computer system comprising:

means for obtaining event data from each of a plurality of nodes of a group of associated nodes;

means for identifying single-node event bursts occurring on the nodes based on the obtained event data;

means for identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node bursts; and

means for storing the obtained event data in an event table and wherein identifying single-node event bursts includes identifying at least one up event that does not have a corresponding down event in the data structure and creating an associated down event for the up event.

51. A computer system comprising:

means for obtaining event data from each of a plurality of nodes of a group of associated nodes;

means for identifying single-node event bursts occurring on the nodes based on the obtained event data;

means for identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node bursts; and

means for storing the obtained event data in an event table and wherein identifying single-node event bursts includes identifying at least one down event that does not have a corresponding up event in the data structure and creating an associated up event for the down event.

52. A computer system comprising:

means for obtaining event data from each of a plurality of nodes of a group of associated nodes;

means for identifying single-node event bursts occurring on the nodes based on the obtained event data;

means for identifying a multi-node event burst occurring in the group of associated nodes based on the identified single-node bursts; and

means for determining a relationship strength value for two nodes having associated single-node event bursts occurring in an identified multi-node event burst.