Determination of related failure events in a multi-node system

• US 20050188240A1
• Filed: 12/19/2003
• Published: 08/25/2005
• Est. Priority Date: 12/19/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ganapathi, Archana, Murphy, Brendan
• Granted Patent: US 7,409,604 B2
• Time in Patent Office: Days
• US Class Current: 714/4
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: H04L 41/064 involving time analysis
• CPC Class Codes: H04L 43/091 Measuring contribution of individual network components to actual service level

Abstract

Systems and methods for determining related node failures in a multi-node system use log data obtained from the nodes. This log data is processed in various ways to indicate clusters of nodes that experience related failures.

Claims

20. A processor-readable media having stored thereon processor executable instructions for performing acts comprising:

obtaining event data from each of a plurality of nodes in a node group;

identifying a multi-node event burst occurring in the node group based on the event data;

determining relationship strength values for pairs of nodes based on the identified multi-node event burst; and

determining at least one cluster of related nodes based on the determined relationship strength values.

1. A method comprising:

obtaining event data from each node in a group of associated nodes;

identifying at least one single-node event burst occurring on each of a plurality of the nodes in the group of nodes based on the obtained event data; and

identifying a multi-node event bursts occurring in the group of associated nodes based on the identified single-node event bursts.

2. A method as recited in claim 1, wherein identifying a single-node event burst on a node comprises identifying outage events on a node occurring within a specified time frame.

3. A method as recited in claim 1, wherein identifying a single-node event burst on a node comprises identifying at least two outage events on a node that occur within a predetermined time from one another.

4. A method as recited in claim 1, wherein identifying a single-node event burst on a node comprises identifying a plurality of outage events on a node, each of the plurality of outage events occurring within a predetermined time of at least one other of the plurality of outage events.

5. A method as recited in claim 1, wherein identifying the multi-node event bursts comprises identifying single-node event bursts occurring on at least two different nodes according to a predetermined timing rule.

6. A method as recited in claim 1, wherein identifying the multi-node event burst comprises identifying single-node event bursts occurring on at least two different nodes within a predetermined time from one another.

7. A method as recited in claim 1, wherein identifying the multi-node event bursts comprises identifying a plurality of single-node event bursts, each of the plurality of single-node event bursts occurring within a predetermined time relative to at least one other of the plurality of outage events, wherein at least two of the single-node event burst occur on different nodes.

8. A method as recited in claim 1, wherein obtaining event data from a node comprises obtaining data from an event log associated with the node.

9. A method as recited in claim 1, wherein obtaining event data from a node comprises obtaining information identifying up events and down events that have occurred on the node.

10. A method as recited in claim 1, wherein obtaining event data from a node comprises obtaining from an event log at the node information identifying up events and down events that have occurred on the node.

11. A method as recited in claim 1, further comprising storing the identified single-node event bursts in a single-node event bursts data structure and wherein the multi-node event bursts is identified based on the single-node bursts stored in the single-node event bursts data structure.

12. A method as recited in claim 1, further comprising storing the obtaining event data in an event table and wherein identifying a single-node event bursts includes identifying at least one up event that does not have a corresponding down event in the data structure and creating an associated down event for the up event.

13. A method as recited in claim 1, further comprising storing the obtaining event data in an event table and wherein identifying a single-node event bursts includes identifying at least one down event that does not have a corresponding up event in the data structure and creating an associated up event for the down event.

14. A method as recited in claim 1, further comprising determining a relationship strength value for two nodes having associated single-node event burst occurring in the identified multi-node event burst.

15. A method as recited in claim 1, further comprising determining a relationship strength value for two nodes having associated single-node event burst occurring in the identified multi-node event burst.

16. A method as recited in claim 1, further comprising:

determining relationship strength values for pairs of nodes based on the identified multi-node event burst; and

determining at least one cluster of related nodes based on the determined relationship strength values.

17. A method as recited in claim 1, further comprising:

determining relationship strength values for pairs of nodes based on the identified multi-node event burst;

determining at least one cluster of related nodes based on the determined relationship strength values; and

determining at least one metric for the at least one cluster.

18. A method as recited in claim 1, further comprising:

determining relationship strength values for pairs of nodes based on the identified multi-node event burst;

determining at least one cluster of related nodes based on the determined relationship strength values;

determining at least one metric for the at least one cluster; and

presenting the determined metric to a user.

19. A method as recited in claim 1, further comprising:

determining relationship strength values for pairs of nodes based on the identified multi-node event burst;

determining at least one cluster of related nodes based on the determined relationship strength values;

determining a plurality of metrics for the at least one cluster; and

presenting the determined plurality of metrics to a user.

21. A processor-readable media as recited in claim 20, further having stored thereon processor executable instructions for performing acts comprising:

determining at least one metric for the at least one cluster; and

presenting the determined metric to a user.

22. A processor-readable media as recited in claim 20, further having stored thereon processor executable instructions for performing acts comprising:

determining a plurality of metrics for the at least one cluster; and

presenting the determined plurality of metrics to a user.

23. A processor-readable media as recited in claim 20, wherein identifying a multi-node event burst comprises:

identifying a single-node event bursts on each of a plurality of nodes based on the obtained event data; and

identifying the multi-node event burst based on the identified single-node event bursts.

24. A processor-readable media as recited in claim 22, wherein identifying a single-node event burst on a node comprises identifying at least two outage events on a node that occur within a predetermined time from one another.

25. A processor-readable media as recited in claim 22, wherein identifying a single-node event burst comprises identifying a plurality of outage events on a node, each of the plurality of outage events occurring within a predetermined time of at least one other of the plurality of outage events.

26. A processor-readable media as recited in claim 20, wherein identifying the multi-node event bursts comprises identifying single-node event bursts occurring on at least two different nodes according to a predetermined timing rule.

27. A method as recited in claim 20, wherein identifying the multi-node event burst comprises identifying single-node event bursts occurring on at least two different nodes within a predetermined relative to one another.

28. A method as recited in claim 20, wherein identifying the multi-node event bursts comprises identifying a plurality of single-node event bursts, each of the plurality of single-node event bursts occurring within a predetermined time relative to at least one other of the plurality of outage events, wherein at least two of the single-node event burst occur on different nodes.

29. A method as recited in claim 20, wherein relationship strength values are determined using a relationship strength rule.

30. A method as recited in claim 20, wherein relationship strength values are determined using a plurality of relationship strength rules.

31. A method as recited in claim 20, wherein relationship strength values are determined using a plurality of relationship strength rules and the similarity of node names.

32. A method as recited in claim 20, wherein relationship strength values are determined using a plurality of relationship strength rules and node name similarity values.

33. A method as recited in claim 20, wherein each node has a name associated therewith and wherein a relationship strength value for a pair of nodes is determined using at least one relationship strength rule and value indicating a relative similarity between the names of the pair of nodes.

35. A processor-readable media as recited in claim 20, wherein the at least one metric is a monitoring time period for the cluster.

36. A processor-readable media as recited in claim 20, wherein the at least on metric is a time period over which all nodes in the cluster are simultaneously down.

37. A processor-readable media as recited in claim 20, wherein the at least on metric is a time period over which at least one of the nodes in the cluster down and not all of the nodes in the cluster are down.

38. A processor-readable media as recited in claim 20, wherein the at least one metric comprises a plurality of metrics.

23-1. A processor-readable media as recited in claim 22, wherein identifying a single-node event burst on a node comprises identifying outage events on a node occurring within a specified time frame of one another.

34. A processor-readable media having stored thereon processor executable instructions for performing operations comprising:

identifying a multi-node event burst occurring in a node group based on the event data collected from each of a plurality of nodes in a node group;

determining a cluster of related nodes based on the identified multi-node event burst; and

determining at least one metric for cluster.

39. A processor-readable media as recited in claim 34, further having stored thereon processor executable instructions for performing an operation comprising:

presenting the determined metric to a user.

40. A computer system comprising:

means for obtaining event data from each of a plurality of nodes of a group of associated nodes;

means for identifying single-node event bursts occurring on the nodes based on the obtained event data;

means for identifying a multi-node event bursts occurring in the group of associated nodes based on the identified single-node bursts.

41. A computer system as recited in claim 40, wherein identifying a single-node event burst on a node comprises identifying outage events on a node occurring within a specified time frame.

42. A computer system as recited in claim 40, wherein identifying a single-node event burst on a node comprises identifying at least two outage events on a node that occur within a predetermined time from one another.

43. A computer system as recited in claim 40, wherein identifying a single-node event burst on a node comprises identifying a plurality of outage events on a node, each of the plurality of outage events occurring within a predetermined time of at least one other of the plurality of outage events.

44. A computer system as recited in claim 40, wherein identifying the multi-node event bursts comprises identifying single-node event bursts occurring on at least two different nodes within a predetermined time.

45. A computer system as recited in claim 40, wherein identifying the multi-node event burst comprises identifying single-node event bursts occurring on at least two different nodes within a predetermined time from one another.

46. A computer system as recited in claim 40, wherein identifying the multi-node event bursts comprises identifying a plurality of single-node event bursts, each of the plurality of single-node event bursts occurring within a predetermined time of at least one other of the plurality of the plurality of single node event bursts, wherein at least two of the single-node event burst occur on different nodes.

47. A computer system as recited in claim 40, wherein obtaining event data from a node comprises obtaining data from an event log associated with the node.

48. A computer system as recited in claim 40, wherein obtaining event data from a node comprises obtaining information identifying up events and down events that have occurred on the node.

49. A computer system as recited in claim 40, wherein obtaining event data from a node comprises obtaining from an event log at the node information identifying up events and down events that have occurred on the node.

50. A computer system as recited in claim 40, further comprising means for storing the identified single-node event bursts in a single-node event bursts data structure and wherein the multi-node event burst is identified based on the single-node bursts stored in the single-node event bursts data structure.

51. A computer system as recited in claim 40, further comprising means for storing the obtaining event data in an event table and wherein identifying single-node event bursts includes identifying at least one up event that does not have a corresponding down event in the data structure and creating an associated down event for the up event.

52. A computer system as recited in claim 40, further comprising means for storing the obtaining event data in an event table and wherein identifying single-node event bursts includes identifying at least one down event that does not have a corresponding up event in the data structure and creating an associated up event for the down event.

53. A computer system as recited in claim 40, further comprising means for determining a relationship strength value for two nodes having associated single-node event bursts occurring in an identified multi-node event burst.

54. A computer system as recited in claim 40, further comprising means for determining a relationship strength value for two nodes having associated single-node event bursts occurring in an identified multi-node event burst.