Method and apparatus for identifying problem causes in a multi-node system

• US 7,701,859 B2
• Filed: 02/14/2006
• Issued: 04/20/2010
• Est. Priority Date: 02/14/2006
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Oracle America, Inc. (Oracle Corporation)
• Inventors: Helgren, Matthew, Belkin, Ruslan V.
• Primary Examiner(s): Trost, IV; William
• Assistant Examiner(s): Jiang; Charles C
• Application Number:
  US11/354,703
• Publication Number:
  US 20070192150A1
• Time in Patent Office: 1,526 Days
• Field of Search: 370/241, 370/242, 714/100, 714/1, 714/25, 714/26
• US Class Current: 370/242
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Abstract

An SLO (service level objective) is represented by a model that includes nodes that represent elements in a system that are used to fulfill the SLO and information that represents dependencies between the elements. Telemetry information is received describing a condition of an element in the system. The telemetry information can be applied to a particular procedure associated with a particular node in the model to determine if there is a problem associated with the element represented by the particular node. At least a portion of the telemetry information is applied to procedures to determine problem cause information describing which elements have problems relating to the SLO. A relative contribution of elements to a problem associated with the SLO is determined by analyzing the problem cause information and the dependencies between the elements.

Claims

1. A computer-implemented method for determining one or more causes of a problem with a service level objective (SLO) in a network management system (NMS), the method comprising:

receiving telemetry information describing a condition of at least a first element in the NMS, wherein the first element is a service or a resource used to fulfill the SLO, and wherein the first element is associated with a first node;

invoking a first procedure associated with the first element, wherein the first procedure determines whether the first element is contributing to the problem with the SLO, wherein the problem is indicated by system performance reaching a confidence level specified in the SLO;

determining that the first element has a dependency relationship with a second element in the NMS;

invoking a second procedure associated with the second element, wherein the second procedure determines whether the second element is contributing to the problem with the SLO, and wherein the second element is associated with a second node;

making a determination, using a monitoring node, of one or more causes of the problem with the SLO based at least partially upon results of the first procedure and the second procedure, wherein the problem is based upon at least one from a group consisting of the first element contributing to the problem with the SLO, and the second element contributing to the problem with the SLO, and wherein the NMS comprises the first node, the second node, and the monitoring node operatively connected; and

generating, based on the determination, a problem report that indicates one or more causes of the problem with the SLO;

wherein the method is performed by a computing device which, as a result of executing special purpose instructions, has been configured to be a special purpose device.

2. The method of claim 1, wherein the telemetry information comprises performance information for at least the first and second elements.

3. The method of claim 1, wherein the operation of invoking the second procedure is performed by the first procedure.

4. The method of claim 1, wherein generating the problem report comprises generating information indicating which one or more of the first and second elements is contributing to the problem with the SLO.

5. The method of claim 4, wherein generating information indicating which one or more of the first and second elements is contributing to the problem with the SLO comprises:

determining a relative contribution made by each of the first and second elements to the problem with the SLO, wherein determining the relative contribution comprises assigning one or more numerical values indicating proportional contribution by the first and second element to the problem with the SLO; and

generating information indicating the relative contribution made by each of the first and second elements.

6. The method of claim 1, further comprising:

prior to generating the problem report;

determining that the second element has a dependency relationship with a third element in the system; and

invoking a third procedure associated with the third element, wherein the third procedure determines whether the third element is contributing to the problem with the SLO;

wherein generating the problem report comprises generating, based at least partially upon determinations made by the first, second, and third procedures, information indicating which one or more of the first, second, and third elements is contributing to the problem with the SLO.

7. The method of claim 6, wherein the operation of invoking the second procedure is performed by the first procedure, and wherein the operation of invoking the third procedure is performed by the second procedure.

8. The method of claim 6, wherein generating information indicating which one or more of the first, second, and third elements is contributing to the problem with the SLO comprises:

determining a relative contribution made by each of the first, second, and third elements to the problem with the SLO, wherein determining the relative contribution comprises assigning one or more numerical values indicating proportional contribution by the first, second, and third elements to the problem with the SLO; and

generating information indicating the relative contribution made by each of the first, second, and third elements.

9. The method of claim 1, wherein determining that the first element has a dependency relationship with the second element comprises:

accessing a dependency model that specifies dependency relationships between the SLO and a plurality of elements, wherein the plurality of elements comprises elements used to fulfill the SLO, and wherein the dependency model specifies how the plurality of elements depend on each other and how the SLO depends on the plurality of elements.

10. The method of claim 9, further comprising:

receiving a set of SLO information indicating one or more performance metrics that need to be met to satisfy the SLO;

receiving a set of repository information identifying the plurality of elements and their interdependencies; and

constructing the dependency model based upon the SLO information and the repository information.

11. A computer-readable storage medium storing one or more sequences of instructions embodied therein for execution on a computer system to perform a computer-implemented method, the method comprising:

receiving telemetry information describing a condition of at least a first element in a network management system (NMS), wherein the first element is a service or a resource used to fulfill a service level objective (SLO) in the NMS, and wherein the first element is associated with a first node;

invoking a first procedure associated with the first element, wherein the first procedure determines whether the first element is contributing to a problem with the SLO, wherein the problem is indicated by system performance reaching a confidence level specified in the SLO;

determining that the first element has a dependency relationship with a second element in the NMS;

invoking a second procedure associated with the second element, wherein the second procedure determines whether the second element is contributing to the problem with the SLO, and wherein the second element is associated with a second node;

making a determination, using a monitoring node, of one or more causes of the problem with the SLO based at least partially upon results of the first procedure and the second procedure, wherein the problem is based upon at least one from a group consisting of the first element contributing to the problem with the SLO, and the second element contributing to the problem with the SLO, and wherein the NMS comprises the first node, the second node, and the monitoring node operatively connected; and

generating, based on the determination, a problem report that indicates one or more causes of the problem with the SLO;

wherein the method is performed by a computing device which, as a result of executing special purpose instructions, has been configured to be a special purpose device.

12. The computer-readable storage medium of claim 11, wherein the telemetry information comprises performance information for at least the first and second elements.

13. The computer-readable storage medium of claim 11, wherein the operation of invoking the second procedure is performed by the first procedure.

14. The computer-readable storage medium of claim 11, wherein generating the problem report comprises generating information indicating which one or more of the first and second elements is contributing to the problem with the SLO.

15. The computer-readable storage medium of claim 14, wherein generating information indicating which one or more of the first and second elements is contributing to the problem with the SLO comprises:

determining a relative contribution made by each of the first and second elements to the problem with the SLO, wherein determining the relative contribution comprises assigning one or more numerical values indicating proportional contribution by the first and second element to the problem with the SLO; and

generating information indicating the relative contribution made by each of the first and second elements.

16. The computer-readable storage medium of claim 11, wherein the one or more sequences of instructions further cause the one or more processors to perform the operations of:

prior to generating the problem report;

determining that the second element has a dependency relationship with a third element in the system; and

invoking a third procedure associated with the third element, wherein the third procedure determines whether the third element is contributing to the problem with the SLO;

wherein generating the problem report comprises generating, based at least partially upon determinations made by the first, second, and third procedures, information indicating which one or more of the first, second, and third elements is contributing to the problem with the SLO.

17. The computer-readable storage medium of claim 16, wherein the operation of invoking the second procedure is performed by the first procedure, and wherein the operation of invoking the third procedure is performed by the second procedure.

18. The computer-readable storage medium of claim 16, wherein generating information indicating which one or more of the first, second, and third elements is contributing to the problem with the SLO comprises:

determining a relative contribution made by each of the first, second, and third elements to the problem with the SLO, wherein determining the relative contribution comprises assigning one or more numerical values indicating proportional contribution by the first, second, and third elements to the problem with the SLO; and

generating information indicating the relative contribution made by each of the first, second, and third elements.

19. The computer-readable storage medium of claim 11, wherein determining that the first element has a dependency relationship with the second element comprises:

accessing a dependency model that specifies dependency relationships between the SLO and a plurality of elements, wherein the plurality of elements comprises elements used to fulfill the SLO, and wherein the dependency model specifies how the plurality of elements depend on each other and how the SLO depends on the plurality of elements.

20. The computer-readable storage medium of claim 19, wherein the one or more sequences of instructions further cause the one or more processors to perform the operations of:

receiving a set of SLO information indicating one or more performance metrics that need to be met to satisfy the SLO;

receiving a set of repository information identifying the plurality of elements and their interdependencies; and

constructing the dependency model based upon the SLO information and the repository information.

21. A computer system, comprising:

one or more processors; and

a storage for storing one or more sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform the operations of;

receiving telemetry information describing a condition of at least a first element in a network management system (NMS), wherein the first element is a service or a resource used to fulfill a service level objective (SLO) in the NMS, and wherein the first element is associated with a first node;

invoking a first procedure associated with the first element, wherein the first procedure determines whether the first element is contributing to a problem with the SLO, wherein the problem in indicated by system performance reaching a confidence level specified in the SLO;

determining that the first element has a dependency relationship with a second element in the NMS;

invoking a second procedure associated with the second element, wherein the second procedure determines whether the second element is contributing to the problem with the SLO, and wherein the second element is associated with a second node;

making a determination, using a monitoring node, of one or more causes of the problem with the SLO based at least partially upon results of the first procedure and the second procedure, wherein the problem is based upon at least one from a group consisting of the first element contributing to the problem with the SLO, and the second element contributing to the problem with the SLO, and wherein the NMS comprises the first node, the second node, and the monitoring node operatively connected; and

generating, based on the determination, a problem report that indicates one or more causes of the problem with the SLO;

wherein the method is performed by a computing device which, as a result of executing special purpose instructions, has been configured to be a special purpose device.

22. The computer system of claim 21, wherein the one or more sequences of instructions further cause the one or more processors to perform the operations of:

prior to generating the problem report;

determining that the second element has a dependency relationship with a third element in the system; and

invoking a third procedure associated with the third element, wherein the third procedure determines whether the third element is contributing to the problem with the SLO;

wherein generating the problem report comprises generating, based at least partially upon determinations made by the first, second, and third procedures, information indicating which one or more of the first, second, and third elements is contributing to the problem with the SLO.

23. The computer system of claim 22, wherein the operation of invoking the second procedure is performed by the first procedure, and wherein the operation of invoking the third procedure is performed by the second procedure.

24. The computer system of claim 22, wherein generating information indicating which one or more of the first, second, and third elements is contributing to the problem with the SLO comprises:

determining a relative contribution made by each of the first, second, and third elements to the problem with the SLO, wherein determining the relative contribution comprises assigning one or more numerical values indicating proportional contribution by the first, second, and third elements to the problem with the SLO; and

generating information indicating the relative contribution made by each of the first, second, and third elements.

25. The computer system of claim 21, wherein determining that the first element has a dependency relationship with the second element comprises:

accessing a dependency model that specifies dependency relationships between the SLO and a plurality of elements, wherein the plurality of elements comprises elements used to fulfill the SLO, and wherein the dependency model specifies how the plurality of elements depend on each other and how the SLO depends on the plurality of elements.

26. The computer system of claim 25, further comprising:

receiving a set of SLO information indicating one or more performance metrics that need to be met to satisfy the SLO;

receiving a set of repository information identifying the plurality of elements and their interdependencies; and

constructing the dependency model based upon the SLO information and the repository information.