Method and apparatus for identifying problem causes in a multi-node system

• US 20070192150A1
• Filed: 02/14/2006
• Published: 08/16/2007
• Est. Priority Date: 02/14/2006
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Oracle America, Inc. (Oracle Corporation)
• Inventors: Helgren, Matthew, Belkin, Ruslan
• Granted Patent: US 7,701,859 B2
• Time in Patent Office: Days
• US Class Current: 370/242
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Abstract

An SLO (service level objective) is represented by a model that includes nodes that represent elements in a system that are used to fulfill the SLO and information that represents dependencies between the elements. Telemetry information is received describing a condition of an element in the system. The telemetry information can be applied to a particular procedure associated with a particular node in the model to determine if there is a problem associated with the element represented by the particular node. At least a portion of the telemetry information is applied to procedures to determine problem cause information describing which elements have problems relating to the SLO. A relative contribution of elements to a problem associated with the SLO is determined by analyzing the problem cause information and the dependencies between the elements.

Claims

1. A machine implemented method of determining a cause of a problem with a service level objective that is represented by a model that includes nodes that represent elements in a system that are used to fulfill the service level objective and information that represents dependencies between the elements, the method comprising:

receiving telemetry information that describes a condition of at least one element in the system, wherein the telemetry information can be applied to a particular procedure associated with a particular node in the model to determine if there is a problem associated with the particular element represented by the particular node;

applying at least a portion of the telemetry information to ones of the procedures to determine problem cause information describing which of the elements has a problem cause relating to the service level objective; and

determining relative contributions of the elements to a problem associated with the service level objective by analyzing the problem cause information and the dependencies between the elements.

2. The method as recited in claim 1, further comprising:

applying problem cause information associated with a first element used to fulfill the service level objective to a procedure associated with a node that represents a second element used to fulfill the service level objective that depends on the first element to determine problem cause information associated with the second element.

3. The method as recited in claim 1, further comprising:

reporting a violation of the service level objective; and

reporting the relative contribution of the elements as one or more causes of the violation of the service level objective.

4. The method as recited in claim 1, further comprising:

reporting no violation of the service level objective; and

reporting the relative contribution of the elements.

5. The method as recited in claim 1, further comprising:

reporting no violation of the service level objective; and

reporting the relative contribution of the elements as a potential future cause or causes of a violation of the service level objective.

6. The method as recited in claim 1, further comprising assigning a confidence value to ones of the relative contributions of the elements.

7. The method as recited in claim 1, wherein the determining relative contributions of the elements includes determining that a single element is the cause of the problem associated with the service level objective.

8. The method as recited in claim 1, wherein a particular element in the system is used to fulfill a plurality of service level objectives.

9. The method as recited in claim 1, further comprising modifying problem cause information determined by a first procedure associated with a first node that represents a first element, wherein said modifying is performed by a second procedure associated with a second node that represents a second element that depends on a first element.

10. The method as recited in claim 9, wherein said modifying includes removing the problem cause information determined by a first procedure.

11. The method as recited in claim 1, further comprising constructing the model.

12. The method as recited in claim 11, wherein the constructing the model includes:

accessing a definition of the service level objective; and

accessing an inventory database that stores dependencies between elements in the system.

13. The method as recited in claim 1, further comprising causing monitoring to be established for at least one of the elements represented by the nodes in order to cause the telemetry information to be available.

14. A machine readable medium having stored thereon a data structure describing a model that includes nodes that represent elements in a system that are used to fulfill a service level objective and information that represents dependencies between the elements, wherein the machine readable medium comprises a set of instructions which, when executed by one or more processors, causes the one or more processors to perform the following operations:

receive telemetry information that describes a condition of at least one element in the system, wherein the telemetry information can be applied to a particular procedure associated with a particular node in the model to determine if there is a problem associated with the particular element represented by the particular node;

apply at least a portion of the telemetry information to ones of the procedures to determine problem cause information describing which of the elements has a problem cause relating to the service level objective; and

determine relative contributions of the elements to a problem associated with the service level objective by analyzing the problem cause information and the dependencies between the elements.

15. The machine readable medium as recited in claim 14, wherein the set of instructions causes the one or more processors to further perform the following operation:

apply problem cause information associated with a first element used to fulfill the service level objective to a procedure associated with a node that represents a second element used to fulfill the service level objective that depends on the first element to determine problem cause information associated with the second element.

16. The machine readable medium as recited in claim 14, wherein the set of instructions causes the one or more processors to further perform the following operations:

report a violation of the service level objective; and

report the relative contribution of the elements as one or more causes of the violation of the service level objective.

17. The machine readable medium as recited in claim 14, wherein the set of instructions causes the one or more processors to further perform the following operations:

report no violation of the service level objective; and

report the relative contribution of the elements.

18. The machine readable medium as recited in claim 14, wherein the set of instructions causes the one or more processors to further perform the following operations:

report no violation of the service level objective; and

report the relative contribution of the elements as a potential future cause or causes of a violation of the service level objective.

19. The machine readable medium as recited in claim 14, wherein the set of instructions causes the one or more processors to further perform the following operation:

assign a confidence value to ones of the relative contributions of the elements.

20. The machine readable medium as recited in claim 19, wherein the set of instructions causes the one or more processors to further perform the following operation:

determine that a single element is the cause of the problem associated with the service level objective.

21. The machine readable medium as recited in claim 14, wherein a particular element in the system is used to fulfill a plurality of service level objectives.

22. The machine readable medium as recited in claim 14, wherein the set of instructions causes the one or more processors to further perform the following operation:

modify problem cause information determined by a first procedure associated with a first node that represents a first element, wherein said modifying is performed by a second procedure associated with a second node that represents a second element that depends on a first element.

23. The machine readable medium as recited in claim 22, wherein the instruction that causes the operation of modifying problem cause information further causes the one or more processors to perform the following operation:

remove the problem cause information determined by a first procedure.

24. The machine readable medium as recited in claim 14, wherein the set of instructions causes the one or more processors to further perform the following operation:

construct the model.

25. The machine readable medium as recited in claim 24, wherein the set of instructions causes the one or more processors to further perform the following operation to construct the model:

access a definition of the service level objective; and

access an inventory database that stores dependencies between elements in the system.

26. The machine readable medium as recited in claim 14, wherein the set of instructions causes the one or more processors to further perform the following operation:

cause monitoring to be established for at least one of the elements represented by the nodes in the model in order to cause the telemetry information to be available.