Proactive and adaptive cloud monitoring

• US 8,904,241 B2
• Filed: 07/27/2011
• Issued: 12/02/2014
• Est. Priority Date: 07/27/2011
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Srivastava, Deepti, Ingham, Andrew, Hsu, Cheng-Lu, Chan, Wilson Wai Shun
• Primary Examiner(s): Truong, Loan L. T.
• Application Number:
  US13/192,231
• Publication Number:
  US 20130031424A1
• Time in Patent Office: 1,224 Days
• Field of Search: 714/47.2, 714/47.3, 714/47.1
• US Class Current: 714/47.3
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis

Abstract

Processes, computer-readable media, and machines are disclosed for reducing a likelihood that active functional components fail in a computing system. An active monitoring component receives metrics associated with different active functional components of a computing system. The different active functional components contribute to different functionalities of the system. Based at least in part on the metrics associated with a particular active functional component, the active monitoring component determines that the particular active functional component has reached a likelihood of failure but has not failed. In response to determining that the particular active functional component has reached the likelihood of failure but has not failed, the active monitoring component causes a set of actions that are predicted to reduce the likelihood of failure.

Claims

1. A process comprising:

an active monitoring component receiving one or more metrics associated with each of a first active functional component and a second active functional component of a plurality of active functional components of a system, wherein the first active functional component contributes to a different functionality of the system than the second active functional component;

based at least in part on the one or more metrics associated with a particular active functional component of the first active functional component or the second active functional component, the active monitoring component determining that the particular active functional component has reached a likelihood of failure but has not failed;

in response to determining that the particular active functional component has reached the likelihood of failure but has not failed, the active monitoring component causing a set of one or more actions that are predicted to reduce the likelihood of failure;

wherein the likelihood of failure is a first likelihood of failure;

wherein the set of one or more actions is a first set of one or more actions;

the active monitoring component receiving one or more updated metrics associated with the particular active functional component;

based at least in part on the one or more updated metrics associated with the particular active functional component, the active monitoring component determining that the particular active functional component has reached a second likelihood of failure but has not failed, wherein the second likelihood of failure is greater than at the first likelihood of failure;

in response to determining that the particular active functional component has reached the second likelihood of failure, the active monitoring component causing a second set of one or more actions that are predicted to reduce the second likelihood of failure;

wherein, at any given likelihood of failure of the particular active functional component, the second set of one or more actions has a greater risk of causing the system to fail than the first set of one or more actions;

wherein the process is performed by one or more computing devices.

2. The process of claim 1, wherein the set of actions is a first set of actions, the process further comprising:

the active monitoring component evaluating changes in metric values after a plurality of actions are caused, wherein each of the plurality of actions was caused to reduce the likelihood of failure of the particular active functional component;

in response to evaluating the changes in metric values, the active monitoring component determining a second set of one or more actions that is more likely than the first set of one or more actions to reduce the likelihood of failure, wherein the second set of actions is different than the first set of actions.

3. The process of claim 2, further comprising:

the active monitoring component receiving one or more updated metrics associated with the particular active functional component;

based at least in part on the one or more updated metrics associated with the particular active functional component, the active monitoring component determining that the particular active functional component has again reached the likelihood of failure; and

in response to determining that the particular active functional component has again reached the likelihood of failure, the active monitoring component causing the second set of one or more actions to reduce the likelihood of failure.

4. The process of claim 1, wherein the one or more metrics comprise one or more of:

a quantity of connections used by said particular active functional component;

a quantity of processor cycles used by said particular active functional component;

a quantity of memory used by said particular active functional component;

a time since a heartbeat message from said particular active functional component;

a quantity of active functional components waiting on a same resource as said particular active functional component;

an amount of time that a resource, upon which said particular active functional component is waiting, has been locked by another active functional component;

a response time of said particular active functional component;

or

a connection quality of said particular active functional component.

5. The process of claim 1, wherein the one or more metrics comprise one or more of:

a quantity of connections used by another active functional component related to said particular active functional component;

a quantity of processor cycles used by another active functional component related to said particular active functional component;

a quantity of memory used by another active functional component related to said particular active functional component;

a time since a heartbeat message from another active functional component related to said particular active functional component;

a quantity of active functional components waiting on a same resource as another active functional component related to said particular active functional component;

an amount of time that a resource, upon which another active functional component related to said particular active functional component is waiting, has been locked by an active functional component;

a response time of another active functional component related to said particular active functional component;

or

a connection quality of another active functional component related to said particular active functional component.

6. The process of claim 1, wherein the set of one or more actions comprises one or more of:

killing one or more processes,

increasing a priority of one or more processes,

decreasing a priority of one or more processes, rejecting one or more requests for new connections,

closing one or more existing connections,

starting one or more active functional components,

stopping one or more active functional components, or

triggering analysis of metrics other than the one or more metrics.

7. The process of claim 1, wherein evaluating the one or more metrics comprises evaluating a change in the one or more metrics from a first time to a second time.

8. The process of claim 1, wherein evaluating the one or more metrics comprises evaluating the one or more metrics using an action specification for the particular active functional component, wherein the action specification uses the one or more metrics to weigh the set of one or more actions, wherein the action specification uses one or more other metrics to weigh another set of one or more actions, and wherein the active monitoring component causing the set of one or more actions is in response to determining that the set of one or more actions has a better weight than the other set of one or more actions.

9. The process of claim 1, further comprising receiving, on an interface, registration information for the particular active functional component, wherein the registration information specifies an action specification that maps the one or more metrics to the set of one or more actions, wherein the action specification also maps one or more other metrics to another set of one or more actions.

10. The process of claim 1, further comprising:

receiving, on an interface, updated registration information for the particular active functional component;

using the updated registration information to automatically update a default action specification, wherein, before the update, the default action specification maps the one or more metrics to the set of actions, and wherein the updated action specification maps the one or more metrics to another set of actions;

wherein the particular active functional component reaches the level of potential failure at least a first time before the update and a second time after the update;

the active monitoring component using the default action specification to determine the set of actions to cause in response to the first time that the particular active functional component has reached the level of potential failure, and the active monitoring component using the updated action specification to determine the other set of actions to cause in response to the second time that the particular active functional component has reached the level of potential failure.

21. The process of claim 1, wherein the particular active functional component is a critical active functional component, wherein the system cannot operate without the critical active functional component, and wherein the set of one or more actions includes stopping a non-critical active functional component of the system without stopping operation of the system.

11. One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause:

an active monitoring component receiving one or more metrics associated with each of a first active functional component and a second active functional component of a plurality of active functional components of a system, wherein the first active functional component contributes to a different functionality of the system than the second active functional component;

based at least in part on the one or more metrics associated with a particular active functional component of the first active functional component or the second active functional component, the active monitoring component determining that the particular active functional component has reached a likelihood of failure but has not failed;

in response to determining that the particular active functional component has reached the likelihood of failure but has not failed, the active monitoring component causing a set of one or more actions that are predicted to reduce the likelihood of failure;

wherein the likelihood of failure is a first likelihood of failure;

wherein the set of one or more actions is a first set of one or more actions;

the active monitoring component receiving one or more updated metrics associated with the particular active functional component;

based at least in part on the one or more updated metrics associated with the particular active functional component, the active monitoring component determining that the particular active functional component has reached a second likelihood of failure but has not failed, wherein the second likelihood of failure is greater than at the first likelihood of failure;

in response to determining that the particular active functional component has reached the second likelihood of failure, the active monitoring component causing a second set of one or more actions that are predicted to reduce the second likelihood of failure;

wherein, at any given likelihood of failure of the particular active functional component, the second set of one or more actions has a greater risk of causing the system to fail than the first set of one or more actions.

12. The one or more non-transitory storage media of claim 11, wherein the set of actions is a first set of actions, wherein the instructions, when executed by one or more computing devices, further cause:

the active monitoring component evaluating changes in metric values after a plurality of actions are caused, wherein each of the plurality of actions was caused to reduce the likelihood of failure of the particular active functional component;

in response to evaluating the changes in metric values, the active monitoring component determining a second set of one or more actions that is more likely than the first set of one or more actions to reduce the likelihood of failure, wherein the second set of actions is different than the first set of actions.

13. The one or more non-transitory storage media of claim 12, wherein the instructions, when executed by one or more computing devices, further cause:

the active monitoring component receiving one or more updated metrics associated with the particular active functional component;

based at least in part on the one or more updated metrics associated with the particular active functional component, the active monitoring component determining that the particular active functional component has again reached the likelihood of failure; and

in response to determining that the particular active functional component has again reached the likelihood of failure, the active monitoring component causing the second set of one or more actions to reduce the likelihood of failure.

14. The one or more non-transitory storage media of claim 11, wherein the one or more metrics comprise one or more of:

a quantity of connections used by said particular active functional component;

a quantity of processor cycles used by said particular active functional component;

a quantity of memory used by said particular active functional component;

a time since a heartbeat message from said particular active functional component;

a quantity of active functional components waiting on a same resource as said particular active functional component;

an amount of time that a resource, upon which said particular active functional component is waiting, has been locked by another active functional component;

a response time of said particular active functional component;

or

a connection quality of said particular active functional component.

15. The one or more non-transitory storage media of claim 11, wherein the one or more metrics comprise one or more of:

a quantity of connections used by another active functional component related to said particular active functional component;

a quantity of processor cycles used by another active functional component related to said particular active functional component;

a quantity of memory used by another active functional component related to said particular active functional component;

a time since a heartbeat message from another active functional component related to said particular active functional component;

a quantity of active functional components waiting on a same resource as another active functional component related to said particular active functional component;

an amount of time that a resource, upon which another active functional component related to said particular active functional component is waiting, has been locked by an active functional component;

a response time of another active functional component related to said particular active functional component;

or

a connection quality of another active functional component related to said particular active functional component.

16. The one or more non-transitory storage media of claim 11, wherein the set of one or more actions comprises one or more of:

killing one or more processes,

increasing a priority of one or more processes,

decreasing a priority of one or more processes, rejecting one or more requests for new connections,

closing one or more existing connections,

starting one or more active functional components,

stopping one or more active functional components, or

triggering analysis of metrics other than the one or more metrics.

17. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by one or more computing devices, cause evaluating the one or more metrics by causing evaluating a change in the one or more metrics from a first time to a second time.

18. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by one or more computing devices, cause evaluating the one or more metrics by causing evaluating the one or more metrics using an action specification for the particular active functional component, wherein the action specification uses the one or more metrics to weigh the set of one or more actions, wherein the action specification uses one or more other metrics to weigh another set of one or more actions, and wherein the instructions, when executed by one or more computing devices, cause the active monitoring component causing the set of one or more actions in response to determining that the set of one or more actions has a better weight than the other set of one or more actions.

19. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by one or more computing devices, further cause receiving, on an interface, registration information for the particular active functional component, wherein the registration information specifies an action specification that maps the one or more metrics to the set of one or more actions, wherein the action specification also maps one or more other metrics to another set of one or more actions.

20. The one or more non-transitory storage media of claim 11, wherein the instructions, when executed by one or more computing devices, further cause:

receiving, on an interface, updated registration information for the particular active functional component;

using the updated registration information to automatically update a default action specification, wherein, before the update, the default action specification maps the one or more metrics to the set of actions, and wherein the updated action specification maps the one or more metrics to another set of actions;

wherein the particular active functional component reaches the level of potential failure at least a first time before the update and a second time after the update;

the active monitoring component using the default action specification to determine the set of actions to cause in response to the first time that the particular active functional component has reached the level of potential failure, and the active monitoring component using the updated action specification to determine the other set of actions to cause in response to the second time that the particular active functional component has reached the level of potential failure.

22. The one or more non-transitory storage media of claim 11, wherein the particular active functional component is a critical active functional component, wherein the system cannot operate without the critical active functional component, and wherein the set of one or more actions includes stopping a non-critical active functional component of the system without stopping operation of the system.