Proactive and adaptive cloud monitoring

• US 9,842,019 B2
• Filed: 12/01/2014
• Issued: 12/12/2017
• Est. Priority Date: 03/25/2010
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Srivastava, Deepti, Ingham, Andrew, Hsu, Cheng-Lu, Chan, Wilson Wai Shun
• Primary Examiner(s): TRUONG, LOAN
• Application Number:
  US14/556,357
• Publication Number:
  US 20150095720A1
• Time in Patent Office: 1,107 Days
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis

Abstract

Processes, computer-readable media, and machines are disclosed for reducing a likelihood that active functional components fail in a computing system. An active monitoring component receives metrics associated with different active functional components of a computing system. The different active functional components contribute to different functionalities of the system. Based at least in part on the metrics associated with a particular active functional component, the active monitoring component determines that the particular active functional component has reached a likelihood of failure but has not failed. In response to determining that the particular active functional component has reached the likelihood of failure but has not failed, the active monitoring component causes a set of actions that are predicted to reduce the likelihood of failure.

Claims

1. A method comprising:

determining that an active functional component of a computer system has reached a likelihood of failure based at least in part on a first value of a metric exceeding a metric threshold;

wherein the first value of the metric reflects performance of the active functional component at a time before a first action is caused to be performed in an attempt to reduce the likelihood of failure of the active functional component;

causing the first action to be performed in an attempt to reduce the likelihood of failure of the active functional component;

after causing the first action to be performed, determining that the active functional component has again reached the likelihood of failure based at least in part on a second value of the metric exceeding the metric threshold;

wherein the second value of the metric reflects performance of the active functional component at a time after the first action is caused to be performed;

causing a second action, that is different from the first action, to be performed in an attempt to reduce the likelihood of failure of the active functional component;

after causing the second action to be performed, obtaining a third value for the metric reflecting performance of the active functional component at a time after the second action is caused to be performed; and

using a machine learning component to determine a third action to be performed to reduce the likelihood of failure of the active functional component based at least in part on the second and the third values of the metric and the first and second actions caused to be performed; and

wherein the method is performed by one or more computing devices.

2. The method of claim 1, further comprising:

causing the third action to be performed in an attempt to reduce the likelihood of failure of the active functional component.

3. The method of claim 1, wherein the third action is the first action or the second action.

4. The method of claim 1, wherein registration information for the active functional component comprises a negative association with a non-critical functional component;

and wherein the third action comprises stopping the non-critical functional component.

5. The method of claim 1, wherein registration information for the active functional component comprises a positive association with a functional component on which the active functional component depends;

and wherein the third action comprises creating a new instance of the functional component on which the active functional component depends.

6. The method of claim 1, further comprising:

determining a first weight indicating an effectiveness of performing the first action at reducing the likelihood of failure of the active functional component;

determining a second weight indicating an effectiveness of performing the second action at reducing the likelihood of failure of the active functional component; and

using the machine learning component to determine the third action to perform to reduce the likelihood of failure of the active functional component based at least in part on the first and second weights.

7. The method of claim 1, further comprising using the machine learning component to determine the third action to perform to reduce the likelihood of failure of the active functional component based at least in part on a value reflecting an impact that performing the third action has on performance of the computer system.

8. The method of claim 1, wherein the metric threshold is a first metric threshold;

and wherein the method further comprises using the machine learning component to determine a second metric threshold for triggering performance of the first action to reduce the likelihood of failure of the active functional component based at least in part on the second and the third values of the metric and the first metric threshold.

19. The method of claim 1, wherein performance of the first action occurs at a time that at least partially overlaps a time when the second action is performed.

20. The method of claim 1, wherein the second action is caused to be performed based, at least in part, on the determining that the active functional component has again reached the likelihood of failure based at least in part on the second value of the metric exceeding the metric threshold.

9. A method comprising:

determining that an active functional component of a computer system has reached a likelihood of failure based at least in part on a first value of a first metric exceeding a first metric threshold;

wherein the first value of the first metric reflects performance of the active functional component at a time before a first action is caused to be performed a first time in an attempt to reduce the likelihood of failure of the active functional component;

causing the first action to be performed the first time in an attempt to reduce the likelihood of failure of the active functional component;

after causing the first action to be performed the first time, determining that the active functional component has again reached the likelihood of failure based at least in part on a second value of a second metric exceeding a second metric threshold that is different than the first metric threshold;

wherein the second value of the second metric reflects performance of the active functional component at a time after the first action is caused to be performed the first time;

causing the first action to be performed a second time in an attempt to reduce the likelihood of failure of the active functional component;

after causing the first action to be performed the second time, obtaining a third value for a third metric reflecting performance of the active functional component at a time after the first action is caused to be performed the second time; and

using a machine learning component to determine a third metric threshold for triggering the first action to be performed a third time to reduce the likelihood of failure of the active functional component based at least in part on the second value of the second metric, the third value of the third metric, and the first and second metric thresholds; and

wherein the method is performed by one or more computing devices.

10. The method of claim 9, further comprising:

in response to detecting a value for a metric exceeding the third metric threshold, causing the first action to be performed the third time in an attempt to reduce the likelihood of failure of the active functional component.

11. The method of claim 9, wherein the second and third metrics are the same metric.

12. The method of claim 9, wherein registration information for the active functional component comprises a negative association with a non-critical functional component;

and wherein the first action to be performed the third time comprises stopping the non-critical functional component.

13. The method of claim 9,

wherein registration information for the active functional component comprises a positive association with a functional component on which the active functional component depends;

and

wherein the first action to be performed the third time comprises creating a new instance of the functional component on which the active functional component depends.

14. The method of claim 9, further comprising:

determining a first weight indicating an effectiveness of performing the first action, at the first time prior to the second time, at reducing the likelihood of failure of the active functional component;

determining a second weight indicating an effectiveness of performing the first action, at the second time, at reducing the likelihood of failure of the active functional component; and

using a machine learning component to determine the third metric threshold for triggering the first action to be performed the third time to reduce the likelihood of failure of the active functional component based at least in part on the first and second weights.

15. The method of claim 9, further comprising:

using the machine learning component to determine the third metric threshold for triggering the first action to be performed the third time to reduce the likelihood of failure of the active functional component based at least in part on a value reflecting an impact that performing the first action has on performance of the computer system.

16. The method of claim 9, further comprising:

using the machine learning component to determine a second action, that is different from the first action, to be performed to reduce the likelihood of failure of the active functional component based at least in part on the second value of the second metric, the third value of the third metric, and the first action caused to be performed.

17. One or more non-transitory computer readable media storing instructions which, when executed by one or more processors cause performance of:

determining that an active functional component of a computer system has reached a likelihood of failure based at least in part on a first value of a metric exceeding a metric threshold;

wherein the first value of the metric reflects performance of the active functional component at a time before a first action is caused to be performed in an attempt to reduce the likelihood of failure of the active functional component;

causing the first action to be performed in an attempt to reduce the likelihood of failure of the active functional component;

after causing the first action to be performed, determining that the active functional component has again reached the likelihood of failure based at least in part on a second value of the metric exceeding the metric threshold;

wherein the second value of the metric reflects performance of the active functional component at a time after the first action is caused to be performed;

causing a second action, that is different from the first action, to be performed in an attempt to reduce the likelihood of failure of the active functional component;

after causing the second action to be performed, obtaining a third value for the metric reflecting performance of the active functional component at a time after the second action is caused to be performed; and

using a machine learning component to determine a third action to be performed to reduce the likelihood of failure of the active functional component based at least in part on the second and the third values of the metric and the first and second actions caused to be performed.

18. One or more non-transitory computer readable media storing instructions which, when executed by one or more processors cause performance of:

determining that an active functional component of a computer system has reached a likelihood of failure based at least in part on a first value of a first metric exceeding a first metric threshold;

wherein the first value of the first metric reflects performance of the active functional component at a time before a first action is caused to be performed a first time in an attempt to reduce the likelihood of failure of the active functional component;

causing the first action to be performed the first time in an attempt to reduce the likelihood of failure of the active functional component;

after causing the first action to be performed the first time, determining that the active functional component has again reached the likelihood of failure based at least in part on a second value of a second metric exceeding a second metric threshold that is different than the first metric threshold;

wherein the second value of the second metric reflects performance of the active functional component at a time after the first action is caused to be performed the first time;

causing the first action to be performed a second time in an attempt to reduce the likelihood of failure of the active functional component;

after causing the first action to be performed the second time, obtaining a third value for a third metric reflecting performance of the active functional component at a time after the first action is caused to be performed the second time; and

using a machine learning component to determine a third metric threshold for triggering the first action to be performed a third time to reduce the likelihood of failure of the active functional component based at least in part on the second value of the second metric, the third value of the third metric, and the first and second metric thresholds.