System and method for statistical application-agnostic fault detection

• US 8,612,802 B1
• Filed: 01/31/2011
• Issued: 12/17/2013
• Est. Priority Date: 01/31/2011
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: Open Invention Network LLC
• Inventors: Havemose, Allan
• Primary Examiner(s): Iqbal, Nadeem
• Application Number:
  US13/017,340
• Time in Patent Office: 1,051 Days
• Field of Search: 714 25- 30, 714/37, 714/38.2, 714/38.13, 714/39, 714/45, 714/46, 714/47.1, 714/48
• US Class Current: 714/26
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3608 using formal methods, e.g. model checking, abstract interpretation theorem proving G06N5/013
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

A system, method, and computer readable medium for statistical application-agnostic fault detection of multi-process applications. The computer readable medium includes computer-executable instructions for execution by a processing system. A multi-process application runs on a host. Interceptors collect statistical events and sends said events to a statistical fault detector. The statistical fault detector creates one or more distributions and compares recent statistical event data to historical statistical event data and uses deviation from historical norm for fault detection. The present invention detects faults both within the application and within the environment wherein the application executes, if conditions within the environment cause impaired application performance. The invention also teaches consensus fault detection and elimination of cascading fault notifications based on a hierarchy of events and event groups. Interception and fault detection is transparent to the application, operating system, networking stack and libraries.

Claims

1. A system for providing statistical fault detection for multi-process applications, the system comprising:

one or more memory locations configured to store said applications executing on a host with a host operating system;

one or more interceptors configured to intercept calls to the host operating system and shared libraries, and configured to generate one or more statistical events based on said intercepted calls;

a statistical fault detector configured to calculate one or more distributions for said one or more statistical events;

one or more additional memory locations configured to store the one or more statistical distributions for each one or more statistical events;

wherein fault detection for said applications is performed by detection of statistically significant deviation of recent events from the corresponding one or more distributions;

wherein the said one or more distributions is one or more of Raw Event Distribution, Temporal Event Distribution, or Spatial Event Distribution;

wherein said Temporal Event Distribution is comprised of a subset of events from the corresponding interceptor subject to a temporal filter; and

wherein said Spatial Event Distribution is comprised of a subset of events from the corresponding interceptor subject to a spatial filter.

2. The system according to claim 1, wherein said operating system is one of Linux, UNIX or Windows;

and

wherein each distribution corresponding to a temporal filter is calculated independently of all other distributions for said temporal filter.

3. The system according to claim 2, wherein said one or more distributions are created by using a number of events that is pre-defined or determined dynamically based on one or more of frequency of data, dynamic data range, minimum number of events for statistical significance, or other measurement.

4. The system according to claim 2, wherein said statistical significant deviation is determined by one or more of a confidence interval of mean plus and minus a factor times the standard deviation, or a custom confidence interval defined by removal of a specific number of extreme events in the distribution and by defining the remaining event data to comprise the lower to upper bounds of the confidence interval.

5. The system according to claim 4, wherein said specific number of events is one of a pre-defined number of events, or a pre-defined percentage of all events.

6. The system according to claim 1, wherein the said one or more distributions is one or more of Raw Event Distribution (RED), Temporal Event Distribution (TED), or Spatial Event Distribution (SED).

7. The system according to claim 6, wherein said Raw Event Distribution is comprised of events from the corresponding interceptor.

8. The system according to claim 6, wherein said Temporal Event Distribution is comprised of a subset of events from the corresponding interceptor subject to a temporal filter.

9. The system according to claim 8, wherein said temporal filter is one or more of time of day, time of day with day of week, day of week, day of month, minutes since midnight, hours since midnight, or other temporal filter.

10. The system according to claim 6, wherein said Spatial Event Distribution is comprised of a subset of events from the corresponding interceptor subject to a spatial filter.

11. The system according to claim 10, wherein said spatial filter is one or more of system name, subnet, transport, storage subsystem, or other system-level characteristics.

12. The system according to claim 1, wherein said distributions are re-calculated upon arrival of new events by removal of old events, addition of new events, and recalculation of the distribution based the new underlying event data.

13. The system according to claim 1, wherein statistically significant deviation of recent events is defined to mean one or more of the number of events are greater than the average number of events in said distribution, the number of events are greater than the average number of events in said distribution plus a factor times the standard deviation of said number of events, or other statistical measure.

14. The system according to claim 1, wherein said applications are considered fault-free if there are no statistically significant deviations of recent events from the underlying distributions.

15. The system according to claim 1, wherein the events are assumed to follow a Gaussian distribution and statistically significant deviation is determined using mean and standard deviation from the Gaussian distribution.

16. The system according to claim 8, wherein each distribution corresponding to a temporal filter is calculated independently of all other distributions for said temporal filter.

17. The system according to claim 8, wherein each distribution corresponding to a temporal filter has its own distribution variability, statistical significance, and fault detection levels.

18. The system according to claim 10, wherein each distribution in the spatial event distribution is comprised of said distribution as calculated on the corresponding host, or as associated with the corresponding subnet, transport, storage subsystem, or other system-level characteristics.

19. The system according to claim 10, wherein each distribution in said spatial event distribution is calculated independently of all other distributions for said spatial filter.

20. The system according to claim 10, wherein each distribution in said spatial distribution has its own distribution variability, statistical significance, and fault detection levels.