Fast detection and diagnosis of system outages

• US 8,990,621 B2
• Filed: 08/29/2012
• Issued: 03/24/2015
• Est. Priority Date: 06/28/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gupta, Manish, Kothari, Ravi, Mann, Vijay, Vishnoi, Anil Kumar
• Primary Examiner(s): DUNCAN, MARC M
• Application Number:
  US13/598,314
• Publication Number:
  US 20130061092A1
• Time in Patent Office: 937 Days
• Field of Search: 714/26, 714/47.1, 714/46, 714/57
• US Class Current: 714/26
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3082 the data filtering being achieved by aggregating or compressing the monitored data
• CPC Class Codes: G06F 11/3093 Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
• CPC Class Codes: G06F 11/327 Alarm or error message display

Abstract

Methods and arrangements for detecting and diagnosing system outages. A system outage is ascertained and aberrant user activities are categorized, as possible contributors to the system outage, based on system impact. User activities and system impact are learned, and user activities and system impact are compared against predetermined rules. A system outage alert is generated, and a user activity responsible for the system outage is displayed.

Claims

1. A method comprising:

ascertaining a system outage;

categorizing aberrant user activities, as possible contributors to the system outage, based on system impact in terms of at least one member taken from the group consisting of;

process forking and closure activities, file system activities, network activities, memory activities, user activity parameter values;

learning user activities and system impact;

comparing user activities and system impact against predetermined rules;

generating a system outage alert;

displaying a user activity responsible for the system outage; and

measuring system impact of user activities via employing data collector agents;

wherein said measuring further comprises aggregating data collected by the data collector agents based on process hierarchy.

2. The method according to claim 1, wherein said ascertaining comprises:

detecting the start and end of a change window; and

comparing process and network snapshots at the start and end of a change window to assimilate information on a transitional event taking place during the change window.

3. The method according to claim 2, wherein:

said detecting comprises detecting the start and end of a change window via at least one taken from the group consisting of;

reading a file, detecting a fork of a remote shell process, detecting a death of a remote shell process;

said comparing comprises comparing the transitional event information with a predetermined list of essential processes and ports;

the transitional event comprises at least one taken from the group consisting of;

death of a process, closure of a network port, closure of a network connection, deletion of a file, modification of a file; and

the predetermined list comprises at least one taken from the group consisting of;

a list created from policy documents, a list created from a firewall policy.

4. The method according to claim 1, wherein the predetermined rules comprise at least one taken from the group consisting of:

learnt rules, static rules.

5. The method according to claim 4, wherein:

the learnt rules comprises rules learnt through association of a user activity and its side effects over a period of time; and

the static rules comprise rules retrieved from at least one of;

a policy document, a firewall policy.

6. The method according to claim 1, wherein said comprises displaying and highlighting on a dashboard a user activity responsible for the system outage.

7. The method according to claim 1, further comprising measuring system impact of a user activity based on at least one taken from the group consisting of:

files modified, files deleted, network ports opened, network ports closed, network connections created, network connections closed, new processes forked, processes killed, in-memory data structures modified.

8. The method according to claim 1, further comprising measuring system impact of a user activity based on supplied parameter values.

9. The method according to claim 1, further comprising sending the aggregated data to at least one central server for processing.