Automatic log sensor tuning

• US 9,507,847 B2
• Filed: 09/27/2013
• Issued: 11/29/2016
• Est. Priority Date: 09/27/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Beisiegel, Michael, Joseph, Dinakaran, Nadgir, Devaprasad K.
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US14/038,918
• Publication Number:
  US 20150095332A1
• Time in Patent Office: 1,159 Days
• Field of Search: 707/758, 709/224, 370/253, 702/127
• US Class Current: 1/1
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04Q 2209/823 where the data is sent when the measured values exceed a threshold, e.g. sending an alarm
• CPC Class Codes: H04Q 9/00 Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom

Abstract

A process for automatic tuning a set of collectors and/or sensors includes: collecting first machine data by a first sensor in a collection framework, processing the first machine data by a first collector in the collection framework to yield first collected machine data, performing analytics on the first collected machine data to generate analytics output, and tuning, based, at least in part, on the analytics output, at least one of the following: the first sensor and the first collector.

Claims

1. A computer program product comprising software stored on a computer-readable storage medium, the software comprising:

first program instructions programmed to identify first machine data from a first enterprise component by a first sensor in a collection framework, the first sensor tuned to a first setting for identifying the first machine data;

second program instructions programmed to process at least a portion of the first machine data by a first collector in the collection framework, having a first configuration, to generate first collected machine data, the first collector configured to distribute the first collected machine data to a search cluster;

third program instructions programmed to, responsive to an alert condition, generate a piped HTTP request for performing analytics on a set of collected machine data in the search cluster including the first collected machine data;

fourth program instructions programmed to receive a single threaded piped HTTP response to the piped HTTP request as analytics output;

fifth program instructions programmed to determine a second configuration for the first collector responsive to the analytics output;

sixth program instructions programmed to execute a sync instruction to the first collector to replace the first configuration of the first collector with the second configuration;

seventh program instructions programmed to receive a second machine data from the first sensor; and

eighth program instructions programmed to process at least a portion of the second machine data according to the second configuration of the first collector to generate second collected machine data;

wherein;

the second collected machine data includes event-specific data determined to be relevant by the performing analytics on the set of collected machine data.

2. The computer program product of claim 1 wherein:

the performance of analytics occurs in real time; and

the receiving a single threaded piped HTTP response occurs in real time.

3. The computer program product of claim 1 further comprising:

ninth program instructions programmed to determine an alert condition is met;

wherein;

the performance of analytics is performed automatically in response to the alert condition being met; and

the alert condition is a threshold stored memory size of the search cluster.

7. The computer program product of claim 1, further comprising:

ninth program instructions programmed to, responsive to a completion of a predetermined time period during which the first collector operates according to the second configuration, returning the first collector to the first configuration to generate the first collected machine data.

9. The computer program product of claim 1, further comprising:

causing the first sensor to identify the second machine data from at least a second enterprise component having an effect of adjusting a scope of data capture corresponding to identifying the second machine data by the first sensor.

4. A computer system comprising:

a processor set; and

a software storage device;

wherein;

the processor set is structured, located, connected and/or programmed to run software stored on the software storage device; and

the software comprises;

first program instructions programmed to identify first machine data from a first enterprise component by a first sensor in a collection framework, the first sensor tuned to a first setting for identifying the first machine data;

second program instructions programmed to process at least a portion of the first machine data by a first collector in the collection framework to generate first collected machine data, the first collector configured to distribute the first collected machine data to a search cluster;

third program instructions programmed to, responsive to an alert condition, generate a piped HTTP request for performing analytics on a set of collected machine data in the search cluster including the first collected machine data;

fourth program instructions programmed to receive a single threaded piped HTTP response to the piped HTTP request as analytics output;

fifth program instructions programmed to determine a second configuration for the first collector responsive to the analytics output;

sixth program instructions programmed to execute a sync instruction to the first collector to replace the first configuration of the first collector with the second configuration;

seventh program instructions programmed to receive a second machine data from the first sensor; and

eighth program instructions programmed to process at least a portion of the second machine data according to the second configuration of the first collector to generate a second collected machine data;

wherein;

the second collected machine data includes event-specific data determined to be relevant by the performing analytics on the set of collected machine data.

5. The computer system of claim 4 wherein:

the performance of analytics occurs in real time; and

the receiving a single threaded piped HTTP response occurs in real time.

6. The computer system of claim 4 further comprising:

ninth program instructions programmed to determine an alert condition is met;

wherein;

the performance of analytics is performed automatically in response to the alert condition being met; and

the alert condition is a threshold stored memory size of the search cluster.

8. The computer system of claim 4, further comprising:

ninth program instructions programmed to, responsive to a completion of a predetermined time period during which the first collector operates according to the second configuration, returning the first collector to the first configuration to generate the first collected machine data.

10. The computer system of claim 4, further comprising:

causing the first sensor to identify the second machine data from at least a second enterprise component having an effect of adjusting a scope of data capture corresponding to identifying the second machine data by the first sensor.