Automatic log sensor tuning

• US 10,169,443 B2
• Filed: 07/18/2016
• Issued: 01/01/2019
• Est. Priority Date: 09/27/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Beisiegel, Michael, Joseph, Dinakaran, Nadgir, Devaprasad K.
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US15/212,598
• Publication Number:
  US 20160330068A1
• Time in Patent Office: 897 Days
• Field of Search: 707648, 707758, 709224, 714 45, 714 48, 715736
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04Q 2209/823 where the data is sent when the measured values exceed a threshold, e.g. sending an alarm
• CPC Class Codes: H04Q 9/00 Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom

Abstract

A process for automatic tuning a set of collectors and/or sensors includes: collecting first machine data by a first sensor in a collection framework, processing the first machine data by a first collector in the collection framework to yield first collected machine data, performing analytics on the first collected machine data to generate analytics output, and tuning, based, at least in part, on the analytics output, at least one of the following: the first sensor and the first collector.

Claims

1. A method comprising:

responsive to an occurrence of an alert condition, generating a piped HTTP request for performing analytics on a first set of machine data in a search cluster, the first set of machine data collected by a collector according to a first configuration of the collector, the first configuration establishing default data collection levels;

receiving a single-threaded, piped HTTP response to the piped HTTP request as analytics output;

determining a second configuration for the collector to collect a second set of machine data responsive to the analytics output, the second configuration establishing debug data collection levels, wherein the debug data collection level causes a list of events to be monitored for data collection in addition to actions of the default data collection levels;

executing a sync instruction to the collector to replace the first configuration with the second configuration; and

causing the collector to collect a second set of machine data by processing new machine data according to the second configuration, the new machine data generated after the occurrence of the alert condition;

wherein;

the second set of machine data includes event-specific data determined to be relevant by the performing analytics on the first set of machine data.

2. The method of claim 1, further comprising:

causing a sensor to record first machine data from a first enterprise component, the first machine data being processed by the collector to collect the first set of machine data.

3. The method of claim 2 further comprising:

responsive to the analytics output, adjusting a scope of data capture corresponding to recording machine data by causing the sensor to record the new machine data from the first enterprise component and a second enterprise component.

4. The method of claim 2, further comprising:

causing the collector to collect the first set of machine data by processing a portion of the recorded machine data according to the first configuration, the first collector programmed to distribute the first set of machine data to the search cluster.

5. The method of claim 1, wherein:

the analytics output is produced by performing analytics on the first set of machine data in real time; and

receiving the single-threaded, piped HTTP response occurs in real time.

6. The method of claim 1, further comprising:

determining the occurrence of the alert condition; and

wherein;

the alert condition occurs when a threshold stored memory size of the search cluster is reached.

7. The method of claim 1, further comprising:

responsive to a completion of a pre-determined time period during which the collector operates according to the second configuration, returning the collector to the first configuration.

8. A computer program product comprising a computer-readable storage medium having a set of instructions stored therein which, when executed by a processor, causes the processor to perform a tuning action by:

responsive to an occurrence of an alert condition, generating a piped HTTP request for performing analytics on a first set of machine data in a search cluster, the first set of machine data collected by a collector according to a first configuration of the collector, the first configuration establishing default data collection levels;

receiving a single-threaded, piped HTTP response to the piped HTTP request as analytics output;

determining a second configuration for the collector to collect a second set of machine data responsive to the analytics output, the second configuration establishing debug data collection levels, wherein the debug data collection level causes a list of events to be monitored for data collection in addition to actions of the default data collection levels;

executing a sync instruction to the collector to replace the first configuration with the second configuration; and

causing the collector to collect a second set of machine data by processing new machine data according to the second configuration, the new machine data generated after the occurrence of the alert condition;

wherein;

the second set of machine data includes event-specific data determined to be relevant by the performing analytics on the first set of machine data.

9. The computer program product of claim 8, further causing the processor to perform a tuning action by:

causing a sensor to record first machine data from a first enterprise component, the first machine data being processed by the collector to collect the first set of machine data.

10. The computer program product of claim 9, further causing the processor to perform a tuning action by:

responsive to the analytics output, adjusting a scope of data capture corresponding to recording machine data by causing the sensor to record the new machine data from the first enterprise component and a second enterprise component.

11. The computer program product of claim 9, further causing the processor to perform a tuning action by:

causing the collector to collect the first set of machine data by processing a portion of the recorded machine data according to the first configuration, the first collector programmed to distribute the first set of machine data to the search cluster.

12. The computer program product of claim 8, wherein:

the analytics output is produced by performing analytics on the first set of machine data in real time; and

receiving the single-threaded, piped HTTP response occurs in real time.

13. The computer program product of claim 8, further causing the processor to perform a tuning action by:

determining the occurrence of the alert condition; and

wherein;

the alert condition occurs when a threshold stored memory size of the search cluster is reached.

14. The computer program product of claim 8, further causing the processor to perform a tuning action by:

responsive to a completion of a pre-determined time period during which the collector operates according to the second configuration, returning the collector to the first configuration.

15. A computer system comprising:

a processor set; and

a computer readable storage medium;

wherein;

the processor set is structured, located, connected, and/or programmed to run program instructions stored on the computer readable storage medium; and

the program instructions which, when executed by the processor set, cause the processor set to perform a tuning action by;

responsive to an occurrence of an alert condition, generating a piped HTTP request for performing analytics on a first set of machine data in a search cluster, the first set of machine data collected by a collector according to a first configuration of the collector, the first configuration establishing default data collection levels;

receiving a single-threaded, piped HTTP response to the piped HTTP request as analytics output;

determining a second configuration for the collector to collect a second set of machine data responsive to the analytics output, the second configuration establishing debug data collection levels, wherein the debug data collection level causes a list of events to be monitored for data collection in addition to actions of the default data collection levels;

executing a sync instruction to the collector to replace the first configuration with the second configuration; and

causing the collector to collect a second set of machine data by processing new machine data according to the second configuration, the new machine data generated after the occurrence of the alert condition;

wherein;

the second set of machine data includes event-specific data determined to be relevant by the performing analytics on the first set of machine data.

16. The computer system of claim 15, further causing the processor set to perform a tuning action by:

causing a sensor to record first machine data from a first enterprise component, the first machine data being processed by the collector to collect the first set of machine data.

17. The computer system of claim 16, further causing the processor set to perform a tuning action by:

responsive to the analytics output, adjusting a scope of data capture corresponding to recording machine data by causing the sensor to record the new machine data from the first enterprise component and a second enterprise component.

18. The computer system of claim 16, further causing the processor set to perform a tuning action by:

causing the collector to collect the first set of machine data by processing a portion of the recorded machine data according to the first configuration, the first collector programmed to distribute the first set of machine data to the search cluster.

19. The computer system of claim 15, further causing the processor set to perform a tuning action by:

the analytics output is produced by performing analytics on the first set of machine data in real time; and

receiving the single-threaded, piped HTTP response occurs in real time.

20. The computer system of claim 15, further causing the processor set to perform a tuning action by:

determining the occurrence of the alert condition;

wherein;

the alert condition occurs when a threshold stored memory size of the search cluster is reached.