Method and system for parameterizing log file location assignments for a log analytics system

• US 10,585,908 B2
• Filed: 04/01/2016
• Issued: 03/10/2020
• Est. Priority Date: 04/03/2015
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Russell, Jerry Paul
• Primary Examiner(s): Kim, Taelor
• Assistant Examiner(s): Ennaji, Lahcen
• Application Number:
  US15/089,005
• Publication Number:
  US 20160292166A1
• Time in Patent Office: 1,439 Days
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3086 where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/84 Mapping; Conversion
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 40/16 Automatic learning of transformation rules, e.g. from examples
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/5074 Handling of user complaints or trouble tickets
• CPC Class Codes: H04L 43/04 : Processing captured monitoring data, e.g. for logfile generation

Abstract

Disclosed is a system, method, and computer program product for implementing a log analytics method and system that can configure, collect, and analyze log records in an efficient manner. The log analytics system, method, and computer program product provide target-based configuration of log monitoring metadata. Variable pathnames (e.g., having variables instead of absolute pathnames) may be specified to identify a directory that contains log files to monitor.

Claims

1. A method comprising:

storing a log collection configuration associated with a plurality of log collection targets, the log collection configuration comprising;

a first included file name pattern, comprising (a) a first fixed portion and (b) a first variable portion, corresponding to a first set of one or more log files to include in log collection;

a first excluded file name pattern, comprising (a) a second fixed portion and (c) a second variable portion, corresponding to a second set of one or more log files to exclude from the log collection;

based at least on the log collection configuration;

replacing the first variable portion, in the first included file name pattern, with a first value associated with a particular log collection target in the plurality of log collection targets, to obtain a second included file name pattern;

replacing the second variable portion, in the first excluded file name pattern, with a second value associated with the particular log collection target, to obtain a second excluded file name pattern;

wherein the second set of one or more log files stored at a second set of one or more file locations that match the second excluded file name pattern are removed from the first set of one or more log files stored at a first set of one or more file locations that match the second included file name pattern to identify a third set of one or more log files to be collected;

transmitting to or storing on the particular log collection target, information including the second included file name pattern and the second excluded file name pattern for collecting the third set of one or more log files.

2. The method of claim 1, wherein the first included file name pattern is overridden on a per-target basis using a mapping table that maps an association to a target.

3. The method of claim 1, wherein a target property is established having metadata to define the first variable portion of the first included file name pattern.

4. The method of claim 1, wherein the first variable portion of the first included file name pattern is resolved based upon a specific host for a target.

5. The method of claim 1, wherein a set of fixed selectable parameters is configured for the first variable portion of the first included file name pattern.

6. The method of claim 1, wherein a rule defines a set of sources for the particular log collection target, along with conditions and actions to be triggered during monitoring of the particular log collection target.

19. The method of claim 1, wherein:

the log collection configuration is associated with a target type;

replacing the first variable portion of the pathname with the first value associated with the particular log collection target and replacing the second variable portion with the second value associated with the particular log collection target are responsive at least to determining that the target is associated with the same target type.

20. The method of claim 1, wherein:

replacing the first variable portion of the pathname with the first value associated with the particular log collection target and replacing the second variable portion with the second value associated with the particular log collection target are responsive at least to detecting an installation of an agent configured to perform the log collection for the target.

21. The method of claim 1, wherein the second included file name pattern is a complete pathname.

7. One or more non-transitory machine-readable media storing instructions which, when executed by one or more processors, cause:

storing a log collection configuration associated with a plurality of log collection targets, the log collection configuration comprising;

a first included file name pattern, comprising (a) a first fixed portion and (b) a first variable portion, corresponding to a first set of one or more log files to include in log collection;

a first excluded file name pattern, comprising (a) a second fixed portion and (c) a second variable portion, corresponding to a second set of one or more log files to exclude from the log collection;

based at least on the log collection configuration;

replacing the first variable portion, in the first included file name pattern, with a first value associated with a particular log collection target in the plurality of log collection targets, to obtain a second included file name pattern;

replacing the second variable portion, in the first excluded file name pattern, with a second value associated with the particular log collection target, to obtain a second excluded file name pattern;

wherein the second set of one or more log files stored at a second set of one or more file locations that match the second excluded file name pattern are removed from the first set of one or more log files stored at a first set of one or more file locations that match the second included file name pattern to identify a third set of one or more log files to be collected;

transmitting to or storing on the particular log collection target, information including the second included file name pattern and the second excluded file name pattern for collecting the third set of one or more log files.

8. The one or more media of claim 7, wherein the first included file name pattern is overridden on a per-target basis using a mapping table that maps an association to a target.

9. The one or more media of claim 7, wherein a target property is established having metadata to define the first variable portion of the first included file name pattern.

10. The one or more media of claim 7, wherein the first variable portion of the first included file name pattern is resolved based upon a specific host for a target.

11. The computer readable medium of claim 7, wherein a set of fixed selectable parameters is configured for the first variable portion of the first included file name pattern.

12. The one or more media of claim 7, wherein a rule defines a set of sources for the particular log collection target, along with conditions and actions to be triggered during monitoring of the particular log collection target.

13. A system, comprising:

at least one device including a hardware processor;

the system being configured to perform operations comprising;

storing a log collection configuration associated with a plurality of log collection targets, the log collection configuration comprising;

a first included file name pattern, comprising (a) a first fixed portion and (b) a first variable portion, corresponding to a first set of one or more log files to include in log collection;

a first excluded file name pattern, comprising (a) a second fixed portion and (c) a second variable portion, corresponding to a second set of one or more log files to exclude from the log collection;

based at least on the log collection configuration;

replacing the first variable portion, in the first included file name pattern, with a first value associated with a particular log collection target in the plurality of log collection targets, to obtain a second included file name pattern;

replacing the second variable portion, in the first excluded file name pattern, with a second value associated with the particular log collection target, to obtain a second excluded file name pattern;

wherein the second set of one or more log files stored at a second set of one or more file locations that match the second excluded file name pattern are removed from the first set of one or more log files stored at a first set of one or more file locations that match the second included file name pattern to identify a third set of one or more log files to be collected;

transmitting to or storing on the particular log collection target, information including the second included file name pattern and the second excluded file name pattern for collecting the third set of one or more log files.

14. The system of claim 13, wherein the first included file name pattern is overridden on a per-target basis using a mapping table that maps an association to a target.

15. The system of claim 13, wherein a target property is established having metadata to define the first variable portion of the first included file name pattern.

16. The system of claim 13, wherein the first variable portion of the first included file name pattern is resolved based upon a specific host for a target.

17. The system of claim 13, wherein a set of fixed selectable parameters is configured for the first variable portion of the first included file name pattern.

18. The system of claim 13, wherein a rule defines a set of sources for the particular log collection target, along with conditions and actions to be triggered during monitoring of the particular log collection target.