Method and system for implementing target model configuration metadata for a log analytics system

• US 10,592,521 B2
• Filed: 04/01/2016
• Issued: 03/17/2020
• Est. Priority Date: 04/03/2015
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Russell, Jerry Paul, Zhu, Shushuai
• Primary Examiner(s): Louie, Oscar A
• Assistant Examiner(s): Gidado, Oluwatosin M
• Application Number:
  US15/088,943
• Publication Number:
  US 20160294649A1
• Time in Patent Office: 1,446 Days
• Field of Search: 709220
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3086 where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/84 Mapping; Conversion
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 40/16 Automatic learning of transformation rules, e.g. from examples
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/5074 Handling of user complaints or trouble tickets
• CPC Class Codes: H04L 43/04 : Processing captured monitoring data, e.g. for logfile generation

Abstract

Disclosed is a system, method, and computer program product for implementing a log analytics method and system that can configure, collect, and analyze log records in an efficient manner. The log analytics system, method, and computer program product provide target-based configuration of log monitoring metadata. The log analytics system, method, and computer program product can be implemented in a cloud-based/SaaS-based architecture.

Claims

1. A method comprising:

receiving, via a log configuration interface of a log analytics system configured to perform log analytics for a plurality of targets in a network, (a) first user input indicating a target type, and (b) second user input indicating a log source;

wherein the log source represents a location of log data and comprises a file pattern including a fixed part and a variable part;

generating, by the log analytics system based on the first user input and the second user input, a log collection configuration that indicates the target type and the log source;

responsive to determining that a first target, of the plurality of targets, is associated with the target type;

associating the log collection configuration with the first target;

responsive to associating the log collection configuration with the first target;

replacing the variable part of the file pattern, associated with the log collection configuration, with first metadata associated with a first target property of the first target to generate a first file path indicating a first location of first log data associated with the first target;

generating first target-side configuration content comprising the first file path;

responsive to determining that a second target, of the plurality of targets, is associated with the target type;

associating the log collection configuration with the second target;

responsive to associating the log collection configuration with the second target;

replacing the variable part of the file pattern, associated with the log collection configuration, with second metadata associated with a second target property of the second target to generate a second file path indicating a second location of second log data associated with the second target;

generating second target-side configuration content comprising the second file path;

distributing, by the log analytics system, the first target-side configuration content to a first agent associated with the first target and the second target-side configuration content to a second agent associated with the second target;

receiving, by the log analytics system from the first agent, the first log data captured for the first target at the first location indicated by the first file path;

receiving, by the log analytics system from the second agent, the second log data captured for the second target at the second location indicated by the second file path; and

storing, by the log analytics system, the first log data and the second log data

wherein the method is performed by at least one device including a hardware processor.

2. The method of claim 1, further comprising:

configuring log monitoring configuration metadata, wherein the log monitoring configuration metadata comprises at least one of a base parser, a log rule applicable to a particular target of the target type, and an identification of the particular log target.

3. The method of claim 1, further comprising:

identifying a parser for the log source, wherein the parser comprises a regular expression to define how to parse the log data for a particular target of the target type.

4. The method of claim 1, wherein the log collection configuration comprises a first portion applicable to the first target and a second portion applicable to the second target.

5. The method of claim 1, wherein the log collection configuration corresponds to an XML file.

6. The method of claim 1, wherein the log collection configuration is generated, modified, or deleted upon a change to an association between the first target and the log source, a change to the first target, or a change to a status of the first agent.

7. The method of claim 1, further comprising:

periodically synchronizing associations between the log collection configuration and the plurality of targets.

8. The method of claim 1, wherein the first target property of the first target comprises a first host of the first target and the second target property of the second target comprises a second host of the second target.

9. One or more machine readable media storing instructions which, when executed by one or more processors, cause:

receiving, via a log configuration interface of a log analytics system configured to perform log analytics for a plurality of targets in a network, (a) first user input indicating a target type, and (b) second user input indicating a log source;

wherein the log source represents a location of log data and comprises a file pattern including a fixed part and a variable part;

generating, by the log analytics system based on the first user input and the second user input, a log collection configuration that indicates the target type and the log source;

responsive to determining that a first target, of the plurality of targets, is associated with the target type;

associating the log collection configuration with the first target;

responsive to associating the log collection configuration with the first target;

replacing the variable part of the file pattern, associated with the log collection configuration, with first metadata associated with a first target property of the first target to generate a first file path indicating a first location of first log data associated with the first target;

generating first target-side configuration content comprising the first file path;

responsive to determining that a second target, of the plurality of targets, is associated with the target type;

associating the log collection configuration with the second target;

responsive to associating the log collection configuration with the second target;

replacing the variable part of the file pattern, associated with the log collection configuration, with second metadata associated with a second target property of the second target to generate a second file path indicating a second location of second log data associated with the second target;

generating second target-side configuration content comprising the second file path;

distributing, by the log analytics system, the first target-side configuration content to a first agent associated with the first target and the second target-side configuration content to a second agent associated with the second target;

receiving, by the log analytics system from the first agent, the first log data captured for the first target at the first location indicated by the first file path;

receiving, by the log analytics system from the second agent, the second log data captured for the second target at the second location indicated by the second file path; and

storing, by the log analytics system, the first log data and the second log data.

10. The one or more media of claim 9, further storing instructions which, when executed by one or more processors, cause:

configuring log monitoring configuration metadata, wherein the log monitoring configuration metadata comprises at least one of a base parser, a log rule applicable to a particular target of the target type, and an identification of the particular log target.

11. The one or more media of claim 9, further storing instructions which, when executed by one or more processors, cause:

identifying a parser for the log source, wherein the parser comprises a regular expression to define how to parse the log data for a particular target of the target type.

12. The one or more media of claim 9, wherein the log collection configuration comprises a first portion applicable to the first target and a second portion applicable to the second target.

13. The one or more media of claim 9, wherein the log collection configuration corresponds to an XML file.

14. The one or more media of claim 9, wherein the log collection configuration is generated, modified, or deleted upon a change to an association between the first target and the log source, a change to the first target, or a change to a status of the first agent.

15. The one or more media of claim 9, further storing instructions which, when executed by one or more processors, cause:

periodically synchronizing associations between the log collection configuration and the plurality of targets.

16. A system comprising:

at least one device including a hardware processor;

the system being configured to perform operations comprising;

receiving, via a log configuration interface of a log analytics system configured to perform log analytics for a plurality of targets in a network, (a) first user input indicating a target type, and (b) second user input indicating a log source;

wherein the log source represents a location of log data and comprises a file pattern including a fixed part and a variable part;

generating, by the log analytics system based on the first user input and the second user input, a log collection configuration that indicates the target type and the log source;

responsive to determining that a first target, of the plurality of targets, is associated with the target type;

associating the log collection configuration with the first target;

responsive to associating the log collection configuration with the first target;

replacing the variable part of the file pattern, associated with the log collection configuration, with first metadata associated with a first target property of the first target to generate a first file path indicating a first location of first log data associated with the first target;

generating first target-side configuration content comprising the first file path;

responsive to determining that a second target, of the plurality of targets, is associated with the target type;

associating the log collection configuration with the second target;

responsive to associating the log collection configuration with the second target;

replacing the variable part of the file pattern, associated with the log collection configuration, with second metadata associated with a second target property of the second target to generate a second file path indicating a second location of second log data associated with the second target;

generating second target-side configuration content comprising the second file path;

distributing, by the log analytics system, the first target-side configuration content to a first agent associated with the first target and the second target-side configuration content to a second agent associated with the second target;

receiving, by the log analytics system from the first agent, the first log data captured for the first target at the first location indicated by the first file path;

receiving, by the log analytics system from the second agent, the second log data captured for the second target at the second location indicated by the second file path; and

storing, by the log analytics system, the first log data and the second log data.

17. The system of claim 16, the operations further comprising:

configuring log monitoring configuration metadata, wherein the log monitoring configuration metadata comprises at least one of a base parser, a log rule applicable to a particular target of the target type, and an identification of the particular log target.

18. The system of claim 16, the operations further comprising:

identifying a parser for the log source, wherein the parser comprises a regular expression to define how to parse the log data for a particular target of the target type.

19. The system of claim 16, wherein the log collection configuration corresponds to an XML file.

20. The system of claim 16, wherein the log collection configuration is generated, modified, or deleted upon a change to an association between the first target and the log source, a change to the first target, or a change to a status of the first agent.

21. The system of claim 16, the operations further comprising:

periodically synchronizing associations between the log collection configuration and the plurality of targets.