Log collection, structuring and processing

• US 10,122,575 B2
• Filed: 06/10/2016
• Issued: 11/06/2018
• Est. Priority Date: 07/01/2010
• Status: Active Grant

• Current Assignee: Logrhythm Incorporated
• Original Assignee: Logrhythm Incorporated
• Inventors: Petersen, Chris, Villella, Phillip
• Primary Examiner(s): Christensen, Scott B
• Assistant Examiner(s): Do, Lam T
• Application Number:
  US15/179,342
• Publication Number:
  US 20160301561A1
• Time in Patent Office: 879 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3089 Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents software debugging using additional hardware using a specific debug interface G06F11/3656; performance evaluation by tracing or monitoring G06F11/3466
• CPC Class Codes: G06F 11/327 Alarm or error message display
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2221/2129 Authenticate client device independently of the user
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Tools for use in obtaining useful information from processed log messages generated by a variety of network platforms (e.g., Windows servers, Linux servers, UNIX servers, databases, workstations, etc.). The log messages may be processed by one or more processing platforms or “log managers” using any appropriate rule base to identify “events” (i.e., log messages of somewhat heightened importance), and one or more “event managers” may analyze the events to determine whether alarms should be generated therefrom. The tools may be accessed via any appropriate user interface of a console that is in communication with the various log managers, event managers, etc., to perform numerous tasks in relation to logs, events and alarms.

Claims

1. A method for use on a plurality of log manager devices of a data system, wherein each of the plurality of log manager devices is configured to monitor a plurality of devices of one or more platforms to identify events for handling by an event manager, the method comprising the steps of:

creating, at a first log manager device of the plurality of log manager devices, a first log processing rule having a filtering setting that identifies logs having a first of a plurality of classifications regarding a type of each log and a second log processing rule having a filtering setting that identifies logs received from a first source on the data system, wherein the first log processing rule includes a first data management setting that specifies to take or not take at least one action in relation to logs having the first classification, and wherein the second log processing rule includes a second data management setting that specifies the other of taking or not taking the at least one action in relation to logs having the first source;

establishing, at the first log manager device, a default setting stipulating that the second data management setting is to be applied instead of the first data management setting as to logs that match the filtering settings of both the first and second log processing rules;

receiving, at the first log manager device, an override setting indicating on that the default setting is to be ignored and the first data management setting is to be applied instead of the second data management setting as to logs that match the filtering settings of both the first and second log processing rules;

transmitting the override setting to each of the plurality of log manager devices based on a designation that the override setting is global;

receiving, at the first log manager device, logs generated at and transmitted from the plurality of devices of the one or more platforms;

identifying, at the first log manager device, at least a first of the received logs that matches the filtering settings of both the first and second log processing rules;

determining, at the first log manager device, that the first data management setting conflicts with the second data management setting; and

operating, based on the determining that first data management setting conflicts with the second data management setting, the first log manager device to implement the first data management setting on the first received log and ignore the second data management setting according to the override setting.

2. The method of claim 1, wherein the first data management setting specifies to take the at least one action, and wherein the second data management setting specifies to not take the at least one action.

3. The method of claim 1, wherein the first data management setting specifies to not take the at least one action, and wherein the second data management setting specifies to take the at least one action.

4. The method of claim 1, wherein the override setting is received on an action by action basis.

5. The method of claim 4, wherein the at least one action includes a plurality of actions, and wherein the receiving the override setting is in relation to a first of the plurality of actions and not in relation to a second of the plurality of actions.

6. The method of claim 1, wherein the at least one action comprises at least one of archiving of the received logs, the storage of the received logs in a database associated with said first loci manager device, and the forwarding of the received logs to a tool that can aggregate numerous received logs in a single view on a user interface.

7. The method of claim 1, wherein said first classification comprises at least one of security, audit and operations.

8. The method of claim 1, wherein each of the received logs includes first and second metadata fields including respective content that identifies a classification and a source of each received log, and wherein the identifying includes parsing the content of the metadata fields and determining that the content matches the first classification and the first source.

17. The method of claim 1, further including:

creating, at a second log manager device, the first log processing rule and a third log processing rule having a filtering setting that identifies logs received from a second source on the data system and a third data management setting that specifies to take or not take the at least one action in relation to logs having the second source;

establishing, at the second log manager device, a second default setting stipulating that the third data management setting is to be applied instead of the first data management setting as to logs that that match the filtering settings of both the first and third log processing rules;

receiving, at the second log manager device, a second override setting indicating that the second default setting is to be ignored and the first data management setting is to be applied and the third data management setting is to be ignored as to logs that match the filtering settings of both the first and third log processing rules;

receiving, at the second loci manager device, logs generated at and transmitted from the plurality of devices of the one or more platforms;

identifying, at the second log manager device, at least a second of the received logs that matches the filtering settings of both the first and third log processing rules; and

operating the second log manager device to implement the first data management setting on the second received log according to the second override setting.

9. A system for use in monitoring a plurality of devices of one or more platforms of a data system to identify events for handling by an event manager, the system comprising:

a processor;

a plurality of log manager devices; and

a non-transitory computer readable medium interconnected to the processor and including one or more non-transitory computer program products that are configured to;

create, at a first log manager device of the plurality of log manager devices, a first log processing rule having a filtering setting that identifies logs having a first of a plurality of classifications regarding a type of each of log and a second log processing rule having a filtering setting that identifies logs received from a first source on the data system, wherein the first log processing rule includes a first data management setting that specifies to take or not take at least one action in relation to logs having the first classification, and wherein the second log processing rule includes a second data management setting that specifies the other of taking or not taking the at least one action in relation to logs having the first source;

establish a default setting stipulating that the second data management setting is to be applied instead of the first data management setting as to logs that that match the filtering settings of both the first and second log processing rules;

receive, at the first log manager device, an override setting indicating that the default setting is to be ignored and the first data management setting is to be applied instead of the second data management setting as to logs that that match the filtering settings of both the first and second log processing rules;

transmit, as instructed by the processor, the override setting to each of the plurality of log manager devices based on a designation that the override setting is global;

receive, at the first log manager device, logs generated at and transmitted from the plurality of devices of said one or more platforms;

identify, at the first log manager device, at least a first of the received logs that matches the filtering settings of both the first and second log processing rules;

determine, at the first log manager device, that the first data management setting conflicts with the second data management setting; and

implement, by the first log manager device, the first data management setting on the first received log and ignore the second data management setting according to the override setting.

10. The system of claim 9, wherein the first data management setting specifies to take the at least one action, and wherein the second data management setting specifies to not take the at least one action.

11. The system of claim 9, wherein the first data management setting specifies to not take the at least one action, and wherein the second data management setting specifies to take the at least one action.

12. The system of claim 9, wherein the override setting is received on an action by action basis.

13. The system of claim 12, wherein the at least one action includes a plurality of actions, and wherein the receiving the override setting is in relation to a first of the plurality of actions and not in relation to a second of the plurality of actions.

14. The system of claim 9, wherein the at least one action comprises at least one of archiving of the received logs, the storage of the received logs in a database associated with said first loci manager device, and the forwarding of the received logs to a tool that can aggregate numerous received logs in a single view on a user interface.

15. The system of claim 9, wherein said first classification comprises at least one of security, audit and operations.

16. The system of claim 9, wherein each of the received logs includes first and second metadata fields including respective content that identifies a classification and a source of each received log, and wherein the identifying includes parsing the content of the metadata fields and determining that the content matches the first classification and the first source.