LOG COLLECTION, STRUCTURING AND PROCESSING

• US 20070283194A1
• Filed: 11/13/2006
• Published: 12/06/2007
• Est. Priority Date: 11/12/2005
• Status: Active Grant

• Current Assignee: Logrhythm Incorporated
• Original Assignee: Logrhythm Incorporated
• Inventors: Petersen, Chris, Villella, Phillip
• Granted Patent: US 7,653,633 B2
• Time in Patent Office: Days
• US Class Current: 714/57
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: H04L 41/06 Management of faults, events, alarms or notifications
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data

Abstract

The present invention generally relates to log message processing such that events can be detected and alarms can be generated. For example, log messages are generated by a variety of network platforms (e.g., Windows servers, Linux servers, UNIX servers, databases, workstations, etc.). Often, relatively large numbers of logs are generated from these platforms in different formats. A log manager described herein collects such log data using various protocols (e.g., Syslog, SNMP, SMTP, etc.) to determine events. That is, the log manager may communicate with the network platforms using appropriate protocols to collect log messages therefrom. The log manager may then determine events (e.g., unauthorized access, logins, etc.) from the log data and transfer the events to an event manager. The event manager may analyze the events and determine whether alarms should be generated therefrom.

Claims

1. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

establishing a number of log processing rules for selectively processing logs based on a content of one or more data fields of the said logs;

identifying a log message associated with a monitored platform;

parsing said log message into a number of data fields and determining a field content of at least one of said fields; and

based on said field content, processing said log message using said processing rules.

2. A method as set forth in claim 1, wherein said step of establishing comprises providing at least one rule governing archiving of log messages on a log content dependent basis.

3. A method as set forth in claim 1, wherein said step of establishing comprises providing at least one rule for distinguishing between logs of a first category and a second category, wherein logs of said first category are prioritized for review.

4. A method as set forth in claim 3, wherein logs of said first type are designated as an event for further processing.

5. A method as set forth in claim 4, wherein said event relates to one of application monitoring, security, operations and auditing or regulatory compliance.

6. A method as set forth in claim 5, wherein said step of parsing comprises identifying data fields corresponding to one or more of a log host, a log message source, an IP address, a program and a login.

7. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

identifying a log message to be archived;

associating metadata with the said identified log message to be archived, wherein said metadata includes information to assist in restoring said message; and

archiving said log message together with said metadata in a data structure for enabling restoration of said log message using said metadata.

8. A method for use in processing logs in a data system, comprising the steps of:

providing a tool for use in accessing archived logs based on metadata describing the logs;

first operating said tool to receive restoration information associated with one or more fields of said metadata; and

second operating said tool to restore one or more logs based on said received restoration information.

9. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

establishing an agent protocol defining communications between an agent for acquiring log messages and a processor for processing log messages;

providing an agent for monitoring log messages associated with a monitored platform; and

operating said agent to identify a log message of interest, process said log message to parse said message and associate metadata with parsed portions of said message, and provide an output in accordance with said agent protocol, wherein said output includes substantially an entirety of a content of said message together with said metadata.

10. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

establishing an agent protocol defining communications between an agent for acquiring log messages and a processor for processing log messages;

receiving, at said processor, a pre-processed log message in accordance with said protocol, where said pre-processed log message includes parsed portions of a raw log message, collectively including substantially an entirety of a content of said raw message, together with metadata defining said parsed portions; and

operating said processor to use said metadata to further process said preprocessed log message.

11. A method for use in processing textual messages in a data system, comprising the steps of:

establishing a tagging notation in relation to a subject matter area of a textual message, said tagging notation including a metadata model for describing parsed portions of said textual message; and

establishing rules operative for converting said tagging notation into regular expression notation.

12. A method as set forth in claim 11, wherein said tagging notation includes at least a first tag and a second tag wherein said first tag is operative for identifying a parsed portion of said textual messages and said second tag is operative to identify a sub rule, wherein said sub rule further defines one of said rules.

13. A method for use in monitoring one or more platforms in a data system, comprising the steps of:

accessing a Windows event log from a monitored platform;

identifying a time stamp associated with the Windows event log; and

normalizing the time stamp such that the time stamp is substantially independent of a processing environment of the monitored platform.

14. A method as set forth in claim 13 wherein said step of normalizing comprises accounting for one of a local time zone, a local clock offset, and a local platform time system.

15. A method for use in monitoring one or more platforms in a data system, comprising the steps of:

providing a log manager for processing log information from one or more monitored platforms;

operating the log manager to remotely access a Windows event log, including a first time stamp, from a monitored platform; and

processing the first time stamp such that the first time stamp is substantially independent of a processing environment of a monitored platform.

16. A method as set forth in claim 15, further comprising the steps of operating the log manager to remotely access a second Windows event log from a second monitored platform and processing a second time stamp associated with the second Windows event log such that the first time stamp and the second time stamp have a common time reference.

17. A method as set forth in claim 15, wherein said step of processing said time stamp is implemented substantially free from operating an agent on said monitored platform for use in normalizing the time stamp information.

18. A method for use in monitoring one or more platforms in a data system, comprising the steps of:

providing a storage for storing a local copy of log message strings for a predetermined length of time; and

in connection with processing a log, determining whether a log message string is in said cache and, if not, obtaining the message string from a remote system.