LOG COLLECTION, STRUCTURING AND PROCESSING

• US 20120005542A1
• Filed: 07/01/2011
• Published: 01/05/2012
• Est. Priority Date: 07/01/2010
• Status: Active Grant

• Current Assignee: Logrhythm Incorporated
• Original Assignee: Logrhythm Incorporated
• Inventors: Petersen, Chris, Villella, Phillip
• Granted Patent: US 9,384,112 B2
• Time in Patent Office: Days
• US Class Current: 714/48
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3089 Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents software debugging using additional hardware using a specific debug interface G06F11/3656; performance evaluation by tracing or monitoring G06F11/3466
• CPC Class Codes: G06F 11/327 Alarm or error message display
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2221/2129 Authenticate client device independently of the user
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Tools for use in obtaining useful information from processed log messages generated by a variety of network platforms (e.g., Windows servers, Linux servers, UNIX servers, databases, workstations, etc.). The log messages may be processed by one or more processing platforms or “log managers” using any appropriate rule base to identify “events” (i.e., log messages of somewhat heightened importance), and one or more “event managers” may analyze the events to determine whether alarms should be generated therefrom. The tools may be accessed via any appropriate user interface of a console that is in communication with the various log managers, event managers, etc., to perform numerous tasks in relation to logs, events and alarms.

Claims

1. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

identifying a set of circumstances with respect to at least one type of storage device usable with said data system;

establishing, using a processing platform of said data system, at least one log processing rule to identify logs of said one or more monitored platforms matching said set of circumstances;

receiving, at said processing platform, logs from one or more monitored platforms;

operating said processing platform to identify received logs that match said set of circumstances; and

taking at least one action with respect to said at least one type of storage device in response to said operating step.

2. The method as set forth in claim 1, wherein said set of circumstances comprises at least one of a) any movement of data to or from the at least one type of storage device, b) one or more particular users and/or processes of said data system moving data within said data system, and c) one or more particular types of data being moved from within said data system.

3. The method as set forth in claim 1, wherein said taking at least one action comprises generating at least one log message indicative of the occurrence of said set of circumstances.

4. The method as set forth in claim 3, wherein said set of circumstances comprises movement of data to or from the at least one type of storage device.

5. The method as set forth in claim 1, wherein said at least one action comprises sending, from said processing platform to a device associated with said data system that is operable to write data to said storage device, a request to limit data from being written to said at least one storage device.

6. The method as set forth in claim 5, wherein said request to limit data from being written to said at least one storage device comprises a request to eject said at least one storage device from said data system.

7. The method as set forth in claim 5, wherein said sending step occurs before said set of circumstances is completed.

8. The method as set forth in claim 1, wherein said at least one action comprises sending, from said processing platform, an alert to at least one receiving entity.

9. The method as set forth in claim 1, wherein the storage device comprises at least one of a hard drive, flash drive, and an optical disc.

10. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

establishing, on a processing platform, a number of log processing rules for selectively processing logs associated with one or more monitored platforms based on a content of one or more data fields of said logs;

receiving, at said processing platform, logs associated with said one or more monitored platforms;

processing, at said processing platform, the received logs using said number of log processing rules;

identifying, using said processing platform, a processed log; and

first operating said processing platform to establish at least one new log processing rule based on one or more data fields of said identified, processed log.

11. The method as set forth in claim 10, further comprising:

second operating said processing platform to process said logs associated with said one or more monitored platforms using said at least one new log processing rule.

12. The method as set forth in claim 11, wherein said at least one new log processing rule identifies at least one event from said received logs.

13. The method as set forth in claim 10, further comprising:

identifying, using said processing platform, at least one event from said received logs using said number of log processing rules.

14. The method as set forth in claim 13, further comprising:

second operating said processing platform to process said at least one event using said at least one new log processing rule.

15. The method as set forth in claim 14, wherein said at least one new log processing rule designates at least one alarm from said at least one event.

16. The method as set forth in claim 10, further comprising:

customizing said at least one new log processing rule.

17. The method as set forth in claim 16, wherein said step of customizing further comprises:

modifying, in said at least one new log processing rule, at least one of said one or more data fields of said identified log message that said at least one new log processing rule is based on.

18. The method as set forth in claim 10, wherein said step of identifying further comprises:

selecting, using a user manipulable feature, the log message on a console associated with said processing platform.

19. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

establishing, on a processing platform, a number of log processing rules for selectively processing logs associated with one or more monitored platforms based on a content of one or more data fields of said logs;

receiving, at said processing platform, logs associated with said one or more monitored platforms;

first operating said processing platform to process said logs associated with said one or more monitored platforms using said number of log processing rules;

identifying, using said processing platform and in response to said first operating step, at least one event from said logs associated with one of said one or more monitored platforms for further processing; and

second operating said processing platform to establish at least one new log processing rule based on one or more data fields of said identified event.

20. The method as set forth in claim 19, further comprising:

second operating said processing platform to process said logs associated with said one or more monitored platforms using said at least one new log processing rule.

21. The method as set forth in claim 20, wherein said at least one new log processing rule identifies at least one event from said received logs.

22. The method as set forth in claim 19, further comprising:

second operating said processing platform to process events identified from said logs associated with one of said one or more monitored platforms using said at least one new log processing rule.

23. The method as set forth in claim 22, wherein said at least one new log processing rule designates at least one alarm from said events identified from said logs.

24. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

establishing, on a processing platform, a number of log processing rules for selectively processing logs associated with one or more monitored platforms based on a content of one or more data fields of said logs;

identifying, using said processing platform, a log message associated with one of said one or more monitored platforms;

first operating said processing platform to establish at least one correlation filter based on one or more metadata fields of said identified log message; and

second operating said processing platform to correlate logs previously received and processed by said processing platform using said at least one correlation filter.

25. The method as set forth in claim 24, wherein said step of second operating further comprises:

searching for logs previously received and processed by said processing platform that match said correlation filter of said identified log message.

26. The method as set forth in claim 24, wherein said step of identifying further comprises:

selecting said log message on a user interface associated with said processing platform.

27. The method as set forth in claim 24, wherein said correlation filter comprises at least one selected from the group of classification, impacted country, impacted origin, direction and domain.

28. The method as set forth in claim 24, wherein the previously received and processed logs correlated by the processing platform were generated by at least two different devices or hosts.

29. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

synchronizing a database of the data system with information from at least one directory service of the data system;

first operating a processing platform to establish at least one correlation filter based on one or more data fields of the synchronized database; and

second operating said processing platform to correlate logs previously received and processed by said processing platform using said at least one correlation filter.

30. The method as set forth in claim 29, wherein the at least one correlation filter comprises at least one of a user name and group name, and wherein the second operating step comprises:

identifying logs previously received and processed by said processing platform including the at least one user name or group name.

31. The method as set forth in claim 30, wherein the at least one correlation filter comprises at least one group name, and wherein the method further comprises:

resolving the at least one group name into at least one user name, wherein the identified logs include the at least one resolved user name.

32. The method as set forth in claim 30, wherein the previously received and processed logs correlated by the processing platform were generated by at least two different devices or hosts.

33. The method as set forth in claim 29, wherein said information from said at least one directory service of the data system comprises at least one of user names, group names, logins, logoffs, logon session duration, total number of logins, initial password creation date, most recent password change date, most recent incorrect password entry date, and combinations thereof.

34. The method as set forth in claim 29, wherein said information from said at least one directory service of the data system is associated with access of said one or more monitored platforms by one or more users of the data system.

35. A method for use on one or more platforms of a data system, comprising the steps of:

specifying, at a user interface, whether or not to take at least one action in relation to a plurality logs received from one or more monitored platforms of the data system based on a classification of the received logs;

receiving, at a processing platform in communication with said user interface, logs from said one or more monitored platforms; and

operating the processing platform to process the received logs in accordance with the specifying step.

36. The method as set forth in claim 35, wherein said at least one action comprises at least one of archiving of the received logs, the storage of the received logs in a database associated with said processing platform, and the forwarding of the received logs to a tool that can aggregate numerous received logs in a single view on said user interface.

37. The method as set forth in claim 35, further comprising:

enabling, at said user interface, the at least one action.

38. The method as set forth in claim 35, wherein said specifying step takes precedence over at least one other conflicting setting of the processing platform.

39. The method as set forth in claim 38, wherein the at least one other conflicting setting specifies whether or not to take at least one action in relation to the plurality of received logs based on a log source of the received logs.

40. The method as set forth in claim 35, wherein said classification of the received logs comprises at least one of security, audit and operations.

41. A processing platform for use in monitoring one or more platforms of a data system, comprising:

a storage module including a number of log processing rules for selectively processing logs associated with one or more monitored platforms based on a content of one or more data fields of said logs;

a receiving module for receiving logs associated with said one or more monitored platforms; and

a processor that is operatively interconnected to the storage module and the receiving module, wherein the processor is operable to;

process said logs associated with said one or more monitored platforms using said number of log processing rules;

identify at least one event from said logs associated with one of said one or more monitored platforms for further processing; and

establish at least one new log processing rule based on one or more data fields of said identified event.

42. The processing platform as set forth in claim 41, wherein said processor is operable to process said logs associated with said one or more monitored platforms using said at least one new log processing rule.

43. The processing platform as set forth in claim 42, wherein said at least one new log processing rule identifies at least one event from said received logs.

44. The processing platform as set forth in claim 41, wherein said processor is operable to process events identified from said logs associated with one of said one or more monitored platforms using said at least one new log processing rule.

45. The processing platform as set forth in claim 44, wherein said at least one new log processing rule designates at least one alarm from said events identified from said logs.