Log collection, structuring and processing

• US 9,384,112 B2
• Filed: 07/01/2011
• Issued: 07/05/2016
• Est. Priority Date: 07/01/2010
• Status: Active Grant

• Current Assignee: Logrhythm Incorporated
• Original Assignee: Logrhythm Incorporated
• Inventors: Petersen, Chris, Villella, Phillip
• Primary Examiner(s): Christensen, Scott B
• Assistant Examiner(s): DO, LAM T
• Application Number:
  US13/175,674
• Publication Number:
  US 20120005542A1
• Time in Patent Office: 1,831 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3089 Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents software debugging using additional hardware using a specific debug interface G06F11/3656; performance evaluation by tracing or monitoring G06F11/3466
• CPC Class Codes: G06F 11/327 Alarm or error message display
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2221/2129 Authenticate client device independently of the user
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Tools for use in obtaining useful information from processed log messages generated by a variety of network platforms (e.g., Windows servers, Linux servers, UNIX servers, databases, workstations, etc.). The log messages may be processed by one or more processing platforms or “log managers” using any appropriate rule base to identify “events” (i.e., log messages of somewhat heightened importance), and one or more “event managers” may analyze the events to determine whether alarms should be generated therefrom. The tools may be accessed via any appropriate user interface of a console that is in communication with the various log managers, event managers, etc., to perform numerous tasks in relation to logs, events and alarms.

Claims

1. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

identifying a set of circumstances with respect to at least one type of storage device associated with said data system, wherein said set of circumstances includes data being moved to or from the at least one type of storage device;

creating, using a processing platform of said data system, at least one log processing rule to identify logs of said one or more monitored platforms matching said set of circumstances, wherein said at least one log processing rule specifies at least one action to be performed based on the set of circumstances matching the at least one log processing rule;

receiving, at said processing platform, logs from one or more monitored platforms;

processing, by said processing platform, the received logs using said at least one log processing rule to identify received logs that match said set of circumstances, wherein the identified, received logs include information identifying data being moved to or from the at least one type of storage device; and

based on the processing of the received logs using said at least one log processing rule, taking the at least one action specified in the at least one log processing rule, wherein said taking the at least one action comprises;

limiting data from being written to said at least one storage device; and

generating at least one alert indicative of the occurrence of said set of circumstances.

2. The method as set forth in claim 1, wherein said set of circumstances further comprises at least one of a) one or more particular users and/or processes of said data system moving data within said data system, or b) one or more particular types of data being moved from within said data system.

3. The method as set forth in claim 1, wherein said at least one action comprises sending, from said processing platform to a device associated with said data system that is operable to write data to said storage device, a request to limit data from being written to said at least one storage device.

4. The method as set forth in claim 3, wherein said request to limit data from being written to said at least one storage device comprises a request to eject said at least one storage device from said data system.

5. The method as set forth in claim 3, wherein said sending step occurs before said set of circumstances is completed.

6. The method as set forth in claim 1, wherein said at least one action comprises sending, from said processing platform, said generated alert to at least one receiving entity.

7. The method as set forth in claim 1, wherein the storage device comprises at least one of a hard drive, flash drive, or an optical disc.

8. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

establishing, on a processing platform, at least one log processing rule for selectively processing logs associated with one or more monitored platforms based on a content of one or more data fields of said logs, wherein each of said at least one log processing rule specifies an action to perform based on content of one or more data fields of logs matching said at least one log processing rule;

receiving, at said processing platform, logs associated with said one or more monitored platforms;

processing, at said processing platform, the received logs using said at least one log processing rule;

identifying, using said processing platform, a processed log; and

first operating said processing platform to create at least one new log processing rule, wherein;

responsive to the processed log being identified using said processing platform, the processing platform generates a template rule including one or more conditions and actions automatically populated based on the one or more data fields of said identified processed log; and

responsive to generating the template rule, the method includes customizing the template rule to create the at least one new log processing rule, wherein the at least one new log processing rule includes at least one action to be performed based on a subsequently received log matching the one or more conditions.

9. The method as set forth in claim 8, further comprising:

second operating said processing platform to process said logs associated with said one or more monitored platforms using said at least one new log processing rule.

10. The method as set forth in claim 9, wherein said at least one new log processing rule identifies at least one event from said received logs.

11. The method as set forth in claim 8, further comprising:

identifying, using said processing platform, at least one event from said received logs using said at least one log processing rule.

12. The method as set forth in claim 11, further comprising:

second operating said processing platform to process said at least one event using said at least one new log processing rule.

13. The method as set forth in claim 12, wherein said at least one new log processing rule designates at least one alarm from said at least one event.

14. The method as set forth in claim 8, wherein said step of customizing further comprises:

modifying, in said at least one new log processing rule, at least one of said one or more data fields of said identified, processed log.

15. The method as set forth in claim 8, wherein said step of identifying further comprises:

selecting, using an iconic feature on a user interface, the log message on a console associated with said processing platform.

16. A method for use in monitoring one or more platforms of a data system, comprising the steps of:

integrating information from at least one directory service of the data system with a database of the data system, wherein the directory service includes information specifying users and user groups, and wherein the user groups each include a plurality of users;

identifying a set of circumstances with respect to at least one monitored platform of the one or more platforms of said data system, wherein said set of circumstances includes at least one action taken with respect to the at least one monitored platform by at least one of the plurality of users included in a specified user group of the user groups;

creating, using a processing platform of said data system, at least one log processing rule to identify logs of said one or more monitored platforms matching said set of circumstances, wherein said at least one log processing rule specifies at least one action to be performed based on the set of circumstances matching the at least one log processing rule;

receiving, at said processing platform, logs from one or more monitored platforms;

processing, by said processing platform, the received logs using the at least one log processing rule to identify received logs that match said set of circumstances, wherein the identified, received logs include information identifying at least one user of the plurality of users and an action taken by said at least one user of the plurality of users with respect to the at least one monitored platform; and

based on the processing of the received logs using the at least one log processing rule, taking the at least one action specified in the at least one log processing rule, wherein the taking the at least one action comprises generating at least one alert indicative of the occurrence of said set of circumstances.

17. The method as set forth in claim 16, further including:

storing said received logs in said database, wherein said processing includes processing one or more of the received, stored logs.

18. The method as set forth in claim 16, wherein the processing includes:

resolving the at least one user group into at least one user name; and

identifying, with the processing platform, one or more of the received logs that are related to the at least one user name.

19. The method as set forth in claim 16, wherein the received logs were generated by at least two different devices or hosts.

20. The method as set forth in claim 16, wherein said information from said at least one directory service of the data system comprises at least one of user names, user group names, logins, logoffs, logon session duration, total number of logins, initial password creation date, most recent password change date, most recent incorrect password entry date, or combination thereof.

21. The method as set forth in claim 16, wherein said information from said at least one directory service of the data system is associated with access of said one or more monitored platforms by one or more users of the data system.

22. A system for use in monitoring one or more platforms of a data system, the system comprising:

a processor; and

a non-transitory computer readable medium interconnected to the processor and including one or more non-transitory computer program products that are configured to;

create at least one log processing rule to identify logs of said one or more monitored platforms matching a set of circumstances, wherein said set of circumstances includes data being moved to or from at least one type of storage device associated with said data system, and wherein said at least one log processing rule specifies at least one action to be performed based on the set of circumstances matching the at least one log processing rule;

receive logs from one or more monitored platforms;

process the received logs using said at least one log processing rule to identify received logs that match said set of circumstances, wherein the identified, received logs including information identifying data being moved to or from the at least one type of storage device; and

based on the processing of the received logs using said at least one log processing rule, take the at least one action specified in the at least one log processing rule, wherein the at least one action comprises;

limiting data from being written to said at least one storage device; and

generating at least one alert indicative of the occurrence of said set of circumstances.

23. The system as set forth in claim 22, wherein said set of circumstances further comprises at least one of a) one or more particular users and/or processes of said data system moving data within said data system, or b) one or more particular types of data being moved from within said data system.

24. The system as set forth in claim 22, wherein said at least one action comprises sending, from said processor to a device associated with said data system that is operable to write data to said storage device, a request to limit data from being written to said at least one storage device.

25. The system as set forth in claim 22, wherein said at least one action comprises sending, from said processor, said generated alert to at least one receiving entity.

26. The system as set forth in claim 22, wherein said at least one storage device comprises at least one of a hard drive, flash drive, or an optical disc.

27. A processing platform for use in monitoring one or more platforms of a data system, comprising:

a storage module including at least one log processing rule for selectively processing logs associated with one or more monitored platforms based on a content of one or more data fields of said logs, wherein each of said at least one log processing rule specifies an action to perform based on content of one or more data fields of logs matching said at least one log processing rule; and

a processor that is operatively interconnected to the storage module, wherein the processor is operable to;

process said logs associated with one or more monitored platforms using said at least one log processing rule;

identify at least one of said logs associated with one of said one or more monitored platforms for further processing; and

create at least one new log processing rule, wherein;

responsive to the at least one of said logs being identified using said processor, the processor generates a template rule including one or more conditions and actions automatically populated based on the one or more data fields of said at least one identified processed log; and

responsive to the processor generating the template rule, the processor facilitates customization of the template rule to create the at least one new log processing rule, wherein the at least one new log processing rule includes at least one action to be performed based on a subsequently received log matching the one or more conditions.

28. The processing platform as set forth in claim 27, wherein said processor is operable to process said logs associated with said one or more monitored platforms using said at least one new log processing rule.

29. The processing platform as set forth in claim 28, wherein said at least one new log processing rule identifies at least one event from said received logs.

30. The processing platform as set forth in claim 27, wherein said processor is operable to process events identified from said logs associated with one of said one or more monitored platforms using said at least one new log processing rule.

31. The processing platform as set forth in claim 30, wherein said at least one new log processing rule designates at least one alarm from said events identified from said logs.