System and method for parsing, summarizing and reporting log data

• US 20050114508A1
• Filed: 11/18/2004
• Published: 05/26/2005
• Est. Priority Date: 11/26/2003
• Status: Active Grant

• Current Assignee: Cloud Software Group
• Original Assignee: TIBCO Software Incorporated (Cloud Software Group)
• Inventors: DeStefano, Jason Michael
• Granted Patent: US 8,234,256 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3082 the data filtering being achieved by aggregating or compressing the monitored data
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A system and method is disclosed which enables network administrators and the like to quickly analyze the data produced by log-producing devices such as network firewalls and routers. Unlike systems of the prior art, the system disclosed herein automatically parses and summarizes log data before inserting it into one or more databases. This greatly reduces the volume of data stored in the database and permits database queries to be run and reports generated while many types of attempted breaches of network security are still in progress. Database maintenance may also be accomplished automatically by the system to delete or archive old log data.

Claims

1. A data processing system comprising:

a local area network;

a log-producing device connected to the local area network; and

,

a log data analyzer connected to the local area network which receives raw log data from the log-producing device over the local area network, parses the raw log data, summarizes the parsed log data, inserts the summarized log data into a database, and generates reports from the summarized log data in response to database queries.

2. A data processing system as recited in claim 1 wherein the log data analyzer parses raw log data by identifying values associated with certain fields within the raw log data.

3. A data processing system as recited in claim 2 wherein the log data analyzer compresses the values identified.

4. A data processing system as recited in claim 1 wherein the log data analyzer summarizes parsed log data by grouping log messages which relate to the same operation.

5. A data processing system as recited in claim 4 wherein the operation involves a series of building and terminating connections.

6. A data processing system as recited in claim 1 wherein the log data analyzer summarizes parsed log data by grouping log messages that comprise a connection sequence.

7. A data processing system as recited in claim 1 wherein the log-producing device is a device selected from the group consisting of routers, firewalls, file servers, VPN servers, e-mail servers and gateways.

8. A data processing system as recited in claim 1 wherein the log data analyzer summarizes the parsed log data periodically.

9. A data processing system as recited in claim 1 wherein the log data analyzer stores a pre-selected number of the most recently received log data messages concerning unapproved operations.

10. A data processing system as recited in claim 1 where in log data analyzer stores a pre-selected number of the most recently received raw log messages.

11. A data processing system as recited in claim 1 wherein the log data analyzer displays currently received log data.

12. A data processing system as recited in claim 1 where in log data analyzer aggregates summarized log data into a plurality of databases containing log data generated during pre-selected time periods of increasing duration.

13. A data processing system as recited in claim 1 wherein the log data analyzer aggregates and correlates log data messages from similar devices.

14. A method of processing log data from a log-producing device connected to a local area network comprising:

receiving raw log data from the log-producing device over the local area network;

parsing the raw log data;

summarizing the parsed log data;

inserting the summarized log data into a database; and

,

generating reports from the summarized log data in response to database queries.

15. A method as recited in claim 14 wherein parsing the raw log data comprises identifying values associated with certain fields within the raw log data

16. A method as recited in claim 15 further comprising compressing the values identified.

17. A method as recited in claim 14 wherein summarizing parsed log data comprises grouping log messages which relate to the same operation.

18. A method as recited in claim 17 wherein the operation involves a series of building and terminating connections.

19. A method as recited in claim 14 wherein summarizing parsed log data comprises grouping log messages that comprise a connection sequence.

20. A method as recited in claim 14 wherein the log-producing device is a device selected from the group consisting of routers, firewalls, file servers, VPN servers, e-mail servers and gateways.

21. A method as recited in claim 14 wherein the summarizing is performed periodically.

22. A method as recited in claim 14 further comprising storing a pre-selected number of the most recently received log data messages concerning unapproved operations.

23. A method as recited in claim 14 further comprising storing a pre-selected number of the most recently received raw log messages.

24. A method as recited in claim 14 further comprising displaying currently received log data.

25. A method as recited in claim 14 further comprising aggregating summarized log data into a plurality of databases containing log data generated during pre-selected time periods of increasing duration.

26. A method as recited in claim 14 further comprising aggregating and correlating log data messages from similar devices.

27. An apparatus for processing log data from a log-producing device connected to a local area network comprising:

a processor; and

,

a medium storing instructions for causing the processor to receive raw log data from the log-producing device over the local area network;

parse the raw log data;

summarize the parsed log data;

insert the summarized log data into a database; and

,

generate reports from the summarized log data in response to database queries.

28. An apparatus as recited in claim 27 wherein the instructions for causing the processor to parse raw log data comprise instructions for identifying values associated with certain fields within the raw log data

29. An apparatus as recited in claim 36 further comprising instructions to compress the values identified.

30. An apparatus as recited in claim 27 further comprising instructions to summarize parsed log data by grouping log messages which relate to the same operation.

31. An apparatus as recited in claim 38 wherein the operation involves a series of building and terminating connections.

32. An apparatus as recited in claim 27 further comprising instructions to summarize parsed log data by grouping log messages that comprise a connection sequence.

33. An apparatus as recited in claim 27 wherein the log-producing device is a firewall.

34. An apparatus as recited in claim 27 wherein the log-producing device is a device selected from the group consisting of routers, firewalls, file servers, VPN servers, e-mail servers and gateways.

35. An apparatus as recited in claim 27 wherein the instructions for causing the processor to summarize parsed log data are performed periodically.

36. An apparatus as recited in claim 27 further comprising instructions for storing a pre-selected number of the most recently received log data messages concerning unapproved operations.

37. An apparatus as recited in claim 27 further comprising instructions for storing a pre-selected number of the most recently received raw log messages.

38. An apparatus as recited in claim 27 further comprising instructions for displaying currently received log data.

39. An apparatus as recited in claim 27 further comprising instructions for aggregating summarized log data into a plurality of databases containing log data generated during pre-selected time periods of increasing duration.

40. An apparatus as recited in claim 27 further comprising instructions for aggregating and correlating log data messages from similar devices.