Systems and methods for highly scalable system log analysis, deduplication and management

• US 9,122,694 B1
• Filed: 10/27/2014
• Issued: 09/01/2015
• Est. Priority Date: 09/24/2014
• Status: Active Grant

• Current Assignee: LogZilla Corporation
• Original Assignee: LogZilla Corporation
• Inventors: Dukes, Clayton Brandon, Piatkowski, Piotr
• Primary Examiner(s): Ly, Cheyne D
• Application Number:
  US14/524,625
• Time in Patent Office: 309 Days
• Field of Search: 707/672, 707/692, 707/698
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0683 Plurality of storage devices
• CPC Class Codes: G06F 40/205 Parsing

Abstract

Systems and methods for parsing raw log data into structured log data, and removing duplicate entries, storing the deduplicated log data into binary format, and managing system events. The subject matter can increase speed of log data analysis and storage, reduce data storage for log data, and manage easily system events.

Claims

1. Non-transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to create an application for system log analysis and deduplication, the application comprising:

(a) a user interface allowing a user to configure one or more parser modules or a plurality of storage modules, wherein the plurality of storage modules are distributed to two or more servers;

(b) the one or more parser modules configured to;

(1) receive raw system log data automatically generated by a system, wherein the raw system log data comprises system events logged in a plain text format;

(2) parse and transform the raw system log data into structured system log data by a hash function, wherein the structured system log data comprises the system events organized in a defined data format; and

(3) transmit the structured system log data to the plurality of storage modules, wherein data entries of the structured system log data associated with a same system event and happening within a first defined time window are transmitted to a same storage module; and

(c) the plurality of storage modules without a database service, the storage modules configured to;

(1) receive the structured system log data from the one or more parser modules;

(2) identify duplicated data entries in the structured system log data, wherein the duplicated data entries are happening within a second defined time window and comprise distinct timestamps and identical system events;

(3) generate a representative system event of the duplicated data entries;

(4) generate and store a serialized data record in a binary format, the serialized data record comprising the distinct timestamps of the duplicated data entries and the representative system event of the duplicated data entries.

2. The media of claim 1, provided that a number of the plurality of storage modules is more than 10.

3. The media of claim 1, provided that a number of the plurality of storage modules is more than 100.

4. The media of claim 1, provided that a number of the plurality of storage modules is more than 1000.

5. The media of claim 1, provided that a number of the plurality of storage modules is more than 10000.