Event context management system

• US 9,811,562 B2
• Filed: 02/24/2016
• Issued: 11/07/2017
• Est. Priority Date: 02/25/2015
• Status: Active Grant

• Current Assignee: Sumo Logic, Inc.
• Original Assignee: FactorChain, Inc. (Sumo Logic, Inc.)
• Inventors: Tidwell, Kenny, Frampton, David, O'Connell, Brendan
• Primary Examiner(s): Dinh, Minh
• Application Number:
  US15/052,636
• Publication Number:
  US 20160248792A1
• Time in Patent Office: 622 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/282 Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H05K 999/99 dummy group

Abstract

A processing device receives a plurality of discrete log entries from a first data store and generates an event for each discrete log entry that satisfies a criterion. To generate an event the processing device determines a source type associated with a discrete log entry, parses the discrete log entry based on the source type, determines a plurality of fields of the discrete log entry, identifies a subset of the plurality of fields, wherein one or more fields in the subset are to be used as keys for indexing events, and assigns a field type to each field in the subset of the plurality of fields. The processing device additionally writes a plurality of event entries for the event into a second data store. A separate event entry is written for each field of the subset of the plurality of fields having an assigned field type.

Claims

1. A system comprising:

a first computing device comprising a first data store;

a second computing device comprising a second data store;

a third computing device, connected to the first computing device, to;

receive a plurality of initial data streams comprising log data from a plurality of data sources, wherein a first initial data stream of the plurality of initial data streams comprises a first plurality of messages comprising first log data that is associated with a first data source of the plurality of data sources; and

cause the first computing device to write the plurality of initial data streams to the first data store, wherein the first initial data stream received from the first data source is stored in a first initial data stream record in the first data store; and

a fourth computing device, connected to the first computing device and to the second computing device, to;

determine a first log format of the first log data in the first initial data stream record;

determine boundaries of a plurality of discrete log entries included in one or more messages of the first plurality of messages based on the first log format;

separate the one or more messages into the plurality of discrete log entries; and

generate an event for a discrete log entry of the plurality of discrete log entries, wherein to generate the event for the discrete log entry the fourth computing device is to;

parse the discrete log entry based on the first log format to identify a plurality of fields;

identify a subset of the plurality of fields to be used as keys for indexing events;

assign a field type to each field in the subset of the plurality of fields; and

cause the second computing device to write a plurality of event entries for the event into the second data store, wherein a separate event entry is written to the second data store for each field of the subset of the plurality of fields having an assigned field type.

2. The system of claim 1, wherein the third computing device is further to:

determine a source type for the first data source based on at least one of an agent that the first data stream is received from, an internet protocol (IP) address associated with the first data source, a host name associated with the first data source, a port associated with the first data source, a file path associated with the first data source, or a uniform resource locator (URL) associated with the first data source, wherein the source type identifies a particular log format;

cause a first data source object that comprises the source type and a first data source identifier (ID) to be generated; and

cause the first computing device to insert metadata comprising the first data source ID into the first data stream record, wherein the source type identifies the first log format.

3. The system of claim 1, wherein a tail end of a first message of the first plurality of messages comprises a first portion of a log entry and a front end of a second message of the first plurality of messages comprises a second portion of the log entry, and wherein the fourth computing device is further to:

combine the first portion with the second portion to reconstruct the log entry.

4. The system of claim 1, wherein the third computing device is further to:

receive a new message from the first data source in the first initial data stream;

determine an amount of time that has elapsed since a previous message was received from the first data source; and

responsive to determining that the amount of time that has elapsed exceeds a threshold, send a notification to the fourth computing device to cause the fourth computing device to start processing the new message from the first initial data stream record.

5. The system of claim 1, wherein the fourth computing device is further to:

generate a first corrected data stream from the one or more messages that have been separated;

cause the first computing device to write the first corrected data stream to a first corrected data stream record in the first data store; and

retrieve the discrete log entry from the first corrected data stream record prior to parsing the discrete log entry.

6. The system of claim 5, wherein the fourth computing device is further to:

determine that parsing has failed for one or more discrete log entries in the first corrected data stream record;

generate a first unparsed data stream from the one or more discrete log entries; and

cause the first computing device to write the first unparsed data stream to a first unparsed data stream record in the first data store.

7. The system of claim 1, wherein to generate the event for a particular log entry the fourth computing device is further to:

determine that the first log format is insufficient to fully parse the particular log entry;

determine a transform usable to further parse the particular log entry;

identify at least one field in the subset based on the transform; and

assign the field type to the at least one field based at least in part on the transform.

8. The system of claim 1, wherein:

the first computing device is to execute a messaging system that guarantees message order; and

the second computing device is to execute a database management system and a database comprising a plurality of rows that are organized into a plurality of tables, wherein each table of the plurality of tables comprises a primary key corresponding to a particular field type.

9. The system of claim 1, wherein a second initial data stream of the plurality of initial data streams comprises a second plurality of messages comprising second log data that is associated with a second data source of the plurality of data sources, and wherein the third computing device is further to cause the first computing device to store the second data stream from the second data source in a second initial data stream record in the first data store.

10. The system of claim 9, wherein the third computing device is further to:

determine that a source type for the second data source is an unknown source type;

cause a data source object that comprises the unknown source type and a data source identifier (ID) to be generated; and

cause the first computing device to insert metadata comprising the data source ID into the second data stream record.

11. The system of claim 10, wherein the fourth computing device is further to:

determine that the source type for the second data source is the unknown source type;

perform an analysis of the second log data;

determine, from the analysis, a second source type for the second data source; and

cause the data source object to be modified to comprise the second source type, wherein the second source type identifies a second log format.

12. A method comprising:

receiving a plurality of initial data streams comprising log data from a plurality of data sources, wherein a first initial data stream of the plurality of initial data streams comprises a first plurality of messages comprising first log data that is associated with a first data source of the plurality of data sources;

writing the plurality of initial data streams to a first data store, wherein the first initial data stream received from the first data source is stored in a first initial data stream record in the first data store;

determining a first log format of the first log data in the first initial data stream record;

determining boundaries of a plurality of discrete log entries included in one or more messages of the first plurality of messages based on the first log format;

separating the one or more messages into the plurality of discrete log entries; and

generating an event for a discrete log entry of the plurality of discrete log entries, wherein generating the event comprises;

parsing the discrete log entry based on the first log format to identify a plurality of fields;

identifying a subset of the plurality of fields to be used as keys for indexing events;

assigning a field type to each field in the subset of the plurality of fields; and

writing a plurality of event entries for the event into a second data store, wherein a separate event entry is written to the second data store for each field of the subset of the plurality of fields having an assigned field type.

13. The method of claim 12, further comprising:

determining a source type for the first data source based on at least one of an agent that the first data stream is received from, an internet protocol (IP) address associated with the first data source, a host name associated with the first data source, a port associated with the first data source, a file path associated with the first data source, or a uniform resource locator (URL) associated with the first data source, wherein the source type identifies a particular log format;

generating a first data source object that comprises the source type and a first data source identifier (ID); and

causing the first computing device to insert metadata comprising the first data source ID into the first data stream record, wherein the source type identifies the first log format.

14. The method of claim 12, wherein a tail end of a first message of the first plurality of messages comprises a first portion of a log entry and a front end of a second message of the first plurality of messages comprises a second portion of the log entry, the method further comprising:

combining the first portion with the second portion to reconstruct the log entry.

15. The method of claim 12, further comprising:

generating a first corrected data stream from the one or more messages that have been separated;

causing the first computing device to write the first corrected data stream to a first corrected data stream record in the first data store; and

retrieving the discrete log entry from the first corrected data stream record prior to parsing the discrete log entry.

16. The method of claim 15, further comprising:

determining that parsing has failed for one or more discrete log entries in the first corrected data stream record;

generating a first unparsed data stream from the one or more discrete log entries; and

causing the first computing device to write the first unparsed data stream to a first unparsed data stream record in the first data store.

17. The method of claim 12, wherein generating the event for a particular log entry comprises:

determining that the first log format is insufficient to fully parse the particular log entry;

determining a transform usable to further parse the particular log entry;

identifying at least one field in the subset based on the transform; and

assigning the field type to the at least one field based at least in part on the transform.

18. The method of claim 12, wherein a second initial data stream of the plurality of initial data streams comprises a second plurality of messages comprising second log data that is associated with a second data source of the plurality of data sources, the method further comprising storing the second data stream from the second data source in a second initial data stream record in the first data store.

19. The method of claim 18, further comprising:

determining that a source type for the second data source is an unknown source type;

generating a data source object that comprises the unknown source type and a data source identifier (ID); and

inserting metadata comprising the data source ID into the second data stream record.

20. The method of claim 19, further comprising:

determining that the source type for the second data source is the unknown source type;

performing an analysis of the second log data;

determining, from the analysis, a second source type for the second data source; and

causing the data source object to be modified to comprise the second source type, wherein the second source type identifies a second log format.