Non-homogenous storage of events in event data store

• US 10,061,805 B2
• Filed: 02/24/2016
• Issued: 08/28/2018
• Est. Priority Date: 02/25/2015
• Status: Active Grant

• Current Assignee: Sumo Logic, Inc.
• Original Assignee: Sumo Logic, Inc.
• Inventors: Tidwell, Kenny, Frampton, David, O'Connell, Brendan
• Primary Examiner(s): Getachew, Abiy
• Application Number:
  US15/052,523
• Publication Number:
  US 20160248791A1
• Time in Patent Office: 916 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/282 Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H05K 999/99 dummy group

Abstract

A processing device receives a plurality of discrete log entries from a first data store and generates an event for each discrete log entry that satisfies a criterion. To generate an event the processing device determines a source type associated with a discrete log entry, parses the discrete log entry based on the source type, determines a plurality of fields of the discrete log entry, identifies a subset of the plurality of fields, wherein one or more fields in the subset are to be used as link keys for linking together events, and assigns a field type to each field in the subset of the plurality of fields. The processing device additionally writes a plurality of event entries for the event into a second data store. A separate event entry is written for each field of the subset of the plurality of fields having an assigned field type.

Claims

1. A method comprising:

receiving, by a processing device, a plurality of discrete log entries from a first data store; and

generating, by the processing device, an event for each discrete log entry of the plurality of discrete log entries that satisfies a criterion, wherein generating the event for a discrete log entry comprises;

determining a source type associated with the discrete log entry;

parsing the discrete log entry based on the source type;

determining a plurality of fields of the discrete log entry;

identifying a subset of the plurality of fields, wherein one or more fields in the subset are to be used as link keys for linking together events;

assigning a field type to each field in the subset of the plurality of fields; and

writing a plurality of event entries for the event into a second data store, wherein a separate event entry is written to the second data store for each field of the subset of the plurality of fields having an assigned field type.

2. The method of claim 1, further comprising:

receiving a first context definition that comprises a specified plurality of fields, wherein the first context definition identifies the specified plurality of fields to use as the link keys; and

storing the first context definition in configuration data.

3. The method of claim 2, wherein the second data store comprises a database, the method further comprising:

receiving a modified version of the first context definition after events for the plurality discrete log entries have been written to the second data store, wherein the modified version of the first context definition comprises an alternate specified plurality of fields; and

replacing the context definition with the modified version of the context definition in the configuration data without modifying a schema of the database.

4. The method of claim 2, wherein the first context definition comprises at least one of:

an identity context definition that is usable to associate users to at least one of devices, locations or aliases, the identity context definition comprising a user field, an internet protocol (IP) address field and a media access control (MAC) address field;

an endpoint context definition that is usable to describe a device on a network, the endpoint context definition comprising an IP address field and a host field;

a network context definition that is usable to describe traffic across a network, the network context definition comprising a source IP address field, a destination IP address field and a port field;

an application context definition that is usable to describe at least one of a service request or a service response, the application context definition comprising an IP address field and a host field;

a data context definition that is usable to describe content of network traffic, the data context definition comprising a transmitted bytes field, a data encoding field and a data characterization field;

or

a threat context definition that is usable to describe a network threat detected by at least one of an intrusion detection system (IDS), a security information and event management (SIEM) system, or a user behavior analytics (UBA) system.

5. The method of claim 4, further comprising:

receiving a plurality of context definitions, the first context definition being one of the plurality of context definitions, wherein the plurality of context definitions comprise the identity context definition, the endpoint context definition, the network context definition, the application context definition, the data context definition and the threat context definition; and

storing the plurality of context definitions in the configuration data.

6. The method of claim 2, further comprising:

receiving a second context definition having a shared context type as the first context definition; and

grouping the first context definition and the second context definition based on the shared context type.

7. The method of claim 2, further comprising:

receiving a query comprising a field value and a time period;

searching the second data store to identify a plurality of events having the time period and at least one field that comprises the field value;

determining, for each event of the plurality of events, a context definition associated with the event;

determining, for each event, field values of fields having assigned field types that are included in the context definition associated with that event;

aggregating information on the field values; and

generating a response to the query that comprises the information.

8. The method of claim 1, wherein the data store comprises a non-homogenous database, and wherein writing the plurality of event entries to the second data store comprises:

sending a first instruction to a database management system (DBMS) to cause the DBMS to store a first event entry in the second data store using a first field of the subset of the plurality of fields as a first index; and

sending a second instruction to the DBMS to cause the DBMS to store a second event entry in the second data store using a second field of the plurality of fields as a second index.

9. The method of claim 1, further comprising:

determining whether the discrete log entry is parsable; and

determining that the discrete log entry satisfies the criterion responsive to determining that the discrete log entry is parsable.

10. A non-transitory computer readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising:

receiving, by the processing device, a plurality of discrete log entries from a first data store; and

generating, by the processing device, an event for each discrete log entry of the plurality of discrete log entries that satisfies a criterion, wherein generating the event for a discrete log entry comprises;

determining a source type associated with the discrete log entry;

parsing the discrete log entry based on the source type;

determining a plurality of fields of the discrete log entry;

identifying a subset of the plurality of fields, wherein one or more fields in the subset are to be used as link keys for linking together events;

assigning a field type to each field in the subset of the plurality of fields; and

writing a plurality of event entries for the event into a second data store, wherein a separate event entry is written to the second data store for each field of the subset of the plurality of fields having an assigned field type.

11. The non-transitory computer readable storage medium of claim 10, the operations further comprising:

receiving a first context definition that comprises a specified plurality of fields, wherein the first context definition identifies the specified plurality of fields to use as the link keys; and

storing the first context definition in configuration data.

12. The non-transitory computer readable storage medium of claim 11, wherein the second data store comprises a database, the operations further comprising:

receiving a modified version of the first context definition after events for the plurality discrete log entries have been written to the second data store, wherein the modified version of the first context definition comprises an alternate specified plurality of fields; and

replacing the context definition with the modified version of the context definition in the configuration data without modifying a schema of the database.

13. The computer non-transitory readable storage medium of claim 11, wherein the first context definition comprises at least one of:

an identity context definition that is usable to associate users to at least one of devices, locations or aliases, the identity context definition comprising a user field, an internet protocol (IP) address field and a media access control (MAC) address field;

an endpoint context definition that is usable to describe a device on a network, the endpoint context definition comprising an IP address field and a host field;

a network context definition that is usable to describe traffic across a network, the network context definition comprising a source IP address field, a destination IP address field and a port field;

an application context definition that is usable to describe at least one of a service request or a service response, the application context definition comprising an IP address field and a host field;

a data context definition that is usable to describe content of network traffic, the data context definition comprising a transmitted bytes field, a data encoding field and a data characterization field;

or a threat context definition that is usable to describe a network threat detected by at least one of an intrusion detection system (IDS), a security information and event management (SIEM) system, or a user behavior analytics (UBA) system.

14. The non-transitory computer readable storage medium of claim 13, the operations further comprising:

receiving a plurality of context definitions, the first context definition being one of the plurality of context definitions, wherein the plurality of context definitions comprise the identity context definition, the endpoint context definition, the network context definition, the application context definition, the data context definition and the threat context definition; and

storing the plurality of context definitions in the configuration data.

15. The non-transitory computer readable storage medium of claim 11, the operations further comprising:

receiving a second context definition having a shared context type as the first context definition; and

grouping the first context definition and the second context definition based on the shared context type.

16. The non-transitory computer readable storage medium of claim 11, the operations further comprising:

receiving a query comprising a field value and a time period;

searching the second data store to identify a plurality of events having the time period and at least one field that comprises the field value;

determining, for each event of the plurality of events, a context definition associated with the event;

determining, for each event, field values of fields having assigned field types that are included in the context definition associated with that event;

aggregating information on the field values; and

generating a response to the query that comprises the information.

17. The non-transitory computer readable storage medium of claim 10, wherein writing the plurality of event entries to the second data store comprises:

sending a first instruction to a database management system (DBMS) to cause the DBMS to store a first event entry in the second data store using a first field of the subset of the plurality of fields as a first index; and

sending a second instruction to the DBMS to cause the DBMS to store a second event entry in the second data store using a second field of the plurality of fields as a second index.

18. The non-transitory computer readable storage medium of claim 11, the operations further comprising:

determining whether the discrete log entry is parsable; and

determining that the discrete log entry satisfies the criterion responsive to determining that the discrete log entry is parsable.

19. A system comprising:

a first data store to store a plurality of discrete log entries;

a second data store; and

a computing device, operatively coupled to the first data store and the second data store, to;

receive a plurality of discrete log entries from the first data store; and

generate an event for each discrete log entry of the plurality of discrete log entries that satisfies a criterion, wherein to generate the event for a discrete log entry the computing device is to;

determine a source type associated with the discrete log entry;

parse the discrete log entry based on the source type;

determine a plurality of fields of the discrete log entry;

identify a subset of the plurality of fields, wherein one or more fields in the subset are to be used as link keys for linking together events;

assign a field type to each field in the subset of the plurality of fields; and

cause a plurality of event entries for the event to be written into the second data store, wherein a separate event entry is written to the second data store for each field of the subset of the plurality of fields having an assigned field type.

20. The system of claim 19 wherein the computing device is further to:

receive a first context definition that comprises a specified plurality of fields, wherein the first context definition identifies the specified plurality of fields to use as the link keys; and

store the first context definition in configuration data.