Adaptive parsing and normalizing of logs at MSSP

• US 10,599,668 B2
• Filed: 10/31/2017
• Issued: 03/24/2020
• Est. Priority Date: 10/31/2017
• Status: Active Grant

• Current Assignee: SecureWorks Corp. (Dell Technologies Inc.)
• Original Assignee: SecureWorks Corp. (Dell Technologies Inc.)
• Inventors: McLean, Lewis
• Primary Examiner(s): Jamshidi, Ghodrat
• Application Number:
  US15/799,458
• Publication Number:
  US 20190130009A1
• Time in Patent Office: 875 Days
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 43/18 Protocol analysers
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A method of normalizing security log data can include receiving one or more security logs including unstructured data from a plurality of devices and reviewing unstructured data of the one or more security logs. The method also can include automatically applying a probabilistic model of one or more engines to identify one or more attributes or features of the unstructured data, and determine whether the identified attributes or features are indicative of identifiable entities, and tagging one or more identifiable entities of the identifiable entities, as well as organizing tagged entities into one or more normalized logs having a readable format with a prescribed schema. In addition, the method can include reviewing the one or more normalized logs for potential security events.

Claims

1. A security system for a network, comprising:

an event management center including at least one processor configured to;

receive security logs including security log data from a plurality of monitored devices;

determine whether one or more parsing scripts or rules are available to parse or normalize the security log data in the received security logs; and

if one or more parsing scripts or rules are available;

apply the one or more parsing scripts or rules to the security log data; and

normalize the security log data and organize the normalized securing log data into a structured format; and

if one or more parsing scripts or rules are not available, provide the security data to one or more engines for parsing or normalization thereof, wherein the one or more engines are stored in a memory of or accessible by the at least one processor, and at least one of the engines is configured to;

receive one or more security logs that comprise the security log data in an unrecognized format or include the security log data that is at least partially unpayable by the one or more parsing scripts or rules accessible by the at least one processor;

identify one or more attributes of the security log data;

determine a probability that the one or more identified attributes represent one or more recognized security log entities; and

if the determined probability meets or exceeds a predetermined threshold probability, isolate and/or tag recognized security log entities and organize isolated and/or tagged recognized security log entities into a structured format to generate normalized security logs;

wherein the normalized security logs are reviewable to determine if a security threat has been detected.

2. The system of claim 1, wherein the event management center comprises a data center of a managed security service provider.

3. The system of claim 1, wherein the event management center comprises a network server.

4. The system of claim 1, wherein the at least one processor includes programming to identify identifiable entities of incoming security logs and select and apply a parsing script or rule corresponding to the identified security log.

5. The system of claim 4, wherein if the at least one processor is unable to identify an entity of an incoming security log with a prescribed level of confidence the processor forwards the incoming security log to the at least one engine.

6. The system of claim 1, further comprising one or more training data sets generated for each of the one or more engines, the one or more training data sets including historically identified features or attributes indicative of identifiable entities of the security logs received by the event management center.

7. The system of claim 1, wherein the one or more engines further include programming to perform probabilistic modeling to determine whether selected attributes, features, commonalities, and/or sequences thereof are indicative of one or more of the identifiable entities of the security logs.

8. The system of claim 1, wherein the probabilistic modeling includes named entity recognition.

9. A method of normalizing security log data, comprising:

receiving a series of security logs from a plurality of devices;

for each security log received, determining whether a parsing script is available to normalize the security log;

if a parsing script is available, normalizing the security log with a selected parsing script and placing the security log into a normalized event format for review and detection of a potential security event;

if a parsing script is not determined to be available for a security log, routing the unparsed security log to one or more engines, wherein the one or more engines;

review unconstructed data of the unparsed security log;

identify one or more attributes or features of the unparsed security log;

determine, with a prescribed level of confidence, whether the identified attributes or features are indicative of one or more identifiable entities in the security log;

tag identifiable entities of the one or more identifiable entities in the security log; and

organize the tagged entities into a structured format with a prescribed schema; and

review the normalized security logs to determine potential security events.

10. The method of claim 9, further comprising generating training data sets for training the one or more engines.

11. The method of claim 10, further comprising updating the training data sets with security log data processed by the one or more engines.

12. The method of claim 10, wherein generating the training data comprises dynamically analyzing sets of data from parsed or readable security logs and sets of data from raw or unstructured security logs.

13. The method of claim 9, wherein the engines further include probabilistic modeling to determine whether selected attributes, features, commonalities, or sequences thereof, are indicative of one or more identifiable entities of the security logs.

14. The method of claim 9, wherein if one or more attributes of the unparsed security log are not identifiable with one or more known entities of the incoming security logs with the prescribed level of confidence, generating an alert.

15. A method of normalizing security log data, comprising:

receiving one or more security logs including unstructured data from a plurality of devices;

reviewing the unstructured data of the one or more security logs to determine whether one or more parsing scripts or rules are available for parsing and/or normalization of the unstructured data;

if one or more parsing scripts or rules are available to parse or normalize the unstructured data, applying the one or more parsing scripts or rules to the unstructured data for parsing or normalization of the unstructured data into one or more normalized logs;

if one or more parsing scripts or rules are not available, applying a probabilistic model of one or more engines to identify one or more attributes or features of the unstructured data, and determine whether the identified attributes or features are indicative of identifiable entities;

tagging one or more identifiable entities of the identifiable entities;

organizing tagged entities into one or more normalized logs having a readable format with a prescribed schema; and

reviewing the one or more normalized logs for potential security events.

16. The method of claim 15, further comprising generating training data sets for training the one or more engines.

17. The method of claim 16, further comprising updating the training data sets with security log data processed by the one or more engines.

18. The method of claim 16, further comprising dynamically analyzing sets of data from parsed or readable security logs and sets of data from raw or unstructured security logs.