METHODS AND APPARATUS FOR ANALYZING SYSTEM EVENTS

• US 20150213358A1
• Filed: 02/03/2015
• Published: 07/30/2015
• Est. Priority Date: 11/17/2009
• Status: Active Grant

• Current Assignee: Hawk Network Defense Inc.
• Original Assignee: Hawk Network Defense Inc.
• Inventors: Shelton, Tim, Harris, David, Wheeler, Todd Jason Jr.
• Granted Patent: US 9,866,426 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 41/0636 based on a decision tree analysis
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H04L 67/30 Profiles

Abstract

Apparatus and methods facilitate analysis of events associated with network and computer systems. Event data, such as security threats, are comparison matched with event rules of event rule sets associated with each network or computer system to determine whether the items are potentially significant. Additionally, the system-event data may be scored where the score is used for prioritizing system-event data as to their significance. Associated with the comparison matching are various analytics that further analyze event data for measuring and analyzing the system-event data according to various algorithms.

Claims

1. A method for use in analyzing system events for one or more network systems or computer systems, the method comprising:

identifying system-event data resulting from system or network events occurring on one or more computer systems that match at least one event rule of an event-rule set in the one or more computer systems or network systems;

normalizing the identified system-event data; and

analyzing the system-event data including at least one of;

determining comparison matching of rules pertaining to the system-event data;

using at least a scoring rule of a scoring-rule configured to assign a score to the system-event data, wherein the score is operable for prioritizing system-event data; and

performing data measurements based on the system-event data.

2. The method as defined in claim 1, wherein analyzing the system-event data further comprises:

generating a matching array having labels assigned for one or more nodes in a Boolean expression tree used for Boolean rule matching; and

determining whether at least one score matches a rule based on analyzing intervals of the matching array.

3. The method as defined in claim 1, wherein analyzing the system-event data further comprises:

reducing a plurality of Boolean expression trees used for Boolean rule matching using a cross relation analysis to remove duplications;

generating a unique hash for each of the plurality of Boolean expression trees representing the Boolean expression tree;

comparing whether the generated hashes match predetermined rules; and

caching the comparison results.

4. The method as defined in claim 1, wherein analyzing the system-event data further comprises:

determining one or more key-value pairs;

assigning a distinct counting for each key-value pair to thereby configure a respective unique key-value pair;

determining for each configured unique key-value pair, if a distinct event value associated with the unique key-value pair exists in a database;

using the distinct event value for comparison if the key-value is determined to already exist; and

accessing a cached HyperLogLog state table if the distinct event value does not exist to perform a HyperLogLog operation if the state table exists for updating the table or creating a new HyperLogLog table if a cached table did not exist; and

perform a comparison operation on the key-value.

5. The method as defined in claim 1, wherein analyzing the system-event data further comprises:

building a balanced AVL tree configured for at least one of IPv4, IPv6, or CIDR address notation comparison, the AVL tree comprising a plurality of nodes having respective index values based on the input addresses;

comparing values comprising at least one of anIPv4, IPv6, or CIDR address input during a run-time stream process to respective index values in the AVL tree to determine whether a match exists.

6. The method as defined in claim 1, wherein analyzing the system-event data further comprises:

building at least one unique sketch profile for at least one input data source including at least one of an event, a netflow, or vulnerability data; and

caching the at least one unique sketch profile in a distributed cache.

7. The method as defined in claim 6, further comprising:

comparing the least one unique sketch profile cached in the distributed cache with a current input data source to determine whether a match exists there between; and

merging the input data source and the cached at least one unique sketch profile in the distributed cache when a match does not exist.

8. The method as defined in claim 1 wherein, analyzing the system-event data further comprises:

accessing a sketch/profile index stored in a distributed cache;

determining, for a given current input system event data, a probability that the current event data would occur in the accessed sketch/profile index given the existing events already sampled/sketched in creating the sketch/profile index;

assigning a profile key/value to the given current system event data with the determined probability.

10. The method as defined in claim 1, wherein analyzing the system-event data further comprises:

applying a further analysis in conjunction with the analysis of the system-event data, the further analysis comprising;

iterating each incident or event in a given collection of incidents using a predetermined statistical model;

determining commonalities in the collection of incidents based on the iteration; and

if the commonalities meet a preconfigured matching criteria, saving the model as a Boolean expression model as a recommended model to be used for Boolean expression tree comparison during correlation.

11. The method as defined in claim 1, wherein analyzing the system-event data further comprises:

creating at least one support vector machine (SVM) model based on a given set of system-event data;

determining for each event in the set where event updates a given sketch profile;

comparing the updated sketch to the at least one SVM model;

attaching SVM details and setting SVM flags if the comparison matches a predetermined threshold.

12. The method as defined in claim 1, wherein analyzing the system-event data further comprises:

applying a further analysis in conjunction with the analysis of the system-event data, the further analysis comprising;

configuring at least one automated unsupervised learning rule by setting specified search parameters;

determining whether the system-event data matches the specified search parameters;

learning the event as good or bad based on the at least one automated unsupervised learning rule; and

updating a database based on the learned event.

13. The method as defined in claim 1, wherein analyzing the system-event data further comprises:

applying an analysis in conjunction with the correlation, the analysis comprising;

applying a Natural Language Processing (NLP) process to build context information related to input data including events and social media data;

comparing the NLP context information with one or more preconfigured NLP context topics configured for monitoring and detecting similarity;

creating or updating incident and track changes if the comparison shows similarity, and an automated Boolean score based upon potential context clues provided in NLP context information.

14. The method as defined in claim 1, the normalization further comprising:

applying a Natural Language Processing (NLP) process to build context information concerning at least one input system log event; and

extracting the NLP context information for application to an available key-value pair.

15. The method as defined in claim 1, further comprising:

building at least one of user and network profiles based on logged event-system data.

9. The method of claim 9, wherein a Boolean rule/key used to score the current system event data, scores the data based at least in part upon the determined probability.

16. An apparatus for use in analyzing system events for one or more network systems or computer systems, the apparatus comprising:

an event matcher configured to identify system-event data resulting from system or network events occurring on one or more computer systems that match at least one event rule of an event-rule set in the one or more computer systems or network systems;

a normalizer configured to normalize the identified system-event data; and

an event comparison engine configured to analyze the system-event data through at least one of;

determining comparison matching of rules pertaining to the system-event data;

using at least a scoring rule of a scoring-rule configured to assign a score to the system-event data, wherein the score is operable for prioritizing system-event data; and

performing data measurements based on the system-event data.