Real-time detection and classification of anomalous events in streaming data

• US 9,319,421 B2
• Filed: 10/14/2013
• Issued: 04/19/2016
• Est. Priority Date: 10/14/2013
• Status: Active Grant

• Current Assignee: UT-Battelle LLC
• Original Assignee: UT-Battelle LLC
• Inventors: Ferragut, Erik M., Goodall, John R., Iannacone, Michael D., Laska, Jason A., Harrison, Lane T.
• Primary Examiner(s): Traore, Fatoumata
• Application Number:
  US14/053,248
• Publication Number:
  US 20150106927A1
• Time in Patent Office: 918 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis

Abstract

A system is described for receiving a stream of events and scoring the events based on anomalousness and maliciousness (or other classification). The events can be displayed to a user in user-defined groupings in an animated fashion. The system can include a plurality of anomaly detectors that together implement an algorithm to identify low probability events and detect atypical traffic patterns. The atypical traffic patterns can then be classified as being of interest or not. In one particular example, in a network environment, the classification can be whether the network traffic is malicious or not.

Claims

1. A method of detecting and classifying anomalous events,

comprising:

receiving an input log file including a plurality of events, wherein each event comprises a data set;

for each event, providing multiple contexts that group the data set into different sub-groups, wherein one or more anomaly detectors are coupled to each context;

generating an anomaly score for each context by using each context'"'"'s one or more anomaly detectors, so that each event is associated with at least two anomaly scores generated for different contexts;

for each event, combining at least the at least two anomaly scores generated for different contexts to generate an overall event score so as to classify the event as being normal or abnormal, wherein combining the anomaly score for each context further includes using domain knowledge in the combination, wherein the using domain knowledge includes modifying functions used in the combination based on whether the event is targeting a protected resource, whether the event violates a network rule, and/or whether a role of a network machine associated with the event is unexpected; and

outputting a plurality of the overall event scores for the input log file, wherein the outputting includes;

displaying the events as dots that travel across a display in time steps as the events are being received as streaming data, and

displaying at least one anomaly score and a maliciousness score associated with each event.

2. The method of claim 1, wherein the overall event score is an indicator of whether the event is malicious network traffic.

3. The method of claim 1, wherein each data set includes at least a source and destination network address.

4. The method of claim 3, wherein the data set further includes a description of the event and a time when the event occurred.

5. The method of claim 1, wherein the log file is a continuous stream of events that are received in real time.

6. The method of claim 1, wherein combining the anomaly score for each context includes reading a policy document to determine weightings for the combination and/or functions used in the combination.

7. The method of claim 1, further including for each event, transforming the data set associated with the event into a predetermined format.

8. The method of claim 1, wherein the combining is a linear combination of the anomaly scores using weighted inputs.

9. The method of claim 1, wherein the combining using a history of a previous event.

10. A non-transitory computer-readable medium having instructions thereon for executing a method of detecting and classifying anomalous events, the method comprising:

receiving a plurality of input network events;

transforming the input network events so that each event is associated with a position parameter, which includes at least a source and destination network address, a value parameter, which is a description of the event, and a time parameter indicating when the event occurred;

generating a plurality of context definitions that aggregate different sub parts of the position, value and time parameters for each input network event;

generating a plurality of anomaly detectors, wherein each anomaly detector is coupled to one or more of the plurality of context definitions;

calculating a plurality of anomaly scores for each input network event using the anomaly detectors, wherein at least two of the plurality of anomaly scores are calculated using different anomaly detectors;

calculating a maliciousness score for each input network event using the plurality of anomaly scores; and

outputting a plurality of the maliciousness scores for the input network events, wherein the outputting includes;

displaying the input network events as dots that travel across a display in time steps as the input network events are being received as streaming data, and

displaying at least one anomaly score and the maliciousness score for each input network event.

11. The computer-readable medium of claim 10, wherein calculating the maliciousness score includes using a linear combination of the anomaly scores.

12. The computer-readable medium of claim 11, further including reading a file including weighting information used in the linear combination.

13. The computer-readable medium of claim 10, wherein calculating the plurality of anomaly scores includes retrieving previous state information associated with a history of the event.

14. The computer-readable medium of claim 10, wherein generating a plurality of anomaly detectors includes using one or more position sub-parts including one or more of the following:

source port address, destination port address, and network protocol.

15. A system for detecting and classifying anomalous events, comprising:

a data model for receiving events and transforming the events into a common format, wherein the transforming comprises transforming the events so that each event is associated with a position parameter, which includes at least a source and destination network address, a value parameter, which is a description of the event, and a time parameter indicating when the event occurred;

a plurality of anomaly detectors for receiving the events from the data model, such that each event is associated with more than one anomaly detector, and generating a plurality of anomaly scores associated with each event;

a classifier coupled to the anomaly detectors for generating a maliciousness score for each event based on the plurality of anomaly scores associated with each event; and

a display device configured to display the events as dots that travel across the display device in time steps as the events are being received as streaming data, and to display at least one anomaly score and the maliciousness score for each event.

16. The system of claim 15, wherein the classifier receives network policy and network role data in generating the maliciousness score.

17. The system of claim 15, wherein the user can control weights used in generating the maliciousness score.