REAL-TIME DETECTION AND CLASSIFICATION OF ANOMALOUS EVENTS IN STREAMING DATA

• US 20150106927A1
• Filed: 10/14/2013
• Published: 04/16/2015
• Est. Priority Date: 10/14/2013
• Status: Active Grant

• Current Assignee: UT-Battelle LLC
• Original Assignee: UT-Battelle LLC
• Inventors: Ferragut, Erik M., Goodall, John R., Iannacone, Michael D., Laska, Jason A., Harrison, Lane T.
• Granted Patent: US 9,319,421 B2
• Time in Patent Office: Days
• US Class Current: 726/23
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis

Abstract

A system is described for receiving a stream of events and scoring the events based on anomalousness and maliciousness (or other classification). The events can be displayed to a user in user-defined groupings in an animated fashion. The system can include a plurality of anomaly detectors that together implement an algorithm to identify low probability events and detect atypical traffic patterns. The atypical traffic patterns can then be classified as being of interest or not. In one particular example, in a network environment, the classification can be whether the network traffic is malicious or not.

Claims

1. A method of detecting and classifying anomalous events, comprising:

receiving an input log file including a plurality of events, wherein each event comprises a data set;

for each event, providing multiple contexts that group the data set into different sub-groups;

generating an anomaly score for each context so that each event has at least two anomaly scores associated therewith;

for each event, combining at least the anomaly scores to generate an overall event score so as to classify the event as being normal or abnormal; and

outputting a plurality of the overall event scores for the input log file.

2. The method of claim 1, wherein the overall event score is an indicator of whether the event is malicious network traffic.

3. The method of claim 1, wherein each data set includes at least a source and destination network address.

4. The method of claim 3, wherein the data set further includes a description of the event and a time when the event occurred.

5. The method of claim 1, wherein the log file is a continuous stream of events that are received in real time.

6. The method of claim 1, wherein combining the anomaly score for each context includes reading a policy document to determine weightings for the combination and/or functions used in the combination.

7. The method of claim 1, wherein combining the anomaly score for each context further includes using domain knowledge in the combination.

8. The method of claim 7, wherein using domain knowledge includes modifying functions used in the combination based on whether the event is targeting a protected resource, whether the event violates a network rule, and/or whether a role of a network machine associated with the event is unexpected.

9. The method of claim 1, further including for each event, transforming the data set associated with the event into a predetermined format.

10. The method of claim 1, wherein the outputting includes displaying the events as dots that travel across the display in time steps as the events are being received as streaming data.

11. The method of claim 10, wherein outputting further includes displaying at least one anomaly score and a maliciousness score associated with each event.

12. The method of claim 1, wherein the combining is a linear combination of the anomaly scores using weighted inputs.

13. The method of claim 1, wherein the combining using a history of a previous event.

14. A computer-readable medium having instructions thereon for executing a method of detecting and classifying anomalous events, the method comprising:

receiving a plurality of input network events;

transforming the input network events so that each event is associated with a position parameter, which includes at least a source and destination network address, a value parameter, which is a description of the event, and a time parameter indicating when the event occurred;

generating a plurality of anomaly detectors that aggregate different sub-parts of the position, value and time parameters;

calculating a plurality of anomaly scores for each input network event using the anomaly detectors; and

calculating a maliciousness score for each event using the plurality of anomaly scores.

15. The computer-readable medium of claim 14, wherein calculating the maliciousness score includes using a linear combination of the anomaly scores.

16. The computer-readable medium of claim 15, further including reading a file including weighting information used in the linear combination.

17. The computer-readable medium of claim 14, wherein calculating the plurality of anomaly scores includes retrieving previous state information associated with a history of the event.

18. The computer-readable medium of claim 14, wherein generating a plurality of anomaly detectors includes using one or more position sub-parts including one or more of the following:

source port address, destination port address, and network protocol.

19. A system for detecting and classifying anomalous events, comprising:

a data model for receiving events and transforming the events into a common format;

a plurality of anomaly detectors for receiving the events from the data model and generating a plurality of anomaly scores associated with each event; and

a classifier coupled to the anomaly detectors for generating a maliciousness score based on the plurality of anomaly scores.

20. The system of claim 19, wherein the classifier receives network policy and network role data in generating the maliciousness score.

21. The system of claim 19, wherein the user can control weights used in generating the maliciousness score.