Detection of anomalous events

• US 9,361,463 B2
• Filed: 12/11/2013
• Issued: 06/07/2016
• Est. Priority Date: 12/11/2013
• Status: Active Grant

• Current Assignee: UT-Battelle LLC
• Original Assignee: UT-Battelle LLC
• Inventors: Ferragut, Erik M., Laska, Jason A., Bridges, Robert A.
• Primary Examiner(s): Abyaneh, Ali
• Assistant Examiner(s): VICTORIA, NARCISO F
• Application Number:
  US14/103,703
• Publication Number:
  US 20150161394A1
• Time in Patent Office: 909 Days
• Field of Search: 726/25
• US Class Current: 1/1
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 9/3273 for mutual authentication network architectures or network communication protocols for achieving mutual authentication in a packet data network H04L63/0869

Abstract

A system is described for receiving a stream of events and scoring the events based on anomalousness and maliciousness (or other classification). The system can include a plurality of anomaly detectors that together implement an algorithm to identify low-probability events and detect atypical traffic patterns. The anomaly detector provides for comparability of disparate sources of data (e.g., network flow data and firewall logs.) Additionally, the anomaly detector allows for regulatability, meaning that the algorithm can be user configurable to adjust a number of false alerts. The anomaly detector can be used for a variety of probability density functions, including normal Gaussian distributions, irregular distributions, as well as functions associated with continuous or discrete variables.

Claims

1. A method comprising:

detecting anomalous events using a first anomaly detector positioned in parallel with a second anomaly detector, the detecting comprising;

receiving a first log file including a first plurality of events from a first data source in the first anomaly detector including a first memory;

receiving a second log file including a second plurality of events from a second data source that is a different type than the first data source in the second anomaly detector including a second memory;

using the first log file, generating a first anomaly score, the generation being derived from an area associated with a probability density function of the first log file, wherein generating the first anomaly score includes using the formula A_f(x)=−

log_bPf(f(X)≦

f(x)) where b>

1;

using the second log file, generating a second anomaly score, the generation being derived from an area associated with a probability density function of the second log file; and

comparing the first and second anomaly scores so as to compare anomalies from the first data source to the second data source, which are of different types.

2. The method of claim 1, wherein a function used to calculate the anomaly score is tunable through user input.

3. The method of claim 1, wherein the first and second log files include disparate event types.

4. The method of claim 1, wherein each of the first and second log files are streaming data.

5. The method of claim 1, wherein comparing the first and second anomaly scores includes generating a linear combination of the anomaly scores using weighted inputs.

6. A method comprising:

detecting anomalous events using a first anomaly detector positioned in parallel with a second anomaly detector, the detecting comprising;

receiving a first log file including a first plurality of events from a first data source in the first anomaly detector including a first memory;

receiving a second log file including a second plurality of events from a second data source that is a different type than the first data source in the second anomaly detector including a second memory;

using the first log file, generating a first anomaly score, the generation being derived from an area associated with a probability density function of the first log file;

using the second log file, generating a second anomaly score, the generation being derived from an area associated with a probability density function of the second log file; and

comparing the first and second anomaly scores so as to compare anomalies from the first data source to the second data source, which are of different types;

wherein generating the first anomaly score includes using the function P_f(A_f(x)≧

α

)≦

b<sup>−

α</sup>

wherein α

is a tunable parameter to change a number of false alerts and b is any number >

1.

7. The method of claim 6, wherein a function used to calculate the anomaly score is tunable through user input.

8. The method of claim 6, wherein the first and second log files include disparate event types.

9. The method of claim 6, wherein each of the first and second log files are streaming data.

10. The method of claim 6, wherein comparing the first and second anomaly scores includes generating a linear combination of the anomaly scores using weighted inputs.

11. A non-transitory computer-readable storage having instructions thereon for executing a method of detecting anomalous events, the method comprising:

receiving a plurality of input network events from disparate network sources in a plurality of anomaly detectors coupled in parallel;

receiving a first log file including a first plurality of the input network events from a first data source in a first anomaly detector of the plurality of anomaly detectors including a first memory;

receiving a second log file including a second plurality of the input network events from a second data source that is a different type than the first data source in a second anomaly detector of the plurality of anomaly detectors including a second memory;

calculating multiple anomaly scores using the plurality of anomaly detectors for each of the plurality of input network events using a function formed at least in part by the expression A_f(x)=−

log_bP_f(f(X)≦

f(x)) where b>

1 wherein f(X) is related to a probability of an occurrence of an event, f(x) is a current event being analyzed, and P_f is a probability determination so that the anomalous scores are comparable; and

comparing the multiple anomaly scores so as to compare anomalies from the first data source to the second data source.

12. The non-transitory computer-readable storage of claim 11, wherein the function used to calculate the anomaly scores is tunable.

13. The non-transitory computer-readable storage of claim 11, wherein receiving the plurality of input network events includes receiving multiple log files that include disparate event types.

14. The non-transitory computer-readable storage of claim 11, wherein receiving the plurality of input network events includes receiving the input network events as streaming data.

15. The non-transitory computer-readable storage of claim 11, further including comparing the anomaly scores, wherein the comparing of anomaly scores includes generating a linear combination of the anomaly scores using weighted inputs.

16. The non-transitory computer-readable storage of claim 11, wherein the network sources are selected from a list including one or more of the following:

a firewall, a workstation, a domain name system (DNS) server, and a web server.

17. The non-transitory computer-readable storage of claim 11, wherein the network events include a timestamp, a source Internet Protocol (IP) address, a destination IP address, a destination port, a protocol and a message code.

18. A non-transitory computer-readable storage having instructions thereon for executing a method, the method comprising:

detecting anomalous events, the detecting including;

receiving a plurality of input network events from disparate network sources in a plurality of anomaly detectors coupled in parallel;

receiving a first log file including a first plurality of input network events from a first data source in a first anomaly detector of the plurality of anomaly detectors;

receiving a second log file including a second plurality of input network events from a second data source that is a different type than the first data source in a second anomaly detector of the plurality of anomaly detectors;

generating multiple anomaly scores using the first and second of the plurality of anomaly detectors for each of the first and second plurality of input network events using a function P_f(A_f(x)≧

α

)b<sup>−

α</sup>

wherein α

is a tunable parameter to change a number of false alerts and b is any number >

1.

19. The non-transitory computer-readable storage of claim 18, wherein the function used to calculate the anomaly scores is tunable.

20. The non-transitory computer-readable storage of claim 18, wherein receiving the plurality of input network events includes receiving multiple log files that include disparate event types.

21. The non-transitory computer-readable storage of claim 18, wherein receiving the plurality of input network events includes receiving the input network events as streaming data.

22. The non-transitory computer-readable storage of claim 18, further including comparing the anomaly scores, wherein the comparing of anomaly scores includes generating a linear combination of the anomaly scores using weighted inputs.

23. The non-transitory computer-readable storage of claim 18, wherein the network sources are selected from a list including one or more of the following:

a firewall, a workstation, a domain name system (DNS) server, and a web server.

24. The non-transitory computer-readable storage of claim 18, wherein the network events include a timestamp, a source Internet Protocol (IP) address, a destination IP address, a destination port, a protocol and a message code.

25. A system for detecting anomalous events, comprising:

a first anomaly detector for receiving a first log file;

a second anomaly detector for receiving a second log file, the second anomaly detector coupled in parallel with the first anomaly detector, wherein both the first and second anomaly detectors include memory registers;

wherein the first and second anomaly detectors calculate anomaly scores for the respective first and second log files, the anomaly detectors using a function formed at least in part by the expression A_f(x)=−

log_bP_f(f(X)≦

f(x)) where b>

1, wherein f(X) is related to a probability of an occurrence of an event, f(x) is a current event being anal zed and P_f is a probability determination; and

a hardware comparator coupled to the anomaly detectors for comparing the anomaly scores.

26. The system of claim 25, wherein the first log file and the second log file are sourced from devices having different platforms.

27. The system of claim 25, wherein the first and second anomaly detectors are tunable to change a number of false alerts.

28. A system for detecting anomalous events, comprising:

a first anomaly detector for receiving a first log file;

a second anomaly detector for receiving a second log file, the second anomaly detector coupled in parallel with the first anomaly detector, wherein both the first and second anomaly detectors include memory registers;

wherein the first and second anomaly detectors calculate anomaly scores for the respective first and second log files, the anomaly detectors using a function P_f(A_f(x)≧

α

)b<sup>−

α</sup>

wherein α

is a tunable parameter to change a number of false alerts, b is any number >

1 and Pf is a probability determination; and

a hardware comparator coupled to the anomaly detectors for comparing the anomaly scores.

29. The system of claim 28, wherein the first log file and the second log file are sourced from devices having different platforms.

30. The system of claim 28, wherein the first and second anomaly detectors are tunable to change a number of false alerts.