Modeling and Outlier Detection in Threat Management System Data

• US 20120304288A1
• Filed: 05/26/2011
• Published: 11/29/2012
• Est. Priority Date: 05/26/2011
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Wright, Jeremy, Hogoboom, John, Spielman, Chaim
• Granted Patent: US 8,528,088 B2
• Time in Patent Office: Days
• US Class Current: 726/22
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Methods, systems, and computer-readable media for identifying potential threats on a network based on anomalous behavior in communication between endpoints are provided. Traffic data for a network is accumulated over some period of time. The traffic data is grouped by one or more keys, such as source IP address, and sets of metric values are calculated for the keys. A mixture distribution, such as a negative binomial mixture distribution, is fitted to each set of metric values, and outlying metric values are determined based on the mixture distribution(s). A list of outliers is then generated comprising key values having outlying metric values in one or more of the sets of metric values.

Claims

1. A method of identifying potential threats on a network comprising:

accumulating traffic data from the network over a period of time;

calculating a first set of metric values for endpoints communicating on the network from the traffic data;

fitting a first mixture distribution to the first set of metric values;

identifying outlying metric values based on the mixture distribution; and

generating a list of outliers comprising the endpoints having an outlying metric value.

2. The method of claim 1, further comprising clustering the endpoints in the list of outliers based on similar communication behavior patterns exhibited by the endpoints.

3. The method of claim 2, further comprising co-clustering across the endpoints with different key values based on a relationship between the endpoints represented in the traffic data.

4. The method of claim 1, wherein the first mixture distribution comprises a negative binomial mixture distribution.

5. The method of claim 1, wherein identifying the outlying metric values comprises calculating a fit quantity for each metric value in the first set of metric values indicating if the metric value would be generated by a high probability from dominant components of the first mixture distribution, and identifying the metric values having the fit quantity below a specific threshold as outliers.

6. The method of claim 1, further comprising calculating a perplexity of the first set of metric values and identifying outliers based on the perplexity of each metric value in the first set of metric values.

7. The method of claim 1, wherein a second set of metric values are calculated for the endpoints from the traffic data, wherein a second mixture distribution is fitted to the second set of metric values, and wherein the list of outliers comprises the endpoints having outlying metric values in the first set of metric values and the second set of metric values.

8. A system comprising:

a memory for storing a program containing computer-executable instructions for identifying potential threats on a network; and

a processor functionally coupled to the memory, the processor being responsive to the computer-executable instructions and operative to;

receive traffic data accumulated from the network over a period of time,

calculate a first set of metric values for endpoints communicating on the network from the traffic data,

fit a first mixture distribution to the first set of metric values,

identify outlying metric values based on the first mixture distribution, and

generate a list of outliers comprising the endpoints having outlying metric values.

9. The system of claim 8, wherein the processor is further operative to cluster the endpoints in the list of outliers based on similar communication behavior patterns exhibited by the endpoints.

10. The system of claim 9, wherein the processor is further operative to co-cluster across the endpoints with different key values based on a relationship between the endpoints represented in the traffic data.

11. The system of claim 8, wherein the first mixture distribution comprises a negative binomial mixture distribution.

12. The system of claim 8, wherein identifying the outlying metric values comprises calculating a fit quantity for each metric value in the first set of metric values indicating if the metric value would be generated by a high probability from dominant components of the first mixture distribution, and identifying the metric values having the fit quantity below a specific threshold as outliers.

13. The system of claim 8, wherein the processor is further operative to calculate a perplexity of the first set of metric values and identify outliers based on the perplexity of each metric value in the first set of metric value.

14. The system of claim 8, wherein the processor is operative to:

calculate a second set of metric values for the endpoints from the traffic data;

fit a second mixture distribution to the second set of metric values; and

generate a list of outliers comprising the endpoints having outlying metric values in the first set of metric values and the second set of metric values.

15. A computer-readable storage medium having computer-executable instructions stored thereon that, when executed by a computer, cause the computer to:

receive traffic data accumulated from a network;

calculate a first set of metric values for endpoints communicating on the network from the traffic data;

fit a negative binomial mixture distribution to the first set of metric values;

identify outlying metric values based on the negative binomial mixture distribution; and

generate a list of outliers comprising the endpoints having an outlying metric value.

16. The computer-readable storage medium of claim 15, having further computer-executable instructions that cause the computer to cluster the endpoints in the list of outliers based on similar communication behavior patterns exhibited by the endpoints.

17. The computer-readable storage medium of claim 16, having further computer-executable instructions that cause the computer to co-cluster across the endpoints with different key values based on a relationship between the endpoints represented in the traffic data.

18. The computer-readable storage medium of claim 15, wherein identifying the outlying metric values comprises calculating a fit quantity for each metric value in the first set of metric values indicating if the metric value would be generated by dominant components of the negative binomial mixture distribution and identifying the metric values having the fit quantity below a specific threshold as outliers.

19. The computer-readable storage medium of claim 15, having further computer-executable instructions that cause the computer to calculate a perplexity of the first set of metric values and identify outliers based on the perplexity of each of the first set of metric values.

20. The computer-readable storage medium of claim 15, having further computer-executable instructions that cause the computer to:

calculate a second set of metric values for the endpoints from the traffic data;

fit a second mixture distribution to the second set of metric values; and

generate a list of outliers comprising endpoints having outlying metric values in the first sets of metric values and the second set of metric values.