Distributed behavior based anomaly detection

• US 9,961,096 B1
• Filed: 09/17/2014
• Issued: 05/01/2018
• Est. Priority Date: 09/17/2013
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Pierce, John Clifton
• Primary Examiner(s): Nguyen, Trong
• Application Number:
  US14/489,152
• Time in Patent Office: 1,322 Days
• Field of Search: 726 22- 25
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Systems and methods for detecting behavior-based anomalies are described herein. In various embodiments, the system includes a context engine for creating behavioral vectors that are transmitted to a long term data store, to behavioral engines configured to create baselines based on historical data, and to sensors configured to observe system resources. According to particular embodiments, the system is configured to collect data regarding the system resources (e.g., via the sensors) and compare the collected data to baselines to determine whether anomalies have occurred.

Claims

1. A system for detecting anomalies, the system comprising:

a memory and at least one processor operatively connected to at least one database, wherein the at least one processor is configured for;

receiving one or more behavioral vector parameters for each of a plurality of metrics from one or more users;

creating a plurality of individual and independent behavioral vectors, wherein each behavioral vector of the plurality of individual and independent behavioral vectors is for a respective metric of the plurality of metrics and is created based on the received one or more behavioral vector parameters for the respective metric of the plurality of metrics, and each behavioral vector of the plurality of individual and independent behavioral vectors comprises identifiers of a plurality of subject computing devices, the respective metric of the plurality of metrics, and an interval;

creating a plurality of baselines, wherein each baseline of the plurality of baselines is for a particular behavioral vector of the plurality of individual and independent behavioral vectors and comprises an expected value for the respective metric of the plurality of metrics collected over the interval, the expected value for the respective metric of the plurality of metrics being determined based on data associated with the particular behavioral vector of the plurality of individual and independent behavioral vectors;

transmitting the plurality of individual and independent behavioral vectors and the plurality of baselines to a plurality of sensors, wherein each sensor of the plurality of sensors is associated with a particular subject computing device of the plurality of subject computing devices, and

wherein the particular subject computing device of the plurality of subject computing devices is associated with a user of the one or more users; and

receiving, from each sensor of the plurality of sensors, a result comprising an observed value for one or more metrics of the plurality of metrics collected over the interval at the particular subject computing device of the plurality of subject computing devices and a comparison, for each of the one or more metrics of the plurality of metrics collected over the interval, of the observed value and the expected value, wherein the comparison identifies a behavioral anomaly at the particular subject computing device of the plurality of subject computing devices.

2. The system of claim 1, wherein:

the at least one processor is further configured for retrieving historical data from the at least one database based on at least one behavioral vector of the plurality of individual and independent behavioral vectors; and

the creating the baseline for the at least one behavioral vector of the plurality of individual and independent behavioral vectors is based on the historical data.

3. The system of claim 2, wherein the at least one processor is further configured for:

determining a standard deviation of the expected value based on the historical data; and

transmitting the standard deviation to the plurality of sensors.

4. The system of claim 3, wherein the result further comprises, for each metric of the plurality of metrics, a comparison of the observed value to the expected value in terms of the standard deviation.

5. The system of claim 2, wherein the at least one processor is further configured for receiving an alert from one or more sensors of the plurality of sensors, the alert indicating an anomaly associated with the comparison of the observed value and the expected value for the one or more metrics of the plurality of metrics collected over the interval by the one or more sensors of the plurality of sensors.

6. The system of claim 5, wherein the alert comprises an indication that the observed value is outside of a predetermined range of the expected value.

7. The system of claim 6, wherein:

the at least one processor is further configured for determining a standard deviation of the expected value based on the historical data and transmitting the standard deviation to the plurality of sensors; and

the predetermined range of the expected value comprises two standard deviations.

8. The system of claim 1, wherein the at least one processor is further configured for:

storing the result at the at least one database;

creating a second plurality of baselines based on the plurality of individual and independent behavioral vectors, each of the second plurality of baselines comprising a second expected value for the respective metric of the plurality of metrics collected over the interval, wherein the second expected value is at least partially derived from the result; and

transmitting the second plurality of baselines to the plurality of sensors.

9. The system of claim 8, wherein the plurality of sensors includes geographically disparate sensors.

10. The system of claim 8, wherein each of the plurality of sensors is associated with a discrete enterprise network system.

11. A computer-implemented method for detecting anomalies by at least one processor and at least one database, the computer-implemented method comprising steps of:

receiving, via the at least one processor, one or more behavioral vector parameters for each of a plurality of metrics from one or more users;

creating, via the at least one processor, a plurality of individual and independent behavioral vectors, wherein each behavioral vector of the plurality of individual and independent behavioral vectors is for a respective metric of the plurality of metrics and is created based on the received one or more behavioral vector parameters for the respective metric of the plurality of metrics, and each behavioral vector of the plurality of individual and independent behavioral vectors comprises identifiers of a plurality of subject computing devices, the respective metric of the plurality of metrics, and an interval;

creating, via the at least one processor, a plurality of baselines, wherein each baseline of the plurality of baselines is for a particular behavioral vector of the plurality of individual and independent behavioral vectors and comprises an expected value for the respective metric of the plurality of metrics collected over the interval, the expected value for the respective metric of the plurality of metrics being determined based on data associated with the particular behavioral vector of the plurality of individual and independent behavioral vectors;

transmitting, via the at least one processor, the plurality of individual and independent behavioral vectors and the plurality of baselines to a plurality of sensors, wherein each sensor of the plurality of sensors is associated with a particular subject computing device of the plurality of subject computing devices, and wherein the particular subject computing device of the plurality of subject computing devices is associated with a user of the one or more users; and

receiving, from each sensor of the plurality of sensors, via the at least one processor, a result comprising an observed value for one or more metrics of the plurality of metrics collected over the interval at the particular subject computing device of the plurality of subject computing devices and a comparison, for each of the one or more metrics of the plurality of metrics collected over the interval, of the observed value and the expected value, wherein the comparison identifies a behavioral anomaly at the particular subject computing device of the plurality of subject computing devices.

12. The computer-implemented method of claim 11, wherein:

the computer-implemented method further comprises steps of retrieving historical data from the at least one database based on at least one behavioral vector of the plurality of individual and independent behavioral vectors; and

the creating the baseline for the at least one behavioral vector of the plurality of individual and independent behavioral vectors is based on the historical data.

13. The computer-implemented method of claim 12, wherein the computer-implemented method further comprises steps of:

determining, via the at least one processor, a standard deviation of the expected value based on the historical data; and

transmitting, via the at least one processor, the standard deviation to the plurality of sensors.

14. The computer-implemented method of claim 13, wherein the result further comprises, for each metric of the plurality of metrics, a comparison of the observed value to the expected value in terms of the standard deviation.

15. The computer-implemented method of claim 12, wherein the computer-implemented method further comprises a step of receiving an alert from one or more sensors of the plurality of sensors, the alert indicating an anomaly associated with the comparison of the observed value and the expected value for the one or more metrics of the plurality of metrics collected over the interval by the one or more sensors of the plurality of sensors.

16. The computer-implemented method of claim 15, wherein the alert comprises an indication that the observed value is outside of a predetermined range of the expected value.

17. The computer-implemented method of claim 16, wherein:

the computer-implemented method further comprises step of determining a standard deviation of the expected value based on the historical data and transmitting the standard deviation to the plurality of sensors; and

the predetermined range of the expected value comprises two standard deviations.

18. The computer-implemented method of claim 11, wherein the computer-implemented method further comprises steps of:

storing, via the at least one processor, the result at the at least one database;

creating, via the at least one processor, a second plurality of baselines based on the plurality of individual and independent behavioral vectors, each of the second plurality of baselines comprising a second expected value for the respective metric of the plurality of metrics collected over the interval, wherein the second expected value is at least partially derived from the result; and

transmitting, via the at least one processor, the second plurality of baselines to the plurality of sensors.

19. The computer-implemented method of claim 18, wherein the plurality of sensors includes geographically disparate sensors.

20. The computer-implemented method of claim 18, wherein each of the plurality of sensors is associated with a discrete enterprise network system.

21. A sensor for behavior-based anomaly detection comprising at least one hardware processor, the at least one hardware processor configured for:

receiving, at the hardware processor of the sensor, a plurality of individual and independent behavioral vectors and an indication of one or more computing resources of a plurality of computing resources to observe while additional sensors observe other computing resources of the plurality of computing resources;

for each of the plurality of individual and independent behavioral vectors, collecting an observation comprising raw data from the one or more computing resources for a first time interval;

receiving a baseline for each of the plurality of individual and independent behavioral vectors, wherein the baseline for each of the plurality of individual and independent behavioral vectors comprises an expected value of the collected observation and a standard deviation of the expected value;

comparing, for each of the plurality of individual and independent behavioral vectors, the collected observation to the expected value;

determining whether a behavioral anomaly has occurred based on the comparing, wherein the behavioral anomaly occurs when the collected observation for a particular behavioral vector of the plurality of individual and independent behavioral vectors is a predetermined amount more or less than the expected value for the particular behavioral vector of the plurality of individual and independent behavioral vectors;

upon determining that the behavioral anomaly has occurred, forwarding information about the behavioral anomaly to a context engine to determine whether the behavioral anomaly is a true anomaly, wherein the context engine is configured to receive additional anomalies from the additional sensors in parallel to processing of the behavioral anomaly; and

transmitting, to a user, an alert that identifies a particular computing resource from the one or more computing resources where the behavioral anomaly has occurred.

22. The sensor of claim 21, wherein each behavioral vector of the plurality of individual and independent behavioral vectors comprises a metric to observe at the one or more computing resources.

23. The sensor of claim 22, wherein the metric comprises a total number of bytes transmitted and received.

24. The sensor of claim 22, wherein the metric comprises a total number of packets transmitted and received by the one or more computing resources.

25. The sensor of claim 21, wherein the behavioral anomaly occurs when the collected observation for the particular behavioral vector of the plurality of individual and independent behavioral vectors is at least one standard deviation more or less than the expected value for the particular behavioral vector of the plurality of individual and independent behavioral vectors.

26. The sensor of claim 21, wherein the behavioral anomaly occurs when the collected observation for the particular behavioral vector of the plurality of individual and independent behavioral vectors is at least two standard deviations more or less than the expected. value for the particular behavioral vector of the plurality of individual and independent behavioral vectors.

27. The sensor of claim 21, wherein the sensor is further configured for transmitting results of the comparing to a data store.

28. The sensor of claim 27, wherein the sensor is further configured for receiving, for each of the plurality of individual and independent behavioral vectors, a second baseline comprising a second expected value of a second observation and a second standard deviation of the second expected value, wherein the second observation comprises raw data collected from the one or more computing resources over a second time interval;

and the second expected value is based at least in part upon the results of the comparing.

29. The sensor of claim 28, wherein the sensor is further configured for:

determining whether a second anomaly has occurred by comparing the second observation to the second expected value for each of the plurality of individual and independent behavioral vectors, wherein the second anomaly occurs when the second observation is a predetermined amount more or less than the second expected value; and

upon determining that the second anomaly has occurred, transmitting a second alert to the user.

30. A computer-implemented method for behavior-based anomaly detection comprising:

providing a sensor for detecting anomalies in network traffic, the sensor comprising at least one processor, wherein the at least one processor is configured for performing steps of;

receiving a plurality of individual and independent behavioral vectors and an indication of one or more computing resources of a plurality of computing resources to observe while additional sensors observe other computing resources of the plurality of computing resources;

for each of the plurality of individual and independent behavioral vectors, collecting, at the sensor, an observation comprising raw data from the one or more computing resources for a first time interval;

receiving, for each of the plurality of individual and independent behavioral vectors, a baseline comprising an expected value of the collected observation and a standard deviation of the expected value;

comparing, for each of the plurality of individual and independent behavioral vectors, the collected observation to the expected value;

determining whether a behavioral anomaly has occurred based on the comparing, wherein the behavioral anomaly occurs when the collected observation for a particular behavioral vector of the plurality of individual and independent behavioral vectors is a predetermined amount more or less than the expected value for the particular behavioral vector of the plurality of individual and independent behavioral vectors;

upon determining that the behavioral anomaly has occurred, forwarding information about the behavioral anomaly to a context engine to determine whether the behavioral anomaly is a true anomaly, wherein the context engine is configured to receive additional anomalies from the additional sensors in parallel to processing of the behavioral anomaly; and

transmitting, to a user, an alert that identifies a particular computing resource from the one or more computing resources where the behavioral anomaly has occurred.

31. The computer-implemented method of claim 30, wherein each behavioral vector of the plurality of individual and independent behavioral vectors comprises a metric to observe at the one or more computing resources.

32. The computer-implemented method of claim 31, wherein the metric comprises a total number of bytes transmitted and received.

33. The computer-implemented method of claim 31, wherein the metric comprises a total number of packets transmitted and received by the one or more computing resources.

34. The computer-implemented method of claim 30, wherein the behavioral anomaly occurs when the collected observation for the particular behavioral vector of the plurality of individual and independent behavioral vectors is at least one standard deviation more or less than the expected value for the particular behavioral vector of the plurality of individual and independent behavioral vectors.

35. The computer-implemented method of claim 30, wherein the behavioral anomaly occurs when the collected observation for the particular behavioral vector of the plurality of individual and independent behavioral vectors is at least two standard deviations more or less than the expected value for the particular behavioral vector of the plurality of individual and independent behavioral vectors.

36. The computer-implemented method of claim 30, wherein the at least one processor is further configured for transmitting results of the comparing to a data store.

37. The computer-implemented method of claim 36, wherein the at least one processor is further configured for receiving, for each of the plurality of individual and independent behavioral vectors, a second baseline comprising a second expected value of a second observation and a second standard deviation of the second expected value, wherein:

the second observation comprises raw data collected from the one or more computing resources over a second time interval; and

the second expected value is based at least in part upon the results of the comparing.

38. The computer-implemented method of claim 37, wherein the at least one processor is further configured for:

determining whether a second anomaly has occurred by comparing the second observation to the second expected value for each of the plurality of individual and independent behavioral vectors, wherein the second anomaly occurs when the second observation is a predetermined amount more or less than the second expected value; and

upon determining that the second anomaly has occurred, transmitting a second alert to the user.

39. The computer-implemented method of claim 38, wherein the predetermined amount is at least two standard deviations.

40. The computer-implemented method of claim 38, wherein the predetermined amount is at least one standard deviation.

41. A system for detecting behavior-based anomalies, the system comprising:

a context engine configured for creating a plurality of individual and independent behavioral vectors based on one or more behavioral vector parameters received from one or more users, each of the plurality of individual and independent behavioral vectors comprising identifiers of a plurality of subject computing devices, a metric indicating data to collect from the plurality of subject computing devices, and an interval indicating a duration during which the data is to be collected from the plurality of subject computing devices;

a behavioral engine operatively connected to the context engine, the behavioral engine configured for creating a plurality of baselines based on the plurality of individual and independent behavioral vectors, wherein each baseline of the plurality of baselines is associated with one of the plurality of individual and independent behavioral vectors and comprises an expected value for respective data collected from the plurality of subject computing devices over the interval and a standard deviation of the expected value for the respective data; and

a plurality of sensors, wherein each sensor of the plurality of sensors is associated with a particular subject computing device of the plurality of subject computing devices that is associated with a user of the one or more users and each sensor of the plurality of sensors is configured for collecting data from the particular subject computing device based on the plurality of individual and independent behavioral vectors and comparing data collected for a particular behavioral vector of the plurality of individual and independent behavioral vectors to a particular baseline of the plurality of baselines associated with the particular behavioral vector to identify a behavioral anomaly at the particular subject computing device.

42. The system of claim 41, wherein at least one of the plurality of subject computing devices is a system network resource.

43. The system of claim 41, wherein data collected by the plurality of sensors from the plurality of subject computing devices is expressed in bytes.

44. The system of claim 41, wherein the comparing comprises comparing the data collected for the particular behavioral vector to the expected value in the particular baseline of the plurality of baselines.

45. The system of claim 41, wherein the comparing further comprises comparing the data collected for the particular behavioral vector to the standard deviation in the particular baseline of the plurality of baselines.

46. The system of claim 41, wherein the context engine is further configured for transmitting the plurality of baseline to the behavioral engine and the plurality of sensors.

47. The system of claim 41, wherein the context engine is further configured for receiving an alert from one or more sensor of the plurality of sensors when the behavioral anomaly is identified.

48. The system of claim 47, wherein:

the expected value in the particular baseline of the plurality of baselines is determined based on the particular behavioral vector and the standard deviation in the particular baseline of the plurality of baselines is determined based on historical data;

the comparing comprises comparing the data collected for the particular behavioral vector to a predetermined number of standard deviations;

the behavioral anomaly is identified when the data collected for the particular behavioral vector by a particular sensor of the one or more sensors is more than the predetermined number of standard deviations from the expected value in the particular baseline of the plurality of baselines; and

the alert comprises an indication that the behavioral anomaly has been identified by the particular sensor.

49. The system of claim 48, wherein:

the system further comprises a database operatively connected to the context engine, the behavioral engine, and the plurality of sensors; and

the historical data is based on results received by the database from one or more selected sensors of the plurality of sensors.

50. The system of claim 49, wherein the one or more selected sensors do not include the particular sensor.

51. The system of claim 49, wherein the one or more selected sensors include the particular sensor.

52. A computer-implemented method for detecting behavior-based anomalies, the method comprising:

providing a context engine configured for creating a plurality of individual and independent behavioral vectors based on one or more behavioral vector parameters received from one or more users, each of the plurality of individual and independent behavioral vectors comprising identifiers of a plurality of subject computing devices, a metric indicating data to collect from the plurality of subject computing devices, and an interval indicating a duration during which the data is to be collected from the plurality of subject computing devices;

providing a behavioral engine operatively connected to the context engine, the behavioral engine configured for creating a plurality of baselines based on the plurality of individual and independent behavioral vectors, wherein each baseline of the plurality of baselines is associated with one of the plurality of individual and independent behavioral vectors and comprises an expected value for respective data collected from the plurality of subject computing devices over the interval and a standard deviation of the expected value for the respective data; and

providing a plurality of sensors, wherein each sensor of the plurality of sensors is associated with a particular subject computing device of the plurality of subject computing devices that is associated with a user of the one or more users and each sensor of the plurality of sensors is configured for collecting data from the particular subject computing device based on the plurality of individual and independent behavioral vectors and comparing data collected for a particular behavioral vector of the plurality of individual and independent behavioral vectors to a particular baseline of the plurality of baselines associated with the particular behavioral vector to identify a behavioral anomaly at the particular subject computing device.

53. The computer-implemented method of claim 52, wherein at least one of the plurality of subject computing devices is a system network resource.

54. The computer-implemented method of claim 52, wherein data collected by the plurality of sensors from the plurality of subject computing devices is expressed in bytes.

55. The computer-implemented method of claim 52, wherein the comparing comprises comparing the data collected for the particular behavioral vector to the expected value in the particular baseline of the plurality of baselines.

56. The computer-implemented method of claim 52, wherein the comparing further comprises comparing the data collected for the particular behavioral vector to the standard deviation in the particular baseline of the plurality of baselines.

57. The computer-implemented method of claim 52, wherein the context engine is further configured for transmitting the plurality of baselines to the behavioral engine and the plurality of sensors.

58. The computer-implemented method of claim 52, wherein the context engine is further configured for receiving an alert from one or more sensor of the plurality of sensors when the behavioral anomaly is identified.

59. The computer-implemented method of claim 58, wherein:

the expected value in the particular baseline of the plurality of baselines is determined based on the particular behavioral vector and the standard deviation in the particular baseline of the plurality of baselines is determined based on historical data;

the comparing comprises comparing the data collected for the particular behavioral vector to a predetermined number of standard deviations;

the behavioral anomaly is identified when the data collected for the particular behavioral vector by a particular sensor of the one or more sensors for the one behavioral vector is more than the predetermined number of standard deviations from the expected value in the particular baseline of the plurality of baselines; and

the alert comprises an indication that the behavioral anomaly has been identified by the particular sensor.

60. The computer-implemented method of claim 59, wherein:

the computer-implemented method further comprises providing a database operatively connected to the context engine, the behavioral engine, and the plurality of sensors; and

the historical data is based on results received by the database from one or more selected sensors of the plurality of sensors.

61. The computer-implemented method of claim 60, wherein the one or more selected sensors do not include the particular sensor.

62. The computer-implemented method of claim 60, wherein the one or more selected sensors include the particular sensor.

63. A system for detecting behavior-based anomalies, the system comprising:

a context engine configured for creating a plurality of individual and independent behavioral vectors, each behavioral vector of the plurality of individual and independent behavioral vectors comprising an indication of a plurality of system network resources, a metric indicating data to collect from the plurality of system network resources, and an interval indicating a duration during which the data is to be collected from the plurality of system network resources;

a behavioral engine operatively connected to the context engine, the behavioral engine configured for creating a baseline based on a particular behavioral vector of the plurality of individual and independent behavioral vectors, the baseline comprising an expected value for the data collected from the plurality of system network resources and a standard deviation of the expected value, wherein the expected value is based upon data received by the behavioral engine from a data store; and

a plurality of sensors associated with the plurality of system network resources, each of the plurality of sensors is configured to receive two or more of the plurality of individual and independent behavioral vectors, and for each respective behavioral vector of the two or more behavioral vectors, each sensor of the plurality of sensors is configured to;

collect data from a particular system network resource of the plurality of system network resources in accordance with the indication, the metric, and the interval in the respective behavioral vector;

create an observation based on the data collected from the particular system network resource for the interval in the respective behavioral vector; and

identify that an anomaly has occurred at the particular system network resource by determining whether the observation is a predetermined number of standard deviations from the expected value.

64. The system of claim 63, wherein the predetermined number of standard deviations is at least two standard deviations.