SYSTEM AND METHOD FOR AUTOMATED NETWORK MONITORING AND DETECTION OF NETWORK ANOMALIES

• US 20180020015A1
• Filed: 06/14/2017
• Published: 01/18/2018
• Est. Priority Date: 06/14/2016
• Status: Active Grant

• Current Assignee: SDN Systems LLC (InterCloud Systems, Inc.)
• Original Assignee: SDN Systems LLC (InterCloud Systems, Inc.)
• Inventors: Munro, Mark, Babenko, Konstantin, Yatskiv, Taras, Makeev, Alexander
• Granted Patent: US 10,404,732 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 17/16 Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002

Abstract

A system and method for unsupervised detection of system anomalies in a network, including one or more network elements, a flow collector configured to collect instances of network data from the one or more network elements, a historical dataset database configured to store the instances of network data, a historical dataset pattern extractor configured to analyze the instances of network data and produce a historical behavioral pattern for each of the instances of network data, and a flow stream processor configured to analyze instances of network data in real time, produce a current behavioral pattern for each of the instances of network data, compare the current behavioral pattern to a corresponding historical behavioral pattern, and detect an anomaly based on the comparison between the current behavioral pattern and the corresponding historical behavioral pattern.

Claims

1. A system for unsupervised detection of system anomalies in a network, comprising:

one or more network elements;

a flow collector configured to collect instances of network data from the one or more network elements;

a historical dataset database configured to store the instances of network data;

a historical dataset pattern extractor configured to analyze the instances of network data and produce a historical behavioral pattern for each of the instances of network data; and

a flow stream processor configured to analyze instances of network data in real time, produce a current behavioral pattern for each of the instances of network data, compare the current behavioral pattern to a corresponding historical behavioral pattern, and detect an anomaly based on the comparison between the current behavioral pattern and the corresponding historical behavioral pattern.

2. The system of claim 1, wherein the network data comprises a timestamp, a session duration, a protocol, a source IP address, a source port, a destination IP address, a destination port, a number of bytes, a number of packets, and a number of transmission control protocol (TCP) flags.

3. The system of claim 1, wherein the historical dataset pattern extractor is configured to process the instances of network data by an unsupervised clustering algorithm.

4. The system of claim 1, wherein the historical dataset pattern extractor is configured to produce a metric threshold value for each of the instances of network data, wherein a value based on the current behavioral pattern in excess of the metric threshold value is indicative of a potential system anomaly.

5. The system of claim 4, further comprising a historical model storage database for storing historical models based on the historical behavioral pattern and the metric threshold value.

6. The system of claim 1, wherein the historical behavioral pattern and the current behavioral pattern are represented as respective vectors, wherein the comparison between the current behavioral pattern and the corresponding historical behavioral pattern is a calculation of standardized Euclidean distance between the respective vectors.

7. The system of claim 1, wherein the flow stream processor is configured to produce a notification upon detection of an anomaly.

8. The system of claim 7, wherein the notification is stored in an anomalies storage database and sent to a system administrator.

9. The system of claim 7, wherein the notification is sent to a rules engine, wherein the rules engine is configured to select a response to the notification.

10. The system of claim 9, wherein the rules engine is configured to convert the notification into a custom behavioral pattern and store the custom behavioral pattern in a historical dataset pattern storage database.

11. A method for unsupervised detection of system anomalies in a network, comprising:

collecting instances of network data from one or more network elements in the network in a flow collector;

storing the instances of network data in a historical dataset database;

analyzing the instances of network data and producing a historical behavioral pattern for each of the instances of network data by a historical dataset pattern extractor;

analyzing the instances of network data in real time and producing a current behavioral pattern for each of the instances of the network data by a flow stream processor;

comparing the current behavioral pattern to a corresponding historical behavioral pattern; and

detecting an anomaly based on the comparison between the current behavioral pattern and the corresponding historical behavioral pattern.

12. The method of claim 11, wherein the network data comprises a timestamp, a session duration, a protocol, a source IP address, a source port, a destination IP address, a destination port, a number of bytes, a number of packets, and a number of transmission control protocol (TCP) flags.

13. The method of claim 11, wherein the historical dataset pattern extractor is configured to process the instances of network data by an unsupervised clustering algorithm.

14. The method of claim 11, wherein the historical dataset pattern extractor is configured to produce a metric threshold value for each of the instances of network data, wherein a value based on the current behavioral pattern in excess of the metric threshold value is indicative of a potential system anomaly.

15. The method of claim 14, further comprising storing historical models based on the historical behavioral pattern and the metric threshold values by a historical model storage database.

16. The method of claim 11, wherein the historical behavioral pattern and the current behavioral pattern are represented as respective vectors, wherein the comparison between the current behavioral pattern and the corresponding historical behavioral pattern is a calculation of standardized Euclidean distance between the respective vectors.

17. The method of claim 11, wherein the flow stream processor is configured to produce a notification upon detection of an anomaly.

18. The method of claim 17, wherein the notification is stored in an anomalies storage database and sent to a system administrator.

19. The method of claim 17, wherein the notification is sent to a rules engine, wherein the rules engine is configured to select a response to the notification.

20. The method of claim 19, wherein the rules engine is configured to convert the notification into a custom behavioral pattern and store the custom behavioral pattern in a historical dataset pattern storage database.