SYSTEMS AND METHODS FOR RULE-BASED ANOMALY DETECTION ON IP NETWORK FLOW

• US 20100153316A1
• Filed: 09/28/2009
• Published: 06/17/2010
• Est. Priority Date: 12/16/2008
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Ringberg, Haakon Andreas, Krishnamurthy, Balachander, Duffield, Nicholas, Haffner, Patrick
• Granted Patent: US 9,258,217 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: H04L 41/00 Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 45/28 using route fault recovery
• CPC Class Codes: H04L 45/38 Flow based routing
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/24 Traffic characterised by specific attributes, e.g. priority or QoS
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H04L 63/30 for supporting lawful interception, monitoring or retaining of communications or communication related information circuit switched telephony call monitoring H04M3/2281

Abstract

A system to detect anomalies in internet protocol (IP) flows uses a set of machine-learning (ML) rules that can be applied in real time at the IP flow level. A communication network has a large number of routers that can be equipped with flow monitoring capability. A flow collector collects flow data from the routers throughout the communication network and provides them to a flow classifier. At the same time, a limited number of locations in the network monitor data packets and generate alerts based on packet data properties. The packet alerts and the flow data are provided to a machine learning system that detects correlations between the packet-based alerts and the flow data to thereby generate a series of flow-level alerts. These rules are provided to the flow time classifier. Over time, the new packet alerts and flow data are used to provide updated rules generated by the machine learning system.

Claims

1. A system comprising:

a plurality of routers configured to route network traffic, each of the routers further configured to generate flow records pertain to the traffic flow through the respective router;

a flow collector in communication with the plurality of routers and receiving the flow records therefrom; and

a flow classifier configured to analyze the flow records from each of the plurality of routers and to identify an anomaly in traffic flow within the network.

2. The system of claim 1 wherein the plurality of routers provide flow data for every flow throughout the network.

3. The system of claim 1, further comprising a packet monitor system associated with a portion of the routers and configured to generate packet data related to packet of network traffic flowing through the portion of the plurality of routers.

4. The system of claim 3, further comprising a machine-learning trainer communicatively coupled to the flow collector and the packet monitor system and configured to generate flow-level rules based on the packet data and the flow data, the flow classification being configured to use the flow-level rules to identify an anomaly in the network traffic.

5. The system of claim 3, further comprising a plurality of machine-learning rule trainers communicatively coupled to the flow collector and the packet monitor system and configured to operate in parallel to generate flow-level rules based on the packet data and the flow data.

6. The system of claim 5 wherein each of the plurality of machine-learning rule trainers is configured to generate a portion of the flow-level rules using selected packet data and the packet-based rules associated with the selected packet data.

7. The system of claim 4 wherein the machine-learning rule trainer is further configured to periodically receive additional flow data from the flow collector and additional packet data and associated packet data rules from the packet monitor system and configured to generate revised flow-level rules based on the additional packet data and the additional flow data.

8. The system of claim 4 wherein the machine-learning rule trainer is further configured to receive additional flow data from the flow collector and additional packet data and associated packet data rules from the packet monitor system and configured to generate revised flow-level rules based on the additional packet data and the additional flow data.

9. The system of claim 4 wherein the packet data has associated alarms based on the packet data, the system further comprising a machine-learning rule trainer communicatively coupled to the flow collector and the packet monitor system to generate flow-level rules that cause alarms that are at least partially equivalent to the associated packet alarms.

10. The system of claim 9 wherein the associated alarms are based on a set of packet rule classification predicates.

11. The system of claim 10 wherein packet rule classification predicates comprise flow header classification rules, packet payload classification rules, and meta-information classification rules.

12. The system of claim 10 wherein the machine-learning rule trainer generates the flow-level rules based on at least one of the set of packet rule classification predicates.

13. A system comprising:

network routing means having network interfaces for routing network traffic and for generating flow records pertain to the traffic flow therethrough;

flow collector means for receiving the generated flow records; and

flow classifier means for analyzing the flow records and identifying an anomaly in traffic flow within the network.

14. The system of claim 13, further comprising packet monitor means for generating packet data related to packet of network traffic flowing through the network routing means.

15. The system of claim 13, further comprising packet monitor means for generating packet data related to packet of network traffic flowing through a portion of the network routing means.

16. The system of claim 14, further comprising a machine-learning rule trainer communicatively coupled to the flow collector means and the packet monitor means and configured to generate flow-level rules based on the packet data and the flow data, the flow classifier means being configured to use the flow-level rules for identifying an anomaly.

17. The system of claim 16 wherein the machine-learning rule generator is further configured to receive additional flow data from the flow collector means and additional packet data and associated packet data rules from the packet monitor means and configured to generate revised flow-level rules based on the additional packet data and the additional flow data.

18. A method comprising:

generating flow records at each of a plurality of network routing interfaces routing network;

receiving the generated flow records from the plurality of network routing interfaces; and

analyzing the flow records to thereby identify an anomaly in traffic flow within the network based on flow-level rules.

19. The method of claim 18, further comprising generating packet data rules related to network packet traffic flowing through the network routing interfaces.

20. The method of claim 18, further comprising generating packet data rules related to network packet traffic flowing through a portion of the network routing interfaces.

21. The method of claim 19, further comprising receiving the packet data, packet data rules, and the flow data and generating flow-level rules based on the packet data, packet data rules, and the flow data.

22. The method of claim 21, further comprising receiving additional flow data and additional packet data and associated packet data rules generating revised flow-level rules based on the additional packet data, packet data rules, and the additional flow data.