Systems and methods for rule-based anomaly detection on IP network flow

• US 9,680,877 B2
• Filed: 12/15/2015
• Issued: 06/13/2017
• Est. Priority Date: 12/16/2008
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Krishnamurthy, Balachander, Duffield, Nicholas, Haffner, Patrick, Ringberg, Haakon Andreas
• Primary Examiner(s): Chang, Tom Y
• Application Number:
  US14/969,591
• Publication Number:
  US 20160105462A1
• Time in Patent Office: 546 Days
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: H04L 41/00 Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 45/28 using route fault recovery
• CPC Class Codes: H04L 45/38 Flow based routing
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/24 Traffic characterised by specific attributes, e.g. priority or QoS
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H04L 63/30 for supporting lawful interception, monitoring or retaining of communications or communication related information circuit switched telephony call monitoring H04M3/2281

Abstract

A system to detect anomalies in internet protocol (IP) flows uses a set of machine-learning (ML) rules that can be applied in real time at the IP flow level. A communication network has a large number of routers equipped with flow monitoring capability. A flow collector collects flow data from the routers throughout the communication network and provides them to a flow classifier. At the same time, a limited number of locations in the network monitor data packets and generate alerts based on packet data properties. The packet alerts and the flow data are provided to a machine learning system that detects correlations between the packet-based alerts and the flow data to thereby generate a series of flow-level alerts. These rules are provided to the flow time classifier. Over time, the new packet alerts and flow data are used to provide updated rules generated by the machine learning system.

Claims

1. A system comprising:

a processor; and

memory comprising instructions that, when executed by the processor, cause the processor to perform operations comprising;

receiving a flow record of a data flow,

receiving a packet alert generated for a data packet, the packet alert generated based on a packet-level rule,

determining a time when the packet alert associated with the data packet was generated,

determining that the time when the packet alert associated with the data packet was generated occurred between a start time and an end time of the data flow,

in response to determining that the time when the packet alert associated with the data packet was generated occurred between the start time and the end time of the data flow, associating the packet alert with the data flow,

attributing, to the packet-level rule, a score in a form of a weighted sum Σ

_kw_kp_k(x) over flow level predicates p_k(x), and

in response to the score exceeding a threshold, constructing a flow-level rule that, when applied to the data flow, generates an alarm associated with the data flow, wherein the flow-level rule identifies an anomaly in the data flow, and wherein a weight w_k is chosen to minimize a classification error such that the flow-level rule, when applied to the data flow, generates the alarm associated with the data flow when the packet-level rule, when applied to the data packet, generates the packet alert for the data packet.

2. A method comprising:

receiving, by a system comprising a processor, a flow record of a data flow;

receiving, by the system, a packet alert generated for a data packet, the packet alert generated based on a packet-level rule,

determining, by the system, a time when the packet alert associated with the data packet was generated;

determining, by the system, that the time when the packet alert associated with the data packet was generated occurred between a start time and an end time of the data flow;

in response to determining that the time when the packet alert associated with the data packet was generated occurred between the start time and the end time of the data flow, associating, by the system, the packet alert with the data flow;

attributing, to the packet-level rule by the system, a score in a form of a weighted sum Σ

_kw_kp_k(x) over flow level predicates p_k(x); and

in response to the score exceeding a threshold, constructing, by the system, a flow-level rule that, when applied to the data flow, generates an alarm associated with the data flow, wherein the flow-level rule identifies an anomaly in the data flow, and wherein a weight w_k is chosen to minimize a classification error such that the flow-level rule, when applied to the data flow, generates the alarm associated with the data flow when the packet-level rule, when applied to the data packet, generates the packet alert for the data packet.

3. A memory storing instructions that, when executed by a processor of a system, cause the processor to perform operations comprising:

receiving a flow record of a data flow;

receiving a packet alert generated for a data packet, the packet alert generated based on a packet-level rule;

determining a time when the packet alert associated with the data packet was generated;

determining that the time when the packet alert associated with the data packet was generated occurred between a start time and an end time of the data flow;

in response to determining that the time when the packet alert associated with the data packet was generated occurred between the start time and the end time of the data flow, associating the packet alert with the data flow;

attributing, to the packet-level rule, a score in a form of a weighted sum Σ

_kw_kp_k(x) over flow level predicates p_k(x); and

in response to the score exceeding a threshold, constructing a flow-level rule that, when applied to the data flow, generates an alarm associated with the data flow, wherein the flow-level rule identifies an anomaly in the data flow, and wherein a weight w_k is chosen to minimize a classification error such that the flow-level rule, when applied to the data flow, generates the alarm associated with the data flow when the packet-level rule, when applied to the data packet, generates the packet alert for the data packet.