Method and Apparatus for Deep Packet Inspection for Network Intrusion Detection

• US 20080201772A1
• Filed: 02/14/2008
• Published: 08/21/2008
• Est. Priority Date: 02/15/2007
• Status: Active Grant

• Current Assignee: Marvell Israel Limited (Marvell Technology Group Limited)
• Original Assignee: Marvell Israel Limited (Marvell Technology Group Limited)
• Inventors: Anker, Tal, Mondaeev, Maxim, Meyouhas, Yosef
• Granted Patent: US 8,448,234 B2
• Time in Patent Office: Days
• US Class Current: 726/13
• CPC Class Codes: H04L 47/12 Avoiding congestion; Recovering from congestion
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

In a method of determining whether a data stream includes unauthorized data, the data stream is analyzed using a hardware filter to detect a presence of one or more of a first set of patterns in the data stream. It is determined whether a packet in the data stream belongs to one of a plurality of data flows to be further inspected based on the analysis of the data stream by the hardware filter. A set of rules is applied to the packet to produce rule match status data if it is determined that the packet belongs to one of the plurality of data flows to be further inspected. The packet is analyzed to determine if the packet includes unauthorized data using software stored on a computer-readable medium and implemented on a processor if the rule match status data indicates that the packet potentially includes unauthorized data.

Claims

1. A network intrusion detection system for determining whether a data stream includes unauthorized data, the system comprising:

a hardware filter, including;

a storage unit to store a first set of data patterns, each one indicative of possible unauthorized data, and a comparator coupled to the storage unit to compare a portion of a packet in the data stream to the first set of data patterns;

a switch coupled to the hardware filter to selectively direct the packet for further packet inspection based on whether the packet is associated with one of a set of further packet inspection enabled flows; and

a packet inspection module coupled to the switch to apply packet inspection rules to deeply inspect packets that are associated with one of the packet inspection enabled flows.

2. The system of claim 1, wherein the hardware filter includes a Bloom filter.

3. The system of claim 1, wherein the hardware filter assigns one of a match indicator or a non-match indicator to a packet descriptor associated with the packet.

4. The system of claim 1, wherein the hardware filter includes a first filter circuit to compare the packet to patterns of a first length and a second filter circuit adapted to compare the packet to patterns of a second length.

5. The system of claim 4, wherein the first length is not equal to the second length and wherein the first filter circuit and the second filter circuit process the packet in parallel.

6. The system of claim 1, wherein the switch is a hardware switch.

7. The system of claim 1, wherein the switch processes a header of the packet and wherein the header includes a priority indicator, a source address of the packet, and a destination address of the packet.

8. The system of claim 7, wherein the packet is associated with a packet descriptor including a header bitmask, wherein each of the first set of data patterns is associated with one or more bits within the header bitmask, and wherein the switch assigns a value to the header bitmask in response to processing the header of the packet.

9. The system of claim 7, wherein the packet is associated with a packet descriptor including a bypass default command indicator indicative of an action to be performed on the packet if the packet inspection module is congested, and wherein the switch assigns a value to the bypass default command indicator in response to processing the header of the packet.

10. The system of claim 7, wherein the packet is associated with a packet descriptor including a queuing traffic class indicator, and wherein the switch assigns a value to the queuing traffic class indicator in response to processing the header of the packet.

11. The system of claim 1, wherein the packet is associated with a packet descriptor including a meter identifier;

wherein the switch assigns a value to the meter identifier if the packet is not associated with one of the set of further packet inspection enabled flows;

and wherein the meter identifier is indicative of a communication protocol with which the packet is associated.

12. The system of claim 1, wherein the packet inspection module includes:

an input for receiving further packet inspection enabled data from the switch;

a memory module storing a first plurality of rules, each rule including a second set of data patterns and an associated pattern logic; and

an engine to apply the first plurality of rules to the packet to generate a plurality of rule status indicators indicative of whether rules in the first plurality of rules are met;

13. The system of claim 12, wherein the first plurality of rules and the second plurality of rules are Sourcefire VRT Certified Rules.

14. The system of claim 1, further comprising:

a processor coupled to the packet inspection module and adapted to execute a set of instructions stored on a computer-readable medium that cause the processor to apply a second plurality of rules to the packet if at least one of the plurality of rule status indicators indicates that at least one of the first plurality of rules is met.

15. The system of claim 14, further comprising a rate monitor, the rate monitor including:

a packet data input coupled to the switch;

a counting unit adapted to maintain a packet count corresponding to a number of packets arriving at the rate monitor; and

a signaling link coupled to the processor, wherein the rate monitor sends an alarm to the processor via the signaling link if the count exceeds a threshold value;

wherein the switch conditionally directs the packet data to the packet data input of the rate monitor.

16. The system of claim 15, wherein the switch is a programmable switch that includes a memory to store a plurality of policy rules and wherein the processor updates the plurality of policy rules in response to receiving the alarm from the rate monitor.

17. The system of claim 15, wherein the counting unit includes a respective counter for each of a plurality of data flows.

18. The system of claim 1, further comprising a reordering module coupled to an output of the packet inspection module to restore a correct ordering of packets inspected by the packet inspection module and packets bypassing the packet inspection module.

19. The system of claim 1, wherein the unauthorized data is one of a virus, a worm, data corresponding to a hijacking attempt, or information sent in violation of a network use policy.

20. The system of claim 1, further including a non-volatile memory storage coupled to the system to supply the data stream.

21. The system of claim 1, wherein at least one of the set of further enabled flows is associated with at least one of a particular source address or a particular destination address;

and wherein the packet inspection module deeply inspects a payload of a packet associated with the at least one of the set of further enabled flows if the hardware filter detects a match of the portion of the packet with at least one of the first set of data patterns.

22. The system of claim 1, wherein the hardware filter, the switch, and the packet inspection module form a pipeline by performing the acts of comparing, selectively directing packets, and applying rules at a same time.

23. A method of determining whether a data stream includes unauthorized data, the method comprising:

analyzing the data stream using a hardware filter to detect a presence of one or more of a first set of patterns in the data stream;

determining whether a packet in the data stream belongs to one of a plurality of data flows to be further inspected based on the analysis of the data stream by the hardware filter;

applying a set of rules to the packet to produce a rule match status data if it is determined that the packet belongs to one of the plurality of data flows to be further inspected and if a presence of one or more of the first set of patterns has been detected in the packet; and

analyzing the packet payload to determine if the packet includes unauthorized data.

24. The method of claim 23, wherein the act of analyzing the packet payload to determine if the packet includes unauthorized data includes using software stored on a computer-readable medium and implemented on a processor if the rule match status data indicates that the packet potentially includes unauthorized data.

25. The method of claim 24, wherein analyzing the data stream using the hardware filter includes assigning a first value to a first-stage filter indicator indicative of a potential presence of unauthorized data in the packet.

26. The method of claim 24, wherein analyzing the data stream using the hardware filter includes analyzing a header of the packet and a payload of the packet.

27. The method of claim 24, further comprising determining a ratio of a number of packets that do not belong to any of the plurality of data flows to be further inspected to a total number packets in the data stream.

28. The method of claim 27, further comprising generating a signal to the processor if the ratio exceeds a threshold value.

29. The method of claim 23, wherein determining whether a packet in the data stream belongs to one of a plurality of data flows to be further inspected includes determining whether the packet is associated with at least one of a particular source address or a particular destination address.

30. The system of claim 23, wherein the acts of analyzing the data stream, determining whether a packet belongs to one of the plurality of data flows, applying the set of rules, and analyzing a packet payload are carried out simultaneously at different pipeline stages.

31. The method of claim 23, further comprising updating the set of rules if the rule match status data indicates a match rate that exceeds a threshold value.

32. The method of claim 23, wherein at least one of applying the set of rules or analyzing the packet using software includes:

applying Sourcefire VRT Certified rules; and

applying a first operation to the packet, including;

locating a length encoder in the packet; and

comparing at least one of a second set of patterns to a section of the packet starting at an offset corresponding to the length encoder.

33. The method of claim 32, wherein at least one of applying the set of rules or analyzing the packet using software includes applying a second operation to the packet, including checking whether a numerical value corresponding to one of the bytes in the packet is greater than a number associated with a rule.

34. A method of determining whether a data stream includes unauthorized data, the method comprising:

filtering the data stream via a first hardware filter by testing each of a plurality of packets in the data stream against a first set of patterns to determine whether the packet includes one or more of the first set of patterns, wherein the first set of patterns includes a first set of root patterns and a first set of non-root patterns;

applying a plurality of pattern matching rules to at least some of the plurality packets in the data stream, wherein the plurality of pattern matching rules is associated with a second set of patterns including a second set of root patterns and a second set of non-root patterns, and wherein applying each of the plurality of pattern matching rules to an individual packet includes;

checking whether the packet includes at least one of the second set of root patterns in a first location within the packet consistent with a rule associated with the at least one of the set of root patterns; and

if the packet includes at least one of the set of root patterns;

checking whether the packet includes at least one of the second set of non-root patterns associated with the rule in a second location within the packet consistent with the rule; and

generating an alarm indicative of a suspected presence of unauthorized data in the data stream.

35. A packet inspection hardware module comprising:

a first memory unit to store a set of patterns, wherein the first memory unit has a first set of physical parameters;

a second memory unit to store a plurality of rules indicative of logic for comparing the set of patterns to the packet, wherein the second memory unit has a second set of physical parameters distinct from the first set of physical parameters; and

a packet inspection engine to apply the plurality of rules to the packet, including;

a matching unit coupled to the first memory unit to compare a section of the packet to the set of patterns;

a rule retrieval unit coupled to the second memory unit and the matching unit to conditionally retrieve rule information corresponding to patterns in the packet detected by the matching unit, the rule information including a set of additional patterns; and

a rule application unit to compare the packet to the set of additional patterns according to logic specified by the retrieved rule information.

36. The module of claim 35, further comprising a character counter adapted to count a number of characters in a portion of the packet to produce a count indication and to apply the count indication in the rule application unit.

37. The module of claim 35, wherein the packet inspection engine stores the retrieved rule information in a separate location from the set of additional patterns associated with the retrieved rule information.

38. The module of claim 35, wherein the packet inspection engine is a pipeline engine, wherein each of the matching unit, the rule retrieval unit, and the pattern application unit are pipeline stages coupled in series.

39. The module of claim 38, wherein the matching unit applies a sliding window to the packet.

40. The module of claim 38, wherein the first memory unit includes a content-addressable memory (CAM) and the second memory includes a random access memory (RAM).

41. The module of claim 38, wherein at least one of the first memory unit or the second memory unit is a content-addressable memory (CAM) to simultaneously compare multiple shifted windows to the set of patterns.

42. The module of claim 38, wherein the first memory unit stores root patterns and the second memory unit stores non-root patterns, wherein at least some of the plurality of rules specify a manner in which a single root pattern and one or more non-root patterns are located in a data stream with respect to each other.

43. The module of claim 42, wherein the first memory is a content-addressable memory (CAM) memory and the second memory is a random access memory (RAM);

and wherein the packet inspection engine populates the second memory during a pipeline stage corresponding to the rule retrieval unit.

47. A method of inspecting a data stream for unauthorized data, comprising:

comparing a data segment to a first set of patterns stored in a content-addressable memory;

if a pattern from the first set of patterns is in the data segment, retrieving a set of packet processing rules corresponding to the pattern and a second set of patterns associated with the set of packet processing rules;

adding the set of packet processing rules to a packet processing rule list;

adding the second set of patterns to a third pattern list;

applying the processing rule list and the third pattern list to the data segment to populate a rule status registry stored in a random-access memory unit separate from the content-addressable memory; and

repeating the acts of comparing, retrieving, and applying for a different data segment.

48. The method of claim 47, wherein each of the first set of patterns stored in the content-addressable memory are root patterns corresponding to a first mandatory condition of at least one of the set of pattern processing rules.

49. The method of claim 47, wherein applying the packet processing rule list to the data segment to populate a rule status registry includes:

checking whether logic of any of the rules in the packet processing rule list has not been met by the data segment; and

deleting one or more processing rules associated with a logic condition that has not been met.

50. A method of inspecting a data stream, the method including:

processing a packet to determine if the packet is one of a candidate for further packet inspection;

generating a descriptor for the packet, the descriptor including an indicator of candidacy for further packet inspection;

storing the packet in a memory buffer;

if the packet is a candidate for further packet inspection;

queuing the descriptor in one of a plurality of queues according to a policy associated with the packet, wherein each of the plurality of queues is associated with a corresponding priority level;

conditionally performing further packet inspection of the packet according to an ordering of the descriptor in one of the plurality of queues; and

the method further comprising;

if the packet is not a candidate for further packet inspection;

not queuing the descriptor in the plurality of queues.

44. The module of claim 53, wherein the first memory unit stores a plurality of root patterns;

wherein the second memory is a non-volatile memory;

wherein the module further comprises a third memory unit to temporarily store a plurality of non-root patterns.

45. The module of claim 44, wherein the first memory is a first content-addressable memory (CAM), the second memory is a random access memory (RAM), and the third memory is a second CAM.

46. The module of claim 44, wherein the packet inspection engine populates the third memory after retrieving the rule information from the second memory.

51. The method of claim 50, wherein processing the packet includes:

filtering the packet by applying a first set of patterns to produce a pattern match indicator;

and

applying policy logic to a header of the packet to produce a policy match indicator.

52. The method of claim 51, wherein performing further packet inspection of the packet includes applying a plurality of pattern processing rules to the packet, wherein each of the plurality of pattern processing rules specifies a set of patterns and a logic defining a relationship between patterns in the set of patterns.

53. The method of claim 52, wherein conditionally performing further packet inspection of the packet includes directing the descriptor to a packet inspection bypass buffer if the pattern match indicator associated with the descriptor does indicate that the packet includes at least one of the first set of patterns.

54. The method of claim 52, further comprising restoring a sequence of a group of packets in the data stream, wherein the sequence corresponds to an order in which the group of packets was received, and wherein the group of packets includes at least one candidate for further packet inspection and at least one non-candidate for further packet inspection.

55. The method of claim 54, wherein conditionally performing a further packet inspection of the packet includes directing the descriptor to a packet inspection bypass buffer if the pattern match indicator associated with the descriptor does indicate that the packet includes at least one of the first set of patterns;

and wherein restoring the sequence of the group of packets includes restoring an ordering between the packets leaving the further packet inspection bypass buffer and the further inspected packets.

56. The method of claim 50, wherein queuing the descriptor in one of a plurality of queues further includes resolving congestion associated with further packet inspection by conditionally directing the descriptor around the plurality of queues to bypass further packet inspection according to a policy associated with the packet corresponding to the descriptor.

57. The method of claim 50, wherein processing a packet includes receiving the packet from a non-volatile memory storage.

58. A packet inspection module comprising:

a packet descriptor queuing module including a plurality of queues, each storing a plurality of descriptors corresponding to a plurality of packets;

a packet inspection engine operatively connected to the packet descriptor queuing module and adapted to;

receive a packet descriptor from one of the plurality of queues;

extract a packet corresponding to the packet descriptor; and

apply one or more pattern matching rules to the packet to determine whether the packet includes unauthorized data; and

the packet inspection module further comprising;

a bypass module operatively connected to the packet descriptor queuing module for temporarily storing packets that belong to a stream with which the extracted packet is associated.

59. The module of claim 58, further comprising a priority scheduler coupled to the packet inspection engine and adapted to direct packet descriptors to the packet inspection engine according to a policy associated with the packet inspection module.

60. The module of claim 59, wherein the policy includes one of Strict Priority or Smooth Deficit Weighted Round Robin.

61. The module of claim 60, further comprising a priority ordering module coupled to an output of the packet inspection engine and to an output of the bypass module and adapted to restore an order of the stream.