METHOD AND APPARATUS FOR RATE LIMITING

• US 20110173342A1
• Filed: 03/28/2011
• Published: 07/14/2011
• Est. Priority Date: 06/16/2000
• Status: Active Grant

• Current Assignee: McAfee, Inc. (McAfee, LLC)
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: Sherlock, Kieran G., Cooper, Geoffrey, Hoy, Mark
• Granted Patent: US 8,849,993 B2
• Time in Patent Office: Days
• US Class Current: 709/233
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 41/5012 determining service availability, e.g. which services are available at a certain point in time
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/06 Generation of reports
• CPC Class Codes: H04L 43/062 related to network traffic
• CPC Class Codes: H04L 43/067 using time frame reporting
• CPC Class Codes: H04L 43/14 using software, i.e. software packages network security related monitoring H04L63/1408
• CPC Class Codes: H04L 43/18 Protocol analysers
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/32 by discarding or delaying data units, e.g. packets or frames
• CPC Class Codes: H04L 63/0227 Filtering policies mail message filtering H04L51/212
• CPC Class Codes: H04L 63/0263 Rule management
• CPC Class Codes: H04L 63/0823 using certificates cryptographic mechanisms or cryptographic arrangements for entity authentication involving certificates H04L9/3263
• CPC Class Codes: H04L 63/083 using passwords cryptographic mechanisms or cryptographic arrangements for entity authentication using a predetermined code H04L9/3226
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 : Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 : Vulnerability analysis
• CPC Class Codes: H04L 63/166 : at the transport layer
• CPC Class Codes: H04L 69/22 : Parsing or analysis of headers

Abstract

A method and apparatus for a network monitor internals mechanism, which serves to translate packet data into multiple concurrent streams of encoded network event data, to contribute to enterprise management, reporting, and global mechanisms for aggregating monitors at a centralized aggregation point, and to facilitate rate limiting techniques because such monitors are not in control (i.e. cannot back pressure flow) is provided.

Claims

1. A rate limiting method for a pipelined network monitoring system having a plurality of stages, said method comprising the steps of:

choosing a heuristic for discarding data at each of said plurality of stages, wherein choosing includes selecting the heuristic that is appropriate for each stage; and

applying a rate limiting process at said each stage, wherein each rate limiting process incorporates the appropriate heuristic.

2. The method of claim 1, wherein applying a rate limiting process includes discarding data based on connection boundaries.

3. The method of claim 1, wherein applying a rate limiting process includes discarding connection details, but maintaining counts of connections.

4. The method of claim 1, wherein applying a rate limiting process includes abstracting connection information by counting and combining like events over a time interval.

5. The method of claim 1, wherein one of the stages receives packets and wherein the heuristic selected for that stage includes an algorithm that discards packets when the number of packets received per unit time is greater than a preselected threshold.

6. The method of claim 1, wherein one of the stages monitors network events and wherein the heuristic selected is such that the rate limiting process chooses not to pass data to the next stage when the rate of new network events per unit time is greater than a preselected threshold.

7. The method of claim 1, wherein the heuristic includes the RLE algorithm.

8. An apparatus for rate limiting a pipelined network monitoring system having a plurality of stages, said apparatus comprising:

means for choosing a heuristic for discarding data at each of said plurality of stages, wherein the means for choosing includes means for selecting the heuristic that is appropriate for each stage; and

means for applying a rate limiting process at said each stage, wherein each rate limiting process incorporates the appropriate heuristic.

9. The apparatus of claim 8, wherein the means for applying a rate limiting process includes means for discarding data based on connection boundaries.

10. The apparatus of claim 8, wherein the means for applying a rate limiting process includes means for discarding connection details, but maintaining counts of connections.

11. The apparatus of claim 8, wherein the means for applying a rate limiting process includes means for abstracting connection information by counting and combining like events over a time interval.

12. The apparatus of claim 8, wherein one of the stages receives packets and wherein the heuristic selected for that stage includes an algorithm that discards packets when the number of packets received per unit time is greater than a preselected threshold.

13. The apparatus of claim 8, wherein one of the stages monitors network events and wherein the heuristic selected is such that the rate limiting process chooses not to pass data to the next stage when the rate of new network events per unit time is greater than a preselected threshold.

14. The apparatus of claim 8, wherein the heuristic includes the RLE algorithm.

15. An apparatus for rate limiting to control amounts of data in a passive network monitoring pipelined system, said apparatus comprising:

a packet filter module having a first rate limiting mechanism for discarding particular incoming packets and passing on non-discarded packets;

a monitor receiving said non-discarded packets from said packet filter module, said monitor processing said non-discarded packets and outputting corresponding events, said monitor having a second rate limiting mechanism for discarding particular incoming events and passing on non-discarded events;

a logger module having a third rate limiting mechanism for discarding unwanted data and passing on said non-discard events, said logger module within a policy monitor module receiving said outputted corresponding events from said monitor; and

a reporting module, having a fourth rate limiting mechanism wherein said non-discarded events are received from said logger module and combined with like events in a “

roll-up”

over a time interval or based on the hierarchy of objects to meet storage requirements;

wherein said second rate limiting mechanism discards particular packets on input when input packet rate exceeds a specified limit.

16. The apparatus of claim 15, wherein said second rate limiting mechanism comprises an algorithm, wherein input to said algorithm as number of packets per second that can be processed by said monitor is controlled by a command, wherein said command runs said monitor on a local network device, generates a stream of events to a local host port, and restricts average processing rate to a predetermined number of packets/second.

17. The apparatus of claim 16, wherein said algorithm works by defining a predetermined measurement interval, such that an argument value is normalized to the number of packets allowed received in a single measurement interval, and when said number of packets is received, no more packets are received until said measurement interval is over.

18. A method for rate limiting to control amounts of data in a passive network monitoring pipelined system, said method comprising:

providing a packet filter module having a first rate limiting mechanism for discarding particular incoming packets and passing on non-discarded packets;

providing a monitor receiving said non-discarded packets from said packet filter module, said monitor processing said non-discarded packets and outputting corresponding events, said monitor having a second rate limiting mechanism for discarding particular incoming events and passing on non-discarded events;

providing a logger module having a third rate limiting mechanism for discarding unwanted data and passing on said non-discard events, said logger module within a policy monitor module receiving said outputted corresponding events from said monitor; and

providing a rate limiting module, having a fourth rate limiting mechanism wherein said non-discarded events are received from said logger module and abstracted over a predetermined time interval to meet storage requirements;

wherein said second rate limiting mechanism discards particular packets on input when input packet rate exceeds a specified limit.

19. The method of claim 18, wherein said second rate limiting mechanism comprises an algorithm, wherein input to said algorithm as number of packets per second that can be processed by said monitor is controlled by a command, wherein said command runs said monitor on a local network device, generates a stream of events to a local host port, and restricts average processing rate to a predetermined number of packets/second.

20. The method of claim 19, wherein said algorithm works by defining a predetermined measurement interval, such that an argument value is normalized to the number of packets allowed received in a single measurement interval, and when said number of packets is received, no more packets are received until said measurement interval is over.