Remote aggregation of network traffic profiling data

• US 7,809,826 B1
• Filed: 01/27/2005
• Issued: 10/05/2010
• Est. Priority Date: 01/27/2005
• Status: Active Grant

• Current Assignee: Juniper Networks Incorporated
• Original Assignee: Juniper Networks Incorporated
• Inventors: Guruswamy, Kowsik
• Primary Examiner(s): Flynn; Nathan
• Assistant Examiner(s): Conaway; James E
• Application Number:
  US11/044,481
• Time in Patent Office: 2,077 Days
• Field of Search: 709/224, 726/23
• US Class Current: 709/224
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/163 In-band adaptation of TCP data exchange; In-band control procedures
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

A plurality of network devices monitor network traffic and generate profiling data that describes packet flows within the network traffic. The network devices output communications that include the profiling data. An aggregation device receives the communications and builds a correlation database to aggregate the profiling data generated by the plurality of network devices. The profiling data may relate low-level network elements associated with the packet flows and application-layer elements extracted from application-layer communications reassembled from the packet flows.

Claims

1. A method comprising:

processing packet flows with a first network device of a network to identify low-level network elements associated with the packet flows;

forming application-layer communications from the packet flows;

processing the application-layer communications with protocol-specific decoders of the first network device to identify application-layer elements;

generating profiling data to associate the application-layer elements of the application-layer communications with the network elements of the packet flows;

applying operations to a cache within the first network device to temporarily store the profiling data describing the packet flows within the network;

mirroring at least a portion of the operations to a second network device; and

applying the operations to a correlation database with the second network device to aggregate the profiling data.

2. The method of claim 1, wherein the network elements comprise low-level network information including at least one of a network host, network peers or a network port associated with the network traffic.

3. The method of claim 1, wherein the application-layer elements comprise named elements that uniquely identify types of the application-layer elements and values for the named elements.

4. The method of claim 1, further comprising:

applying the operations comprises applying database operations to the cache in the first network device to temporarily store the network elements and the application-layer elements; and

defining relationships within the cache to associate the network elements of the packet flows with the application-layer elements of the application-layer communications.

5. The method of claim 4,

wherein the network elements comprise data defining hosts associated with the packet flows, data defining peers for each of the packet flows, and data defining ports associated with the packet flows, and

wherein defining relationships within the database comprises mapping each of the application-layer elements to a respective one of the peers.

6. The method of claim 1, wherein processing the application-layer communications comprises processing the application-layer communications with protocol-specific decoders for at least one of a HyperText Transfer Protocol (HTTP) decoder, a File Transfer Protocol (FTP) decoder, a Network News Transfer Protocol (NNTP) decoder, a Telnet decoder, a Domain Name System (DNS) decoder, a Gopher decoder, a Finger decoder, a Post Office Protocol (POP) decoder, a Secure Socket Layer (SSL) protocol decoder, a Lightweight Directory Access Protocol (LDAP) decoder, a Secure Shell (SSH) decoder, a Server Message Block (SMB) decoder or a Simple Mail Transfer Protocol (SMTP) decoder.

7. The method of claim 1, wherein mirroring the operations comprises outputting only those operations that insert network elements into the cache.

8. The method of claim 1, wherein the cache includes entries storing the profile data, the method further comprising replacing entries within the cache in accordance with a least recently used algorithm.

9. The method of claim 1, wherein the operations comprise database operations.

10. The method of claim 1, wherein the first network device comprises a firewall and the second device comprises a remote device coupled to the firewall by a network.

11. The method of claim 1, further comprising:

defining a database trigger, with the second network device, for the correlation database to detect database operations that have changed the aggregated profiling data stored within the correlation database, wherein the database trigger is defined to detect database operations that have utilized a specified combination of the low-level network elements and the application-layer elements; and

maintaining, with the second network device, a log to record the detected database operations by updating the log when the database trigger fires to record in the log the database operation that fired the database trigger.

12. The method of claim 11, further comprising presenting a user interface by which a user defines the database triggers.

13. A method comprising

receiving communications from a plurality of network devices, wherein the communications include profiling data describing low-level network elements associated with packet flows monitored by the network devices, and wherein the profiling data also describe application-layer elements carried by the packet flows that have been identified by protocol-specific decoders of the network devices;

building a correlation database to aggregate the profiling data received from the plurality of network devices;

defining a database trigger for the correlation database to detect database operations that have changed the aggregated profiling data stored within the correlation database, wherein the database trigger is defined to detect database operations that have utilized a specified combination of the low-level network elements and the application-layer elements; and

maintaining a log to record the detected database operations by updating the log when the database trigger fires to record in the log the database operation that fired the database trigger.

14. The method of claim 13, wherein the communications describe operations applied by the network devices to respective caches within the plurality of network devices.

15. The method of claim 14, wherein building a correlation database comprises applying the operations to the correlation database to aggregate the profiling data cached by the plurality of network devices.

16. A system comprising:

an aggregation device; and

a plurality of network devices that monitor network traffic and generate profiling data that describes packet flows within the network traffic, wherein the network devices output communications that include the profiling data,

wherein each of the plurality of network devices comprises a processor, a memory and a cache, wherein the processor executes a plurality of modules stored to the memory, and wherein the plurality of modules include;

a flow analysis module to process the packet flows and identify low-level network elements associated with the packet flows;

an analysis engine to form application-layer communications from the packet flows;

a plurality of protocol-specific decoders to process the application-layer communications to generate application-layer elements; and

a profiler to generate the profiling data by correlating the application-layer elements of the application-layer communications with the network elements of the packet flows, apply operations to the cache to temporarily store the profiling data describing the packet flows and mirror at least a portion of the operations to the aggregation device, and

wherein the aggregation device includes a processor and a correlation database, wherein the aggregation device receives the communications and applies the mirrored operations to the correlation database to aggregate the profiling data generated by the plurality of network devices.

17. The system of claim 16, wherein the profiling data maps the application-layer elements to the network elements.

18. The system of claim 16, wherein the plurality of protocol-specific decoders include at least one of a HyperText Transfer Protocol (HTTP) decoder, a File Transfer Protocol (FTP) decoder, a Network News Transfer Protocol (NNTP) decoder or a Simple Mail Transfer Protocol (SMTP) decoder.

19. The system of claim 16, wherein the network elements comprise low-level network information including at least one of a network host, network peers or a network port associated with the network traffic.

20. The system of claim 16, wherein the application-layer elements comprise named elements that uniquely identify types of the application-layer elements and values for the named elements.

21. The system of claim 16, wherein at least one of the network devices has limited persistent storage of less than 64 megabytes.

22. The system of claim 16, wherein at least one of the network devices is a firewall or a switch.

23. The system of claim 16, wherein the aggregation device further presents a user interface by which a user defines a database trigger to detect database operations that have changed the aggregate profiling data stored within the correlation database, wherein the user interface allows the user to define the database trigger by specifying a combination of the low-level network elements and the application-layer elements that the detected database operations have utilized, records the detected database operations in a log and updates the log when the database trigger fires to record in the log file the database operation fired the database trigger.

24. The system of claim 23, wherein the aggregation device further presents a user interface by which a user defines the database triggers.

25. A non-transitory computer-readable storage medium comprising instructions that cause a programmable processor within a network device to:

present a user interface by which a user specifies at least one profiling network device and at least one aggregation network device;

configure the profiling network devices to monitor network traffic, generate profiling data that describes packet flows within the network traffic, and output communications to the aggregation network device that include the profiling data,

wherein configuring the profiling network devices to generate the profiling data comprises;

processing packet flows with a first network device of a network to identify low-level network elements associated with the packet flows;

forming application-layer communications from the packet flows;

processing the application-layer communications with protocol-specific decoders to identify application-layer elements;

generating the profiling data to associate the application-layer elements of the application-layer communications with the network elements of the packet flows; and

configure the aggregation device to;

receive the communications from the network devices;

build a correlation database to aggregate the profiling data generated by the plurality of network devices;

define a database trigger, with the second network device, for the correlation database to detect database operations that have changed the aggregated profiling data stored within the correlation database, wherein the database trigger is defined to detect database operations that have utilized a specified combination of the low-level network elements and the application-layer elements; and

maintain, with the second network device, a log to record the detected database operations by updating the log when the database trigger fires to record in the log the database operation that fired the database trigger.

26. A system comprising:

a plurality of network devices that monitor network traffic and generate profiling data that describes packet flows within the network traffic, wherein the network devices output communications that include the profiling data,

wherein each of the plurality of network devices comprises a processor and a memory,

wherein the processor executes a plurality of modules stored to the memory, and wherein the plurality of modules include;

a flow analysis module to process the packet flows and identify low-level network elements associated with the packet flows;

an analysis engine to form application-layer communications from the packet flows;

a plurality of protocol-specific decoders to process the application-layer communications to generate application-layer elements; and

a profiler to generate the profiling data by correlating the application-layer elements of the application-layer communications with the network elements of the packet flows; and

an aggregation device receives the communications and builds a correlation database to aggregate the profiling data generated by the plurality of network devices,

wherein the aggregation device further presents a user interface by which a user defines a database trigger to detect database operations that have changed the aggregate profiling data stored within the correlation database, wherein the user interface allows the user to define the database trigger by specifying a combination of the low-level network elements and the application-layer elements that the detected database operations have utilized, and

wherein the aggregation device records the detected database operations in a log and updates the log when the database trigger fires to record in the log file the database operation fired the database trigger.