HyperLock technique for high-speed network data monitoring

• US 7,617,314 B1
• Filed: 05/20/2005
• Issued: 11/10/2009
• Est. Priority Date: 05/20/2005
• Status: Active Grant

• Current Assignee: NetScout Systems Incorporated
• Original Assignee: Network General Technology
• Inventors: Sternin, Jeffrey Y., Valladao, Michael Richard, Raghavendran, Sanjeevan, Bansod, Shilpa Pradeep, Iyer, Venkatesh Ramachandran
• Primary Examiner(s): Pwu; Jeffrey
• Assistant Examiner(s): BEHARRY, NOEL R
• Application Number:
  US11/134,808
• Time in Patent Office: 1,635 Days
• Field of Search: 709/219, 709/223, 709/224
• US Class Current: 709/224
• CPC Class Codes: H04L 41/0213 Standardised network management protocols, e.g. simple network management protocol [SNMP]
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

In one embodiment, a network architecture includes a plurality of application monitoring modules for monitoring network traffic data in a plurality of network segments. Network monitoring modules include a staging area that receives network traffic data from a packet capture and analysis engine and an indexing area that stores the data in meta-flow tuples with associated measures divided into time interval buckets. Index tables store dimension-based sorted pointers to the storage locations in the data buckets. HyperLock queries collect time aggregated results for measure based operators with respect to a queried dimension. For each value of the queried dimension, the time interval buckets are traversed compiling a partial result that is finally stored in a stack as the time aggregated value. The stored sorted pointers are used to determine the starting location in each bucket with respect to the next value of the queried dimension.

Claims

1. A method for monitoring network traffic, the method implemented on a computer, the method comprising:

generating two or more index tables associated with two or more time intervals, each index table comprising a set of pointers to a corresponding set of storage locations storing traffic data captured over the time interval associated with the index table, the traffic data in each set of storage locations organized in a plurality of tuples with associated network traffic measures, each tuple including a set of dimension values uniquely identifying network communications between two end-points, wherein the set of pointers are sorted in an order based on at least one of the dimension values in the tuples for the traffic data captured over the interval of time corresponding to the index table; and

in response to a query with respect to a dimension requesting a time aggregated result based on the network traffic measures, aggregating partial results based on network traffic measures into the time aggregated result by accessing the storage locations storing traffic data captured over different time intervals following the order of the pointers in each index table, the aggregating performed by a processor of the computer, the aggregating further comprising;

following the order of the pointers in a first index table associated with a first time interval as long as a dimension value pointed to by each pointer is a specified dimension value; and

upon reaching a pointer in the first index table pointing to a new dimension value different from the specified dimension value, following the order of the pointers in a second index table corresponding to a second time interval as long as a dimension value pointed to by each pointer is the specified dimension value.

2. The method of claim 1, wherein the storage locations comprise an indexing area wherein each set of storage locations correspond to a data bucket of a plurality of data buckets, each data bucket associated with the time interval over which the traffic data stored in the corresponding set of storage locations was captured.

3. The method of claim 2, wherein the plurality of data buckets rotate over time generating a continuous storage mechanism with a maximum storage time, a first data bucket being reused after elapsing the maximum storage time.

4. The method of claim 1, wherein the dimension values in the tuples include a value for one or more of a client, a server, a virtual circuit, a protocol, an application, an interval, or a network segment.

5. The method of claim 1, wherein each index table further comprises a plurality of arrays of pointers, each array associated with a dimension for which dimension values are included in the tuples, wherein the pointers in each array are sorted in an order based on the dimension values for the associated dimension in the tuples for the traffic data captured over the interval of time corresponding to the index table.

6. The method of claim 1, wherein aggregating partial results based on network traffic measures into the time aggregated result by accessing the storage locations storing traffic data captured over different time intervals following the order of the pointers in each index table comprises:

accessing a first set of storage locations associated with a first time interval by accessing the storage location pointed to by a first pointer in a first index table for the query dimension;

aggregating partial results based on the network traffic measures in the storage locations associated with the first time interval further associated with tuples having the first value for the query dimension;

storing the pointer value of a current pointer of the first index table in response to determining a second value different than the first value for the query dimension in a tuple pointed to by the current pointer;

accessing a second set of storage locations associated with a second time interval by accessing the storage location pointed to by a first pointer in a second index table for the query dimension;

aggregating the partial results based on the network traffic measures in storage locations associated with the second time interval further associated with tuples having the first value for the query dimension; and

storing the time aggregated result upon determining a second value different than the first value for the query dimension in a tuple in a last set of storage locations associated with a last time interval.

7. The method of claim 1, wherein the query comprises a queried dimension and a measure based operator configured to provide the time aggregated result based on the network traffic measures for values of the queried dimension.

8. The method of claim 7, wherein the query further comprises a filter field, the filter field for specifying one or more of dimension values or network traffic measures based on which the operator is applied.

9. The method of claim 7, wherein the query includes one of a top number or a bottom number of the queried dimension by the operator time aggregated result.

21. The method of claim 1, wherein each index table is re-created each time the traffic data captured over the time interval associated with the index table is updated based on newly available traffic data.

22. The method of claim 1, wherein aggregating partial results based on network traffic measures into the time aggregated result by accessing the storage locations storing traffic data captured over different time intervals following the order of the pointers in each index table further comprises:

after following the order of the pointers in the second index table as long as a dimension value pointed to by each pointer is the specified dimension value, following the order of the pointers in the first index table as long as a dimension value pointed to by each pointer is the new dimension value.

10. A computer based system for monitoring network traffic comprising:

means for generating two or more index tables associated with two or more time intervals, each index table comprising a set of pointers to a corresponding set of storage locations storing traffic data captured over the time interval associated with the index table, the traffic data in each set of storage locations organized in a plurality of tuples with associated network traffic measures, each tuple including a set of dimension values uniquely identifying network communications between two end-points, wherein the set of pointers are sorted in an order based on at least one of the dimension values in the tuples for the traffic data captured over the interval of time corresponding to the index table;

means for receiving a query with respect to a dimension requesting a time aggregated result based on the network traffic measures; and

means for aggregating partial results based on network traffic measures into the time aggregated result by accessing the storage locations storing traffic data captured over different time intervals following the order of the pointers in each index table, the means for aggregating a partial result coupled to the means for receiving a query for aggregating the partial results in response to receiving the query, the means for aggregating comprising hardware, the aggregating further comprising;

following the order of the pointers in a first index table associated with a first time interval as long as a dimension value pointed to by each pointer is a specified dimension value; and

upon reaching a pointer in the first index table pointing to a new dimension value different from the specified dimension value, following the order of the pointers in a second index table corresponding to a second time interval as long as a dimension value pointed to by each pointer is the specified dimension value.

11. A computer readable storage medium for monitoring network traffic comprising:

computer program instructions for generating two or more index tables associated with two or more time intervals, each index table comprising a set of pointers to a corresponding set of storage locations storing traffic data captured over the time interval associated with the index table, the traffic data in each set of storage locations organized in a plurality of tuples with associated network traffic measures, each tuple including a set of dimension values uniquely identifying network communications between two end-points, wherein the set of pointers are sorted in an order based on at least one of the dimension values in the tuples for the traffic data captured over the interval of time corresponding to the index table; and

computer program instructions for, in response to a query with respect to a dimension requesting a time aggregated result based on the network traffic measures, aggregating partial results based on network traffic measures into the time aggregated result by accessing the storage locations storing traffic data captured over different time intervals following the order of the pointers in each index table, the aggregating further comprising;

following the order of the pointers in a first index table associated with a first time interval as long as a dimension value pointed to by each pointer is a specified dimension value; and

upon reaching a pointer in the first index table pointing to a new dimension value different from the specified dimension value, following the order of the pointers in a second index table corresponding to a second time interval as long as a dimension value pointed to by each pointer is the specified dimension value.

12. A network monitoring appliance for monitoring network traffic comprising:

a network data staging area configured to receive network traffic data from a packet classification and analysis engine in two or more time based data storage buckets, wherein the data storage buckets are associated with a timer for rotating the data storage buckets over time;

a network data indexing area comprising sets of storage locations for storing traffic data captured over a plurality of time intervals, the indexing area coupled to the staging area for periodically updating the traffic data in a current bucket within a time period, the indexing area further configured to store two or more index tables associated with two or more time intervals, each index table for storing a set of pointers to a corresponding set of storage locations; and

a processing unit configured to receive a query from a monitoring console, the query referenced with respect to a query dimension and requesting a time aggregated result based on network traffic measures stored in the storage locations, the processing unit for aggregating partial results from each of a plurality of sets of storage locations, the partial results associated with a value for the query dimension, wherein the processing unit accesses each of the plurality of storage locations based on the pointers in the two or more index tables, the aggregating further comprising;

following the order of the pointers in a first index table associated with a first time interval as long as a dimension value pointed to by each pointer is a specified dimension value; and

upon reaching a pointer in the first index table pointing to a new dimension value different from the specified dimension value, following the order of the pointers in a second index table corresponding to a second time interval as long as a dimension value pointed to by each pointer is the specified dimension value.

13. The network monitoring appliance of claim 12, wherein the staging area and the indexing area are implemented in random access memory.

14. The network monitoring appliance of claim 12, wherein each of the sets of storage locations in the indexing area are configured into a corresponding data bucket, each data bucket associated with a time interval over which the traffic data stored in the corresponding set of storage locations was captured.

15. The network monitoring appliance of claim 14, wherein the use of the plurality of data buckets rotates over time generating a continuous storage mechanism with a maximum storage time, a first data bucket being reused after elapsing the maximum storage time.

16. The network monitoring appliance of claim 12, wherein the network traffic data received from the packet classification and analysis engine is formatted into a plurality of dimension tuples and associated network traffic measures, wherein the dimensions in the tuples include one or more of a client, a server, a virtual circuit, a protocol, an application, an interval, or a network segment.

17. The network monitoring appliance of claim 16, wherein each index table further comprises a plurality of arrays of pointers, each array associated with a dimension, wherein the pointers in each array are sorted in an order based on values for the dimension in the tuples of the traffic data captured over the interval of time corresponding to the index table.

18. The network monitoring appliance of claim 12, wherein the processing unit is further configured to provide a set of default queries for selection by a user at the monitoring console.

19. The network monitoring appliance of claim 12, wherein the processing unit is further configured to aggregate partial results from each of a plurality of sets of storage locations with respect to a set of default queries.

20. The network monitoring appliance of claim 19, wherein the set of default queries are internally provided by the processing unit without being received from the monitoring console.