Methods and apparatus for collecting, storing, processing and using network traffic data

• US 6,279,037 B1
• Filed: 08/10/1998
• Issued: 08/21/2001
• Est. Priority Date: 05/28/1998
• Status: Expired due to Term

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: 3Com Corporation (HP Inc.)
• Inventors: Iddon, Robin, Brown, Ronnie, Tams, Jonathan, Pearce, Mark Adrian
• Primary Examiner(s): Dinh, Dung C.
• Assistant Examiner(s): FIELDS, KENNETH WAYNE
• Application Number:
  US09/131,717
• Time in Patent Office: 1,107 Days
• Field of Search: 709/223, 709/224, 709/220
• US Class Current: 709/224
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 43/12 Network monitoring probes

Abstract

Methods and apparatus for collecting, storing, processing and using data, such a RMON2 network traffic data, are described. Network traffic probes are identified and attempts are made to configure the identified probes to generate network traffic data sets which are as close to a preselected common data format as possible. Application layer traffic data is collected in addition to network layer traffic data when possible. In an RMON2 embodiment, the common data format includes the use of delta count values as opposed to absolute count values. The common data format of the present invention utilizes terminal count mode format as opposed to all count mode format for the presentation of RMON2 application layer information. To minimize the amount of data processing required to put a probe'"'"'s network traffic data into the desired common format and to maximize the amount of information collected, network data is obtained from a probe using one of the available RMON2 table formats. In order to avoid various problems with known data aging processes used to limit the growth of network traffic databases, a database of collected network traffic information which includes multiple parallel sets of data stored at different resolutions is created and maintained. The data sets for each individual resolution are stored in a separate FIFO data structure and with the oldest data records in the FIFO being overwritten when allocated data space becomes fully utilized.

Claims

1. A method of processing and storing data in a computer system including processor circuitry, and a data storage device, the method comprising the steps of:

storing first and second sets of records in separate first-in, first-out data structures, respectively, on the data storage device, the first and second sets of records being of different data resolutions and corresponding to overlapping periods of time;

operating the processor circuitry to receive data collected over a period of time; and

operating the processor circuitry to update at least one record in each of the stored first and second sets of records with the received data such that a previous record included in each of the first and second data structures is replaced;

periodically collecting network traffic data, wherein the collected network traffic data includes byte and packet count information associated with each of a plurality of monitored conversations between devices included in the computer system;

storing the collected network traffic data in a buffer; and

operating the processor circuitry to retrieve network traffic data from the buffer, the retrieved network traffic data being received by the processor circuitry;

wherein the step of operating the processor circuitry to update at least one record in each of the stored first and second sets of records includes the steps of;

updating a record corresponding to a first conversation in the first set of records; and

updating a record corresponding to the first conversation in the second set of records.

2. The method of claim 1, further comprising the step of:

allocating fixed amounts of storage space on the data storage device for storing each one of the first and second first-in, first-out data structures used to store the first and second sets of records.

3. The method of claim 1, wherein the first set of records include hourly records and the second set of records includes daily records.

4. The method of claim 1,

wherein the network traffic data stored in the buffer includes time stamp information indicating the period of time in which the network traffic data was collected;

and

wherein the step of operating the processor circuitry to update at least one record in each of the stored first and second sets of records includes the step of;

examining at least one time stamp included in the buffered network traffic data.

5. The method of claim 1,

wherein the processor circuitry includes first and second central processing units, and

wherein the step of operating the processor circuitry to update at least one record in each of the stored first and second sets of records includes the step of operating the first processor to update the first set of records while operating the second processor to update the second set of records.

6. The method of claim 1, wherein the computer system further includes a display device, the method further comprising the step of:

displaying data corresponding to overlapping periods of time at different resolutions on the display device.

7. A computer system for monitoring network traffic data comprising:

a memory;

a data storage device; and

a processor to execute instructions stored in the memory,

wherein the memory stores;

instructions to store first and second sets of records in separate first-in, first-out data structures, respectively, on the data storage device, the first and second sets of records being of different data resolutions and corresponding to overlapping periods of time;

instructions to receive data collected over a period of time;

instructions to update at least one record in each of the stored first and second sets of records with the received data such that a previous record included in each of the first and second data structures is replaced;

instructions to periodically collect network traffic data, wherein the collected network traffic data includes byte and packet count information associated with each of a plurality of monitored conversations between devices included in the computer system;

instructions to store the collected network traffic data in a buffer; and

instructions to retrieve network traffic data from the buffer, the retrieved network traffic data being received by the processor;

wherein the instructions to update at least one record in each of the stored first and second sets of records include instructions to;

update a record corresponding to a first conversation in the first set of records; and

update a record corresponding to the first conversation in the second set of records.

8. The computer system of claim 7, wherein the memory further comprises instructions to:

allocate fixed amounts of storage space on the data storage device for storing each one of the first and second first-in, first-out data structures used to store the first and second sets of records.

9. The computer system of claim 7, wherein the first set of records include hourly records and the second set of records includes daily records.

10. The computer system of claim 7,

wherein the network traffic data stored in the buffer includes time stamp information indicating the period of time in which the network traffic data was collected;

and

wherein the instructions to update at least one record in each of the stored first and second sets of records include instructions to;

examine at least one time stamp included in the buffered network traffic data.

11. The computer system of claim 7,

wherein the processor includes first and second central processing units, and

wherein the instructions to operate the processor to update at least one record in each of the stored first and second sets of records includes instructions to operate the first processor to update the first set of records while operating the second processor to update the second set of records.

12. The computer system of claim 7 further including a display device, the memory further comprising instructions to:

display data corresponding to overlapping periods of time at different resolutions on the display device.

13. A computer program product system for monitoring network traffic data, said computer program product comprising a computer usable medium having computer readable program code means embodied in said medium for causing a processor in a computer to:

store first and second sets of records in separate first-in, first-out data structures, respectively, on a data storage device, the first and second sets of records being of different data resolutions and corresponding to overlapping periods of time;

receive data collected over a period of time;

update at least one record in each of the stored first and second sets of records with the received data such that a previous record included in each of the first and second data structures is replaced;

periodically collect network traffic data, wherein the collected network traffic data includes byte and packet count information associated with each of a plurality of monitored conversations between devices included in the computer system;

store the collected network traffic data in a buffer; and

retrieve network traffic data from the buffer, the retrieved network traffic data being received by the processor;

wherein the causing the processor to update at least one record in each of the stored first and second sets of records includes;

updating a record corresponding to a first conversation in the first set of records; and

updating a record corresponding to the first conversation in the second set of records.

14. The computer program product of claim 13, wherein the computer readable program code means further causes the processor to:

allocate fixed amounts of storage space on the data storage device for storing each one of the first and second first-in, first-out data structures used to store the first and second sets of records.

15. The computer program product of claim 13, wherein the first set of records include hourly records and the second set of records includes daily records.

16. The computer program product of claim 13,

wherein the network traffic data stored in the buffer includes time stamp information indicating the period of time in which the network traffic data was collected;

and

wherein the computer readable program code means to update at least one record in each of the stored first and second sets of records includes computer readable program code means to examine at least one time stamp included in the buffered network traffic data.

17. The computer program product of claim 13,

wherein the processor includes first and second central processing units, and

wherein the computer readable program code means to update at least one record in each of the stored first and second sets of records includes computer readable program code means to operate the first processor to update the first set of records while operating the second processor to update the second set of records.

18. The computer program product of claim 13 wherein the computer readable program code means further causes the computer to display data corresponding to overlapping periods of time at different resolutions on a display device.