Apparatus and method for collecting and analyzing communications data

• US 20020105911A1
• Filed: 05/23/2001
• Published: 08/08/2002
• Est. Priority Date: 11/24/1998
• Status: Active Grant

• Current Assignee: Niksun, Inc.
• Original Assignee: Niksun, Inc.
• Inventors: Heybey, Andrew, Pruthi, Parag, Kurien, Thekkthalackal V.
• Granted Patent: US 7,492,720 B2
• Time in Patent Office: Days
• US Class Current: 370/241
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 41/5003 Managing SLA; Interaction between SLA and QoS
• CPC Class Codes: H04L 41/5029 Service quality level-based billing, e.g. dependent on measured service level customer is charged more or less
• CPC Class Codes: H04L 41/5061 characterised by the interaction between service providers and their network customers, e.g. customer relationship management
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 related to network traffic
• CPC Class Codes: H04L 43/0829 Packet loss
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 43/0858 One way delays
• CPC Class Codes: H04L 43/0864 Round trip delays
• CPC Class Codes: H04L 43/0888 Throughput
• CPC Class Codes: H04L 43/091 Measuring contribution of individual network components to actual service level
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: Y04S 40/00 Systems for electrical power generation, transmission, distribution or end-user application management characterised by the use of communication or information technologies, or communication or information technology specific aspects supporting them

Abstract

A method for monitoring data on a first communication line. Data is received from the first communication line and a plurality of packets are extracted from the data. Statistics are then recursively generated, the statistics corresponding to the plurality of packets.

Claims

1. A method for collecting and analyzing first data on a first communication line comprising the steps of:

(a) receiving the first data from the first communication line;

(b) segregating the first data into packets;

(c) selecting packets based on a respective first characteristic of each packet;

(d) associating a respective index with each of the selected packets;

(e) converting each of the selected packets into a respective record including its respective index;

(f) storing the records;

(g) storing a count based on a respective second characteristic of each of the selected packets received during each of a plurality of successive periods.

2. A method for monitoring data on a first communication line comprising:

receiving the data from the first communication line;

extracting a plurality of packets from the received data; and

recursively generating statistics corresponding to the plurality of packets.

3. A method for collecting and analyzing first data transmitted on a first communication line comprising the steps of:

(a) receiving the first data from the first communication line;

(b) segregating the first data into packets;

(c) selecting packets based on a respective type of each packet;

(d) associating a respective index with each of the selected packets;

(e) converting each of the selected packets into a respective record including its respective index;

(f) storing the records;

(g) receiving a first duration value corresponding to a first time period;

(h) generating at least one statistic corresponding to packets received during each successive first time periods; and

(i) storing the value of at least one statistic for a plurality of successive time periods.

4. The method according to claim 3 further comprising the step of generating a further statistic using at least one of the stored records and the stored at least one statistic.

5. The method according to claim 3 further comprising the steps of:

receiving a second duration value corresponding to a second time period longer than the first time period; and

generating a further statistic corresponding to packets received during the second time period encompassing at least one first time period using at least one of the stored records and the stored at least one statistic.

6. The method according to claim 3 wherein the at least one statistic includes at least one of a number of bits, a number of bytes, a number of packets, a number of packets of a predetermined protocol, a number of packets of a predetermined protocol layer, a number of packets having a predetermined source address, and a number of packets having a predetermined destination address of a respective packet.

7. The method according to claim 3 wherein a record includes a portion of its respective packet plus its respective index.

8. The method according to claim 3 wherein the respective index of a packet corresponds to a time when the packet was received.

9. A method according to claim 8 further comprising the steps of receiving a time signal from a global positioning satellite indicating an absolute time and generating the time when a packet was received based on the absolute time.

10. A method according to claim 3 wherein the received packets originated from a second communication line and each received packet includes an indication of a respective first time corresponding to its transmission on the second communication line, the method further comprising:

determining a respective second time when each packet is received from the first communication line; and

generating a transmission delay corresponding to each packet by subtracting its respective second time from its respective first time.

11. A method according to claim 10 wherein the transmission delay corresponding to each packet is calculated based on a time the packet is detected on a first link, a time the packet is detected on a second link, a rate of data transmission on the first link, and a length of the packet on the first link.

12. The method according to claim 3 wherein step (a) includes receiving the first data from the first communication line using a non-intrusive coupling to the first communication line.

13. The method according to claim 3 further comprising the step of receiving information indicating at least one of a type of packets to be selected in step (c), the first duration value, and an identification of which statistics to generate in step (h).

14. The method according to claim 13 wherein the information is received via the first communication line.

16. A method according to claim 10 including the step of receiving a time signal from a global positioning satellite for determining the second time.

17. A method according to claim 3 wherein step (h) includes generating at least one quality of service value for each successive time period corresponding to packets of a predetermined packet type received during each respective successive time period.

18. A method according to claim 17 wherein the at least one quality of service value indicates at least one of a round trip delay and a retransmit rate corresponding to the packets of the predetermined packet type received during the respective successive time period.

19. A method according to claim 3 wherein step (c) includes determining a respective type of each packet based on at least one of a corresponding application type, a corresponding packet content, a packet source, and a packet destination.

20. A method according to claim 3 wherein the first communication line is in a network including a user and step (h) includes generating at least one user profiling statistic for each successive time period corresponding to packets destined to or from a user which are received during each respective successive time period.

21. A method according to claim 20 wherein the at least one user profiling statistic is generated using the records stored in the memory.

22. A method according to claim 3 further comprising the step of receiving an input from a user identifying a user statistic and step (h) includes generating the user statistic.

23. A method according to claim 3 further comprising the steps of:

receiving an input from a user identifying a user statistic; and

generating the user statistic using at least one of the stored statistics and the stored records.

24. A method according to claim 3 wherein the first data is transmitted on the first communication line according to a first protocol, the method further comprising the step of receiving decoding parameters corresponding to the first protocol and step (a) includes receiving the first data according to the decoding parameters and step (b) includes segregating the received first data into packets according to the decoding parameters.

25. A method according to claim 3 wherein

step (h) includes generating a separate count for each of a plurality of packet types corresponding to a number of packets of each respective one of the plurality of packet types received during each respective successive time period;

and the method further comprises the step of

displaying the plurality of packet types and their corresponding statistics accumulated over a second time period greater than the first time period in a first portion of a display area of a display device.

26. A method according to claim 25 wherein the plurality of packet types are selectable by a user and the method further comprises the steps of:

receiving a user selection of one of the packet types by a user;

displaying statistics corresponding to packets of the selected packet type received during the second time period.

27. A method according to claim 25 further comprising the step of graphically displaying the statistics corresponding to the plurality of packet types in a plot over the second time period in a second portion of the display area.

28. A method according to claim 25 further comprising the step of:

generating a graphical display of the statistics corresponding to the plurality of packet types in a plot ranging over the second time period in a second portion of the display area, wherein the plot includes a plurality of discrete selectable time periods;

receiving a user selection of a range of the plurality of discrete selectable time periods;

updating the graphical display to display statistics corresponding to the selected time periods.

29. A method according to claim 3 further comprising the steps of:

identifying packets belonging to a data stream;

storing stream identification information as one of a separate record and a separate field in the records corresponding to the identified packets.

30. A method according to claim 29 further comprising the step of recreating the data stream using the stored records and the stored stream identification information.

31. A method according to claim 3 wherein

the first communication line is in a network including a second communication line,

step (d) includes determining a respective time value corresponding to a time when each of the selected packets was received and the respective index associated with each selected packet includes its respective time value,

and step (e) includes converting each of the selected packets into a respective record which includes a uniquely identifiable portion of the respective selected packet;

and the method further comprises the steps of;

receiving second data from the second communication line;

segregating the second data into packets;

determining a respective type of each packet received from the second communication line;

selecting packets received from the second communication line based on their respective type;

determining a respective time when each of the selected packets received from the second communication line was received;

associating a respective index with each of the selected packets received from the second communication line, the respective index corresponding to a time when the packet was received from the second communication line;

convert each of selected packets received from the second communication line into a respective record that includes a uniquely identifiable portion the selected packet from the second communication line and includes its respective index;

storing the records corresponding to the selected packets received from the second communication line in a third memory;

comparing the uniquely identifiable portions of packets in records stored in the memory to those in records stored in the third memory to determine which selected packets received from the first communication line correspond to packets received from the second communication line to generate matched pairs of packets;

generating a transmission delay corresponding to each matched pair of packets using their respective indexes.

32. The method according to claim 31 wherein the transmission delay for a particular matched packet is calculated based on a time of receipt of the matched packet on the first transmission line, a time of receipt of the matched packet on the second transmission line, a rate of data transmission on the first transmission line, and a length of the matched packet on the first transmission line.

33. A method according to claim 32 wherein the first communication line transmits data using a first protocol and the second communication line transmits data using a second protocol different from the first protocol.

34. A method according to claim 31 wherein the transmission delay is generated at a level of accuracy less than 10 microseconds.

37. The method according to claim 36 wherein the packets correspond to a plurality of protocols at a protocol layer wherein step (a) includes displaying a table listing each of the plurality of protocols and a corresponding number of packets.

38. The method according to claim 37, further comprising the steps of detecting selection of one of the plurality of protocols and updating at least one of the table to display packet properties corresponding to packets of the selected protocol and the plot to display a packet properties of the selected protocol.

39. The method according to claim 36 further comprising the steps of:

receiving a selection of a sub-interval of time less than the interval of time by one of entry of times in a text field and selection of a time interval along the time axis of the plot; and

updating at least one of the tables and the plots responsive to the selected sub-interval of time.

40. The method according to claim 37, further comprising the steps of detecting selection of one of the plurality of protocols and updating the plot to simultaneously display both the packet properties of the selected protocol and the packet properties of all protocols.

41. The method according to claim 36 further comprising the steps of receiving a start time and an end time spanning a duration longer than the time interval and updating at least one of the tables and the plots to correspond to the duration.

42. The method according to claim 36 wherein a subset of the packets correspond to a host and the method further comprises detecting selection of the host and displaying statistics corresponding to the subset of packets.

43. The method according to claim 36 wherein the packets correspond to a plurality of protocols and a subset of the packets correspond to one of the plurality of the protocols and the method further comprises detecting selection of the one of the plurality of the protocols and displaying statistics corresponding to the selected one of the plurality of protocols.

79. A method according to claim 3, further comprising the steps of:

(l) selecting packets corresponding to particular communication sessions; and

(m) identifying which of the packets selected in step (l) correspond to unsuccessful communication sessions.

15. Canceled.

35. A method for routing first data from a first point to a second point in a communication network including a plurality of paths between the first point and the second point, the method comprising the steps of:

generating flow statistics based on the flow of second data on the plurality of paths between the first point and the second point, the second data being separate from the first data;

routing the first data between the first and second points based on the generated flow statistics.

36. A method of displaying one or more packet properties or statistics on a display device having a display area, the one or more packet properties or statistics corresponding to packets received from a first communication line during an interval of time, the method comprising:

(a) displaying a table listing the one or more packet properties or statistics in a first portion of the display area; and

(b) displaying a plot of the one or more packet properties or statistics over the time interval in a second portion of the display area, the plot having two axes, the packet properties or statistics being represented on one axis of the plot, and time represented on the other axis of the plot.

44. A method of synchronizing a host clock on a host computer with an interface clock on a network interface card comprising the steps of:

receiving an interface clock value from the network interface card;

generating an adjusted interface clock value by subtracting a predetermined service time from the interface clock value;

providing the adjusted interface clock value to the host computer;

receiving a host clock value;

correlating the adjusted interface clock value with the host clock value.

46. The method according to claim 45 wherein step (c) includes extracting a uniquely identifying portion of each of the first plurality and second plurality of packets and comparing the uniquely identifying portions of the first plurality of packets to those of the second plurality of packets to generate matched pairs of packets.

47. The method according to claim 45 wherein the first plurality of packets are received by a first network monitor, the second plurality of packets are received by a second network monitor, step (a) includes time stamping the first plurality of packets, step (b) includes time stamping the second plurality of packets, and the method further comprises the step of synchronizing the first and second network monitors with an absolute clock reference.

48. The method according to claim 47 wherein the network monitors are synchronized by receiving a time signal from a global positioning satellite.

50. A system according to claim 49 wherein the first clock and the second clock are synchronized within one microsecond of each other.

51. A system according to claim 49 wherein the synchronizing means comprises:

a receiver associated with the first network monitor for receiving a time signal from a common clock reference; and

a receiver associated with the second network monitor for receiving a time signal from the common clock reference.

52. A system according to claim 51 wherein the common clock reference is a global positioning satellite system.

45. A method of monitoring communications on a network having a first communication line and a second communication line different from the first communication line, the method comprising the steps of:

(a) receiving a first plurality of packets from the first communication line;

(b) receiving a second plurality of packets from the second communication line; and

(c) correlating the first plurality and the second plurality of packets.

49. A system for monitoring communication on a network including a first communication line and a second communication line, the system comprising:

a first network monitor having a first clock and coupled to the first communication line for receiving first data from the first communication line;

a second network monitor having a second clock and coupled to the second communication line for receiving second data from the second communication line;

synchronizing means for synchronizing the first clock with the second clock.

53. A method for billing a user of a communication network comprising steps of:

determining a quality of service metric corresponding to the user'"'"'s use of the communication network;

billing the user based on the determined quality of service.

54. A method of transmitting data onto a communication line comprising the steps of:

associating a time with the data;

generating a packet including a plurality of fields, wherein at least one field corresponds to the associated time and at least one field corresponds to the data;

transmitting the packet onto the communication line.

55. An apparatus for transmitting data onto a communication line comprising:

a clock for providing a time value corresponding to the data;

an encoder for encoding a packet, the packet including a plurality of fields, wherein at least one field corresponds to the time value and at least one field corresponds to the data;

a transmitter for providing the packet to a communication line.

57. The system of claim 56, wherein the recording means includes:

means for detecting the packets travelling over a selected one of the network paths;

means for indexing the detected packets;

means for creating records corresponding to the packets; and

means for storing the records in memory.

58. The system of claim 57, wherein the detecting means includes means for determining packet properties and means for selecting certain ones of the data packets traveling the network path and having a predetermined packet property.

59. The system of claim 57, wherein the indexing means comprises means for timestamping the data packets.

60. The system of claim 59, in which the network is monitored at two points on the network, the system further comprising timestamping means at the two points, and wherein the timestamping means comprises GPS means for synchronizing the indexing of the data packets at the two points.

61. The system of claim 57 further comprising a computer with a data storage device, a network card having network card memory, and a CPU in logical communication with the network card, and wherein the record storing means comprises:

means for storing the records on the network card memory;

means for detecting relative inactivity of the CPU; and

means for transferring records stored on the network card memory to the data storage device in response to detection of the relative inactivity of the CPU.

62. The system of claim 61, wherein the record storing means further comprises:

a long-term storage device having replaceable storage media loaded therein so as to receive data thereon, and a magazine of additional storage media operatively connected to the long-term storage device;

means for detecting when the data storage device of the computer contains a predetermined amount of data corresponding to the recorded packets, said predetermined amount of data corresponding to the data storage device being full;

means for transferring said data from the data storage device to the long-term storage device in response to detecting the data storage device is full;

means for detecting when the replaceable media loaded in the long-term storage device is full; and

means for replacing the full storage media with storage media from the magazine.

63. The system of claim 56, wherein the displaying means includes a computer display screen having a graphical user interface and a control program for the graphical user interface, the control program having:

means for displaying at least some of the packet properties over time on a graph having two axes, the packet properties being represented on one axis of the graph, the time being represented on the other axis of the graph;

means for varying the timescale of the graph in response to user selection; and

means for displaying at least some of the statistical information in at least one table adjacent to the graph, the tables having user-selectable cells therein; and

means, responsive to selection of one of the cells, for displaying additional statistical information corresponding to the selected cell.

56. A system for monitoring a computer network over an interval of time, in which data packets travel across network paths during such time interval, the system comprising:

means for recording the packets substantially continuously during the time interval;

means for extracting packet properties from the recorded packets at multiple, successive sampling times during said time interval, the time from the beginning of the time interval to successive sampling times defining subintervals of time during which the network has been monitored, the subintervals being increasingly longer as packet properties are extracted over successive sampling times during the time interval;

means for tabulating the extracted packet properties for the subintervals to create statistical information;

means for altering the recording of the packets and the extraction of packet properties in response to user selections;

means for creating, during the time interval, data corresponding to graphical representations of the statistical information in response to the creation of the statistical information;

means for displaying said graphical representations in response to the creation of said graphical representations.

64. A system for monitoring data on a first communication line comprising:

a packetizer for receiving the data from the first communication line and extracting a plurality of packets from the data; and

means for recursively generating statistics corresponding to the plurality of packets.

65. A system according to claim 64 wherein the means for recursively generating statistics comprises:

means for generating records corresponding to the plurality of packets;

a filter for selecting packets from the plurality of packets received by the system during a time period;

means for generating first statistics corresponding to the plurality of packets received by the system during the time period;

a memory for storing at least one of the selected packets, the records corresponding to the plurality of packets, and the first statistics;

a packet type identifier for identifying a type of packets received from one of the packetizer, the record generator, and the memory;

means for generating second statistics using at least one of the first statistics and the received packets.

66. A system according to claim 64 further comprising a display device for displaying the generated statistics.

67. A system according to claim 65 further comprising a user interface for defining at least one of the first statistics, the second statistics, and the time period.

68. A system according to claim 65 further comprising:

a time source synchronized to an absolute time source; and

means for determining a time when each respective one of the plurality of packets was received from the first communication line.

69. A system according to claim 68 wherein the time source is a receiver for receiving a time signal from a global positioning satellite.

70. An article of manufacture comprising a computer useable medium having computer readable program code means embodied therein for monitoring data on a first communication line, the computer readable program code means in said article of manufacture comprising computer readable program code means for causing a computer to effect:

receiving the data from the first communication line;

extracting a plurality of packets from the received data; and

recursively generating statistics corresponding to the plurality of packets.

71. A computer program product comprising a computer useable medium having computer readable program code means embodied therein for monitoring data on a first communication line, said computer readable program code means in said computer program product comprising computer readable program code means for causing a computer to effect:

receiving the data from the first communication line;

extracting a plurality of packets from the received data; and

recursively generating statistics corresponding to the plurality of packets.

72. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for monitoring data on a first communication line, said method steps comprising:

receiving the data from the first communication line;

extracting a plurality of packets from the received data; and

recursively generating statistics corresponding to the plurality of packets.

73. An article of manufacture comprising a computer useable medium having computer readable program code means embodied therein for displaying packet properties on a display device having a display area, the packet properties corresponding to packets received from a first communication line during an interval of time, the computer readable program code means in said article of manufacture comprising computer readable program code means for causing a computer to effect:

(a) displaying a table listing the packet properties in a first portion of the display area; and

(b) displaying a plot of the packet properties over the time interval in a second portion of the display area, the plot having two axes, the packet properties being represented on one axis of the plot, and time represented on the other axis of the plot.

74. A computer program product comprising a computer useable medium having computer readable program code means embodied therein for causing displaying packet properties on a display device having a display area, the packet properties corresponding to packets received from a first communication line during an interval of time, said computer readable program code means in said computer program product comprising computer readable program code means for causing a computer to effect:

(a) displaying a table listing the packet properties in a first portion of the display area; and

(b) displaying a plot of the packet properties over the time interval in a second portion of the display area, the plot having two axes, the packet properties being represented on one axis of the plot, and time represented on the other axis of the plot.

75. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for displaying packet properties on a display device having a display area, the packet properties corresponding to packets received from a first communication line during an interval of time, said method steps comprising:

(a) displaying a table listing the packet properties in a first portion of the display area; and

(b) displaying a plot of the packet properties over the time interval in a second portion of the display area, the plot having two axes, the packet properties being represented on one axis of the plot, and time represented on the other axis of the plot.

76. An article of manufacture comprising a computer useable medium having computer readable program code means embodied therein for collecting and analyzing first data transmitted on a first communication line, the computer readable program code means in said article of manufacture comprising computer readable program code means for causing a computer to effect:

(a) receiving the first data from the first communication line;

(b) segregating the received first data into packets;

(c) determining a respective type of each packet;

(d) selecting packets based on their respective type;

(e) associating a respective index with each of the selected packets;

(f) converting each of the selected packets into a respective record including its respective index;

(g) storing the records in a first memory;

(h) receiving a first duration value corresponding to a first time period;

(i) generating at least one statistic corresponding to packets received during each successive time period; and

(j) storing the at least one statistic for each successive time period in a second memory.

77. A computer program product comprising a computer useable medium having computer readable program code means embodied therein for collecting and analyzing first data transmitted on a first communication line, said computer readable program code means in said computer program product comprising computer readable program code means for causing a computer to effect:

(a) receiving the first data from the first communication line;

(b) segregating the received first data into packets;

(c) determining a respective type of each packet;

(d) selecting packets based on their respective type;

(e) associating a respective index with each of the selected packets;

(f) converting each of the selected packets into a respective record including its respective index;

(g) storing the records in a first memory;

(h) receiving a first duration value corresponding to a first time period;

(i) generating at least one statistic corresponding to packets received during each successive time period; and

(j) storing the at least one statistic for each successive time period in a second memory.

80. A method for processing data according to claim 79 wherein:

step (l) comprises selecting packets corresponding to TCP sessions; and

step (m) comprises identifying packets corresponding to unsuccessful TCP sessions; and

the method further comprise the step of;

(n) generating a ratio of a quantity of unsuccessful TCP sessions to a quantity of total TCP sessions during a time period.

81. A method according to claim 80 wherein in step (m) the unsuccessful TCP sessions are identified to be those TCP sessions that were closed by other than a corresponding server.

78. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for collecting and analyzing first data transmitted on a first communication line, said method steps comprising:

(a) receiving the first data from the first communication line;

(b) segregating the received first data into packets;

(c) determining a respective type of each packet;

(d) selecting packets based on their respective type;

(e) associating a respective index with each of the selected packets;

(f) converting each of the selected packets into a respective record including its 11 respective index;

(g) storing the records in a first memory;

(h) receiving a first duration value corresponding to a first time period;

(i) generating at least one statistic corresponding to packets received during each successive time period; and

(j) storing the at least one statistic for each successive time period in a second memory.