Auto-detection of limiting factors in a TCP connection

• US 6,772,375 B1
• Filed: 12/22/2000
• Issued: 08/03/2004
• Est. Priority Date: 12/22/2000
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Banga, Gaurav
• Primary Examiner(s): IQBAL, NADEEM
• Application Number:
  US09/748,070
• Time in Patent Office: 1,320 Days
• Field of Search: 714/47, 714/56, 714/51, 714/18, 714/20, 714/21, 714/36, 714/37, 714/39, 709/220, 709/221, 709/223, 709/225, 709/229, 709/234
• US Class Current: 714/47.2
• CPC Class Codes: H04L 43/0841 Round trip packet loss
• CPC Class Codes: H04L 43/0864 Round trip delays
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/163 In-band adaptation of TCP data exchange; In-band control procedures

Abstract

This invention comprises auto-diagnosis logic that can be implemented in operating systems in an appliance-like auto-diagnosis module coupled to the TCP receiver, the TCP sender or both. TCP events are sampled and a set of statistics on these events is maintained. Receiver side TCP diagnostic techniques include detecting sender'"'"'s re-transmission timeouts, evaluating the average size of packets being received, determining if a receiver is a bottleneck, and performing other evaluations of an incoming data stream. Sender side diagnostic techniques include flagging transmission timeouts, monitoring the average size of a transmitted packet, evaluating if the advertised window accounts for the delay-bandwidth product of the network connecting the receiver and the sender systems, performing bottleneck checks, and other evaluations of an outgoing data stream. The results are aggregated using system attributes. Systems with common problem areas and attributes are grouped together. The TCP auto-diagnosis logic can be performed on-line or off-line.

Claims

1. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP receiver;

at least one second computing device, including a processor and a memory;

wherein said second computing device is a TCP sender;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein the instructions for defining said set of data comprise instructions to perform at least one of

determining average packet size of incoming data packets, and

determining whether the first computing device creates a bottleneck in data transfer.

2. An apparatus as in claim 1, wherein said instructions for defining a set of data concerning TCP operations includes defining a set of events to monitor and a time granularity in which said events will be monitored.

4. An apparatus as in claim 1, wherein said set of instructions for evaluating said set of data concerning TCP operations includes a subset of instructions for evaluating the average packet size of incoming packets.

6. An apparatus as in claim 1, wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for checking whether the first computing device causes a bottleneck check in data transfer between the first and second computing devices.

10. An apparatus as in claim 1, wherein said set of instructions also includes a subset of instructions to aggregate said result, the subset including instructions to identify at least one of following items:

one or more attributes of client systems; and

one or more problems that occurred within the client systems.

11. An apparatus as in claim 10, wherein said subset of instructions to aggregate includes an instruction to aggregate based upon information stored in said database.

12. An instruction as in claim 10, wherein said subset of instructions to aggregate include an instruction to aggregate based upon said result.

32. Apparatus as in claim 10, wherein the subset of instructions to perform aggregation includes instructions to identify at least one IP subnet number of a client system.

33. Apparatus as in claim 10, wherein the subset of instructions to perform aggregation includes instructions to identify type and version of OS of at least one client system.

34. Apparatus as in claim 10, wherein the subset of instructions to perform aggregation includes instructions to identify last configuration change date of at least one client system.

35. Apparatus as in claim 10, wherein the subset of instructions to perform aggregation includes instructions to identify delay distance to at least one client system.

36. Apparatus as in claim 10, wherein the subset of instructions to perform aggregation includes instructions to identify virtual LAN information of at least one client system.

37. Apparatus as in claim 10, wherein the subset of instructions to perform aggregation includes instructions to generate a historical summary of auto-diagnosis information.

3. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP receiver;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP sender;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes a subset of instructions for detecting retransmission timeouts at a TCP receiver, said subset of instructions for detecting retransmission timeouts includes

an instruction to tag each packet that is inserted into a TCP reassembly queue with a timestamp and a sequence number;

an instruction to read each tag and sort so as to identify said tags that are not in sequential order;

an instruction to generate a flag for every said tag that is not in sequential order; and

store said flags in said memory included in said first computing device.

5. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP receiver;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP sender;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes a subset of instructions for evaluating the average packet size of incoming packets, said subset of instructions for evaluating the average packet size includes

an instruction to identify packets that are included in burst of back-to-back packets;

an instruction to evaluate the size of said identified packets;

an instruction to generate a flag if any packet included in said burst of back-to-back packets falls below a pre-set threshold; and

an instruction to store said flags in said memory included in said first computing device.

7. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP receiver;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP sender;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for performing a bottleneck check at a receiver TCP, one of said at least three subsets of instructions includes

an instruction to monitor the advertised window size in said first computing device;

an instruction to monitor the rate at which packets arrive at said first computing device;

an instruction to generate a flag if the size of said advertised window falls below a certain threshold while said rate at which packets arrive remains within a predefined range; and

an instruction to store said flags in said memory included in said first computing device.

8. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP receiver;

at least one second computing device, including a processor and a memory wherein said second computing device is a TCP sender;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for performing a bottleneck check at a receiver TCP, one of said at least three subsets of instructions includes

an instruction to monitor the number of cycles associated with said processor included in said first computing device, that are utilized for processing of TCP protocol;

an instruction to monitor the amount of cycles of said processor associated with said first computing device that is characterized as idle time;

an instruction to compare the result of monitoring the amount of cycles of said processor associated with said first computing device that is characterized as idle time with the result of monitoring the number of cycles of said processor included in said first computing device that is utilized for processing of TCP protocol;

an instruction to generate a flag if the ratio between the amount of cycles of said processor associated with said first computing device that is characterized as idle time and the number of cycles of said processor included in said first computing device that is utilized for processing of TCP protocol falls below a pre-set threshold; and

an instruction to store said flag in said memory included in said first computing device.

9. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP receiver;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP sender;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for performing a bottleneck check at a receiver TCP, one of said at least three subsets of instructions includes

an instruction to monitor the rate at which said first computing device reads from the TCP buffers;

an instruction to compare said rate at which said first computing device reads to a standard rate;

an instruction to generate a flag if said rate at which said first computing device reads exceeds said standard rate by a predetermined value; and

an instruction to store said flag in said memory included in said first computing device.

13. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory;

wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device.

14. An apparatus as in claim 13, wherein said instructions for defining a set of data concerning TCP operations includes defining a set of events to monitor and a time granularity in which said events will be monitored.

16. An apparatus as in claim 13, wherein said set of instructions for evaluating said set of data concerning TCP operations includes a subset of instructions for evaluating the average packet size of outgoing packets.

18. An apparatus as in claim 13, wherein said set of instructions includes a subset of instructions for monitoring the receiver window associated with said second computing device.

20. An apparatus as in claim 13, wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for checking whether the first computing device causes a bottleneck check in data transfer between the first and second computing devices.

24. An apparatus as in claim 13, wherein said set of instructions also includes instructions to aggregate said result, said instructions to aggregate including instructions to identify at least one of following items:

one or more attributes of one or more client systems; and

one or more problems that occurred within the one or more client systems.

25. An apparatus as in claim 24, wherein said instructions to aggregate include an instruction to aggregate based upon information stored in said database.

26. An apparatus as in claim 24, wherein said instructions to aggregate include an instruction to aggregate based upon said result.

38. Apparatus as in claim 24, wherein the subset of instructions to perform aggregation includes instructions to identify at least one IP subnet number of a client system.

39. Apparatus as in claim 24, wherein the subset of instructions to perform aggregation includes instructions to identify type and version of OS of at least one client system.

40. Apparatus as in claim 24, wherein the subset of instructions to perform aggregation includes instructions to identify last configuration change date of at least one client system.

41. Apparatus as in claim 24, wherein the subset of instructions to perform aggregation includes instructions to identify delay distance to at least one client system.

42. Apparatus as in claim 24, wherein the subset of instructions to perform aggregation includes instructions to identify virtual LAN information of at least one client system.

43. Apparatus as in claim 24, wherein the subset of instructions to perform aggregation includes instructions to generate a historical summary of auto-diagnosis information.

15. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes a subset of instructions for detecting retransmission timeouts at said first computing device, said subset of instructions for detecting retransmission timeouts includes

an instruction to determine when a re-transmission time expires and packets are retransmitted;

an instruction to measure the number of re-transmissions occurred within a predetermined time period;

an instruction to generate a flag if said number of re-transmissions exceeds a predetermined threshold; and

an instruction to store said flags in said memory included in said first computing device.

17. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes a subset of instructions for evaluating the average packet size of outgoing packets, said subset of instructions for evaluating the average packet size includes

an instruction to identify packets that are included in burst of back-to-back packets;

an instruction to evaluate the size of said identified packets;

an instruction to generate a flag if any packet included in said burst of back-to-back packets falls below a pre-set threshold; and

an instruction to store said flags in said memory of said first computing device.

19. An apparatus, including

at least one first computing device, including a processor and a memory wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions includes a subset of instructions for monitoring the receiver window associated with said second computing device, said subset of instructions for monitoring said receiver window includes

an instruction to monitor the variation of space available in said receiver window over time;

an instruction to generate a flag if said variation in space available in said window fits a particular predetermined pattern; and

an instruction to store said flags in said memory included in said first computing device.

21. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for performing a bottleneck check at a sender TCP, one of said at least three subsets of instructions includes

an instruction to monitor the size of a buffer associated with said first computing device;

an instruction to monitor the rate at which packets are transmitted;

an instruction to generate a flag if the space available in said buffer falls below a certain threshold while said rate at which packets are transmitted remains within a predefined range;

an instruction to store said flag in said memory included in said first computing device.

22. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for performing a bottleneck check at a sender TCP, one of said at least three subsets of instructions includes

an instruction to monitor the amount of unsent queued data in a buffer associated with said first computing device;

an instruction to count the number of times that the first computing device is unable to transmit said unsent queued data during a pre-set interval of time;

an instruction to generate a flag if said number of times said first computing device is unable to transmit exceeds a pre-set threshold; and

an instruction to store said flag in said memory included in said first computing device.

23. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said first computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for performing a bottleneck check at a sender TCP, wherein one of at least three subsets of instructions includes

an instruction to monitor the rate that said first computing device performs protocol processing;

an instruction to monitor the data consumption bandwidth of said second computing device;

an instruction to compare said rate of protocol processing with said data consumption bandwidth;

an instruction to generate a flag if the ratio between said rate and said bandwidth exceeds a pre-set threshold; and

an instruction to store said flag in said memory included in said first computing device.

27. A method, including steps of

defining a set of data concerning TCP operations;

evaluating the performance of a TCP receiver or a TCP sender;

generating a set of data regarding said performance; and

aggregating said set of data according to at least one of following items;

one or more attributes of one or more client systems; and

one or more problems that occurred within the one or more client systems.

28. A method as in claim 27, wherein said step of defining a set of data includes steps of

defining a set of events to monitor; and

defining a time granularity in which said set of events will be monitored.

29. A method as in claim 27, wherein said step of evaluating said performance includes steps of

detecting retransmission timeouts at a TCP receiver by looking to the time and order in which data packets are received

evaluating the average size of incoming packets included in a burst of back-to-back packets;

monitoring the advertised window size of said TCP receiver and comparing the advertised window size to the rate at which said data packets are received;

monitoring the number of processor cycles used to perform TCP protocols and comparing the number of processor cycles used to perform TCP protocols to the number of cycles characterized as idle time; and

monitoring the rate at which said TCP receiver reads from a buffer and comparing the rate at which said TCP receiver reads from the buffer to a standard rate.

30. A method as in claim 27, wherein said step of evaluating said performance includes steps of

detecting retransmission timeouts by monitoring when a re-transmission time expires and packets are re-transmitted by the TCP sender;

evaluating the average size of outgoing packets by monitoring the size of said outgoing packets included in a burst of back-to-back outgoing packets;

monitoring the size of a receiver window associated with the TCP receiver;

monitoring patterns in the size of said receiver window;

monitoring the size of a buffer associated with said TCP sender and comparing to the rate at which said packets are transmitted;

monitoring the amount of unsent queued data in said buffer associated with the TCP sender; and

monitoring the rate at which protocol processing is performed and comparing the rate at which protocol processing is performed to data consumption bandwidth.

31. A method as in claim 27, wherein said step of aggregating includes aggregating based upon (1) attributes of a computing device and (2) results associated with said set of data.

44. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said second computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes a subset of instructions for evaluating the average packet size of outgoing packets, said subset of instructions for evaluating the average packet size includes

an instruction to identify packets that are included in burst of back-to-back packets;

an instruction to evaluate the size of the identified packets;

an instruction to generate a flag if any packet included in said burst of back-to-back packets falls below a pre-set threshold; and

an instruction to store said flags in said memory.

45. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said second computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions includes a subset of instructions for monitoring the receiver window associated with said second computing device, said subset of instructions for monitoring said receiver window includes

an instruction to monitor the variation of space available in said receiver window over time;

an instruction to generate a flag if said variation in space available in said window fits a particular predetermined pattern; and

an instruction to store said flags in said memory included in said second computing device.

46. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said second computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for performing a bottleneck check at a sender TCP, one of said at least three subsets of instructions includes

an instruction to monitor the size of a buffer associated with said first computing device;

an instruction to monitor the rate at which packets are transmitted;

an instruction to generate a flag if the space available in said buffer falls below a certain threshold while said rate at which packets are transmitted remains within a predefined range;

an instruction to store said flag in said memory associated with said second computing device.

47. An apparatus, including

at least one first computing device, including a processor and a memory, wherein said first computing device is a TCP sender;

at least one second computing device, including a processor and a memory, wherein said second computing device is a TCP receiver;

a communications network coupled to said first computing device and said second computing device;

a set of instructions coupled to said memory associated with said second computing device, wherein said set of instructions includes instructions for (1) defining a set of data concerning TCP operations, (2) evaluating said set of data concerning TCP operations, and (3) generating a result of said evaluating; and

a database coupled to said memory associated with said first computing device;

wherein said set of instructions for evaluating said set of data concerning TCP operations includes at least three subsets of instructions for performing a bottleneck check at a sender TCP, one of said at least three subsets of instructions includes

one of said at least three subsets of instructions includes

an instruction to monitor the amount of unsent queued data in a buffer associated with said first computing device;

an instruction to count the number of times that the first computing device is unable to transmit said unsent queued data during a pre-set interval of time;

an instruction to generate a flag if said number of times said first computing device is unable to transmit exceeds a pre-set threshold; and

an instruction to store said flag in said memory associated with said second computing device.

48. A system for performing TCP diagnostics, comprising:

a processor coupled to a first TCP information appliance communicating with one or more second TCP information appliances over a communications network, and a memory coupled to the processor, the memory storing a set of instructions that, when executed by the processor, cause the processor to (1) define a set of data concerning TCP communications, the set of data including at least one of determining average packet size of data packets, and determining whether the first TCP information appliance creates a bottleneck in data transfer between the first TCP information appliance and the one or more second TCP information appliances, and (2) evaluate the set of data to generate a result.

49. A system in accordance with claim 48, further comprising a database coupled to the processor, wherein the set of instructions, when executed by the processor, further causes the processor to aggregate the set of data according to at least one of the following:

one or more attributes of the one or more second TCP information appliances; and

one or more problems in the one or more second TCP information appliances;

wherein the database stores the one or more attributes and the one or more problems used to aggregate the set of data.