Network packet tracing

• US 10,284,460 B1
• Filed: 12/28/2015
• Issued: 05/07/2019
• Est. Priority Date: 12/28/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Bshara, Nafea, Tracy, Leonard Thomas, Volpe, Thomas A., Davis, Mark Bradley, Kelly, Mark Noel, Callaghan, Stephen, Pietsch, Justin Oliver, Crabbe, Edward
• Primary Examiner(s): Kassim, Khaled M
• Application Number:
  US14/981,013
• Time in Patent Office: 1,226 Days
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 45/26 Route discovery packet

Abstract

Network packet tracing may be implemented on packet processors or other devices that perform packet processing. As network packets are received, a determination may be made as to whether tracing is enabled for the network packets. For those network packets with tracing enabled, trace information may be generated and the network packets modified to include the trace information such that forwarding decisions for the network packets ignore the trace information. Trace information indicate a packet processor as a location in a route traversed by the network packets and may include ingress and egress timestamps. Forwarding decisions may then be made and the network packets sent according to the forwarding decisions. Tracing may be enabled or disabled by packet processors for individual network packets. Trace information may also be truncated at a packet processor.

Claims

1. A networking device, comprising:

a plurality of physical network interfaces; and

a packet processor, configured to;

receive a network packet via one of the physical network interfaces;

evaluate the network packet to determine that tracing is enabled for the network packet;

generate trace information that identifies the packet processor as a location in a route traversed by the network packet;

insert the trace information into the network packet to generate a modified network packet, wherein the insertion is performed such that the trace information that is inserted into the network packet is ignored when a forwarding decision is determined by the packet processor for the modified network packet;

make a forwarding decision for the modified network packet that ignores the inserted trace information; and

send the modified network packet via one of the physical interfaces according to the forwarding decision.

2. The networking device of claim 1, wherein to generate the trace information, the packet processor is configured to determine an ingress timestamp that indicates when the network packet was received at the packet processor and an egress timestamp that indicates when the network packet was sent from the same packet processor.

3. The networking device of claim 1, wherein to insert the trace information into the network packet, the packet processor is configured to:

append the trace information to the network packet, wherein the trace information comprises a magic number to identify the trace information in the network packet; and

update the network packet to include a cyclic redundancy check (CRC) for the network packet that includes the trace information appended to the network packet.

4. The networking device of claim 1, further comprising:

at least one processor; and

a memory, comprising program instructions that when executed by the at least one processor cause the at least one processor to implement a controller for the packet processor;

wherein the packet processor is further configured to;

receive another network packet via one of the physical network interfaces;

evaluate the other network packet to determine that trace information for the other network packet is to be captured;

provide the other network packet to the controller;

wherein the controller is configured to;

locate the trace information in the other network packet; and

remove the trace information from the other network packet.

5. A method, comprising:

receiving, at a packet processor, a network packet;

determining, by the packet processor, that tracing is enabled for the network packet;

in response to determining that tracing is enabled for the network packet;

modifying, by the packet processor, the network packet to include trace information that identifies the packet processor as a location in a route traversed by the network packet, wherein the modification of the network packet is performed such that the trace information that is included in the network packet is ignored when determining by the packet processor a forwarding decision for the modified network packet;

determining, by the packet processor, a forwarding decision for the modified network packet that ignores the inserted trace information; and

sending, by the packet processor, the modified network packet according to the forwarding decision.

6. The method of claim 5, further comprising:

determining, by the packet processor, at least one of;

an ingress timestamp that indicates when the network packet was received at the packet processor;

or

an egress timestamp that indicates when the network packet was sent from the same packet processor; and

wherein the ingress timestamp or the egress time stamp are included in the trace information in the modified network packet.

7. The method of claim 5, further comprising:

determining, by the packet processor, a workload metric for the packet processor; and

wherein the workload metric for the packet processor is included in the trace information in the modified network packet.

8. The method of claim 5, further comprising:

receiving, at the packet processor, another network packet;

determining, by the packet processor, that tracing is enabled for the other network packet;

in response to the determining, evaluating, by the packet processor, the other network packet for sufficient space to include trace information that identifies the packet processor as a location in a route traversed by the other network packet;

in response to determining that sufficient space is not available in the other network packet to include the trace information, updating the other network packet to indicate that tracing is truncated for the other network packet;

determining, by the packet processor, a forwarding decision for the other network packet; and

sending, by the packet processor, the other network packet according to the forwarding decision for the other network packet, wherein the other network packet does not include the trace information for the other network packet.

9. The method of claim 5, further comprising:

receiving, at the packet processor, another network packet;

determining, by the packet processor, that tracing is disabled for the other network packet;

in response to determining that tracing is disabled, updating the other network packet to indicate that tracing is disabled;

determining, by the packet processor, a forwarding decision for the other network packet; and

sending, by the packet processor, the updated other network packet according to the forwarding decision for the other network packet.

10. The method of claim 5, further comprising:

prior to receiving the network packet, updating, by a networking device, the network packet to indicate that tracing is enabled for the network packet.

11. The method of claim 10, wherein the packet processor is implemented as part of another networking device in a provider network that also includes the networking device, wherein the network packet is transmitted on behalf of a customer of the provider network from the networking device, wherein the updating of the network packet to indicate that tracing is enabled is performed in response to receiving a request to enable tracing of network traffic for the customer of the provider network via a programmatic interface for the provider network.

12. The method of claim 5, further comprising:

receiving, at the packet processor, another network packet, wherein the other network packet includes trace information for one or more other networking devices that previously processed the other network packet;

providing, by the packet processor, the other network packet to a network monitor; and

performing, by one or more computing devices that implement the networking monitor;

locating the trace information in the other network packet for the one or more other networking devices; and

obtaining the trace information from the other network packet.

13. The method of claim 5, further comprising:

receiving, at the packet processor, another network packet, wherein the other network packet includes trace information for one or more other networking devices that previously processed the other network packet;

locating, by the packet processor, the trace information for the one or more other networking devices; and

obtaining, by the packet processor, the trace information for the one or more other networking devices.

14. A system, comprising:

a device configured to perform packet processing, the device comprising;

one or more network interfaces configured to transmit and receive packets via a network connection to the device; and

a packet processing pipeline, configured to;

receive a network packet via one of the one or more network interfaces;

evaluate the network packet to determine that tracing is enabled for the network packet;

modify the network packet to include trace information that identifies the device as a location in a route traversed by the network packet, wherein the modification of the network packet is performed such that the trace information that is included in the network packet is ignored when making a forwarding decision by the packet processing pipeline for the modified network packet;

make a forwarding decision for the modified network packet that ignores the inserted trace information; and

send the modified network packet according to the forwarding decision via one of the network interfaces.

15. The system of claim 14, wherein the modification of the network packet is performed such that content included in the network packet when received remains unchanged.

16. The system of claim 14, wherein the packet processing pipeline is further configured to:

receive another network packet via one of the one or more network interfaces;

determine that tracing is to be enabled for the other network packet;

in response to the determination;

update the other network packet to indicate that tracing is enabled for the other network packet;

modify the other network packet to include trace information that identifies the device as a location in a route traversed by the network packet, wherein the modification of the other network packet is performed such that the trace information is ignored when making a forwarding decision for the other network packet;

make a forwarding decision for the modified other network packet; and

send the modified other network packet according to the forwarding decision for the modified other network packet via one of the network interfaces.

17. The system of claim 16, wherein to determine that tracing is to be enabled for the other network packet, the packet processing pipeline is configured to randomly select the other network packet out of a plurality of network packets received at the device for packet processing.

18. The system of claim 16, wherein to determine that tracing is to be enabled for the other network packet, the packet processing pipeline is configured to identify a matching access control list (ACL) rule that for the other network packet that indicates that tracing is to be enabled for the other network packet.

19. The system of claim 16, wherein to update the other network packet to indicate that tracing is enabled for the other network packet, the packet processing pipeline is further configured to modify one of:

a Multicast bit for a source Media Access Control (MAC);

a field for a EtherType;

or

a reserved bit in Internet Protocol (IP) header flags.

20. The system of claim 14, wherein the packet processing pipeline is further configured to:

receive another network packet;

determine that tracing is enabled for the other network packet;

in response to the determination, evaluate the other network packet for sufficient space to include trace information that identifies the device as a location in a route traversed by the other network packet;

in response to a determination that sufficient space is not available in the other network packet to include the trace information, modify the other network packet to replace trace information that identifies one or more other devices to include the trace information that identifies the device;

make a forwarding decision for the other network packet; and

send the other network packet according to the forwarding decision for the other network packet, wherein the other network packet does not include the trace information for the other network packet.

21. The system of claim 14, wherein the packet processing pipeline is further configured to:

receive another network packet, wherein the other network packet includes trace information for one or more other networking devices that previously processed the other network packet;

remove the trace information for the one or more other networking devices;

evaluate the network packet to determine one or more packet characteristics;

make a forwarding decision for the other network packet; and

send the other network packet without the trace information according to the forwarding decision via one of the network interfaces.

22. The system of claim 21, wherein the trace information for the one or more other networking devices includes respective ingress and egress timestamps for the one or more other networking devices, and wherein the system further comprises one or more computing devices that implement a network monitor, the network monitor configured to:

receive the extracted trace information from the device; and

determine respective latencies for the one or more other networking devices according to the respective ingress and egress timestamps.

23. The system of claim 14, wherein the device is an application specific integrated circuit (ASIC), a system-on-chip (SoC), or a field-programmable gate array (FPGA).