Ultra-scalable, disaggregated internet protocol (IP) and ethernet switching system for a wide area network

• US 10,693,814 B2
• Filed: 09/17/2018
• Issued: 06/23/2020
• Est. Priority Date: 09/17/2018
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Cai, Biaodong
• Primary Examiner(s): Mohebbi, Kouroush
• Application Number:
  US16/132,996
• Publication Number:
  US 20200092228A1
• Time in Patent Office: 645 Days
• Field of Search: None
• CPC Class Codes: H04L 49/101 using crossbar or matrix
• CPC Class Codes: H04L 49/201 Multicast operation; Broadcast operation
• CPC Class Codes: H04L 49/351 for local area network [LAN], e.g. Ethernet switches
• CPC Class Codes: H04L 49/352 Gigabit ethernet switching [GBPS]
• CPC Class Codes: H04L 49/70 Virtual switches
• CPC Class Codes: H04L 49/9036 Common buffer combined with individual queues
• CPC Class Codes: H04L 49/9042 Separate storage for different parts of the packet, e.g. header and payload
• CPC Class Codes: H04L 9/0643 Hash functions, e.g. MD5, SHA, HMAC or f9 MAC
• CPC Class Codes: Y02D 30/00 Reducing energy consumption in communication networks

Abstract

Systems and Methods for IP and Ethernet switching in an ultra-scalable disaggregated wide area common carrier (WACC) disaggregated networking switching system. The WACC network switching system may include an Ethernet fabric having a set of M Ethernet switches each including a set of N switch ports, and a set of N input/output (IO) devices each including a set of W IO ports, a set of M Ethernet ports, an IO side packet processor (IOSP), and a fabric side packet processor (FSP). Each Ethernet switch may establish switch queues. Each IO device may establish a set of M hierarchical virtual output queues each including a set of N ingress-IOSP queues and ingress-virtual output queues, a set of W egress-IOSP queues, a set of M ingress-FSP queues, and a set of N hierarchical virtual input queues each including a set of N egress-FSP queues and egress-virtual input queues.

Claims

1. A wide area common carrier (WACC) disaggregated networking switching system comprising:

an Ethernet fabric including a set of M Ethernet switches each comprising a set of N switch ports, each Ethernet switch to establish switch queues, wherein a variable i having a value ranging from 1 to M to denote the ith Ethernet switch of the set of M Ethernet switches, wherein a variable j having a value ranging from 1 to N to denote the jth switch port of the set of N switch ports; and

a set of N input/output (IO) devices each comprising;

a set of W IO ports, wherein a variable x having a value ranging from 1 to W to denote the xth IO port of the W IO ports;

a set of M Ethernet ports, wherein the ith Ethernet port of the jth IO device is connected to the jth switch port of the ith Ethernet switch;

an IO side packet processor (IOSP) configured to;

establish a set of M hierarchical virtual output queues (H-VOQs) each comprising a set of N ingress-IOSP queues (I-IOSPQs) and I-VOQs, wherein the ith H-VOQ corresponds to the ith Ethernet port of the jth IO device, and wherein the jth I-IOSPQ of the ith H-VOQ corresponds to the jth IO device; and

establish a set of W egress-IOSP queues (E-IOSPQs), wherein the xth E-IOSPQ corresponds to the xth IO port; and

a fabric side packet processor (FSP) configured to;

establish a set of M ingress-FSP queues (I-FSPQs), wherein the ith I-FSPQ corresponds to the ith Ethernet switch; and

establish a set of N hierarchical virtual input queues (H-VIQs) each comprising a set of N egress-FSP queues (E-FSPQs) and E-VIQs, wherein the jth H-VIQ corresponds to the jth IO device, and wherein the jth E-FSPQ of the jth H-VIQ corresponds to the jth IO device.

2. The WACC disaggregated networking switching system of claim 1, wherein an IOSP of an IO device of the set of N IO devices configured to, when an IP packet is received at an IO port of the set of W IO ports:

determine an egress IO device of the set of N IO devices and an egress port number of the egress IO device based on packet header information of the received IP packet and an I-IOSP forwarding information base (FIB) of the IOSP of the IO device;

classify the packet to a flow and a traffic class based on the packet header information, the egress IO device, and the egress port number;

generate an equal cost multi-path forwarding (ECMP) hash key from a 5-Tuple of the packet header information using a hashing algorithm;

identify a micro-flow for the ECMP hash key in a fabric ECMP hash of the IOSP of the IO device;

generate a switch number of a corresponding Ethernet switch based on the micro-flow;

queue packet data of the packet and first metadata to an I-IOSPQ of an H-VOQ corresponding respectively to the egress IO device and the switch number, wherein the first metadata comprising the packet header information, the traffic class, an internal flow identification (ID) corresponding to the flow, the egress port number, an egress IO device ID corresponding to the egress IO device, and the switch number;

de-queue the packet data and the first metadata from the I-VOQs of the H-VOQ using a scheduling algorithm; and

transmit the de-queued packet data and the first metadata to the FSP of the ingress IO device, wherein the FSP configured to;

queue an Ethernet packet including the packet data, second metadata, and a first media access control (MAC) header to an I-FSPQ corresponding to the switch number of the metadata; and

de-queue the Ethernet packet from the I-FSPQ using the scheduling algorithm; and

transmit the Ethernet packet to the egress IO device via an Ethernet switch corresponding to the switch number.

3. The WACC disaggregated networking switching system of claim 1, wherein an Ethernet switch of the set of M Ethernet switches configured to, when a first Ethernet packet is received from an IO device of the set of N IO devices:

identify an egress port of the set of N switch ports of the Ethernet switch based on an egress port number of first metadata of the received first Ethernet packet;

generate a second MAC header based on the egress port number and an egress IO device of the set of N IO devices of the first metadata;

generate second metadata from the first metadata by removing the egress IO device;

queue a second Ethernet packet including the packet data, the second metadata, and the second MAC header to switch queues of the Ethernet switch;

de-queue the second Ethernet packet from the switch queues using a scheduling algorithm; and

transmit the second Ethernet packet to the egress IO device via the egress port of the Ethernet switch.

4. The WACC disaggregated networking switching system of claim 1, wherein an FSP of an IO device of the set of N IO devices configured to, when a second Ethernet packet is received from an Ethernet switch of the set of M Ethernet switches at an Ethernet port of the IO device of the set of N IO devices:

determine an ingress IO device of the set of N IO devices based on an E-FSP FIB of the FSP and an internal flow ID of second metadata of the received second Ethernet packet;

queue the packet data and the second metadata of the second Ethernet packet to an E-FSPQ of an H-VIQ corresponding respectively to the ingress IO device and an egress IO port of second metadata of the received second Ethernet packet;

de-queue the packet data and the second metadata from the E-VIQs of the H-VIQ using a scheduling algorithm; and

transmit the de-queued packet data and the second metadata to an IOSP of the egress IO device, wherein the IOSP configured to;

queue a second IP packet including the packet data and packet header information of the second metadata to an E-IOSPQ of the set of W E-IOSPQs corresponding to an egress port of the second metadata; and

de-queue the second IP packet from the E-IOSPQ using the scheduling algorithm; and

transmit the second IP packet via the egress port.

5. The WACC disaggregated networking switching system of claim 1, wherein

the switch queues of each Ethernet switch comprise a set of P priority switch queues, wherein a variable k having a value ranging from 1 to P to denote the kth priority switch queue, wherein packet data in the switch queues is de-queued using a scheduling algorithm, wherein

each I-IOSPQ of the set of N I-IOSPQs of each H-VOQ of the set M H-VOQs of an IOSP of each IO device of the set of N IO devices comprising a set of P priority I-IOSPQs, wherein packet data in each I-IOSPQ is de-queued using the scheduling algorithm, wherein

the I-VOQs of each H-VOQ of the set M H-VOQs of the IOSP of each IO device of the set of N IO devices comprising a set of P priority I-VOQs, wherein packet data in the I-VOQs is de-queued using the scheduling algorithm, wherein

each E-IOSPQ of the set of W E-IOSPQs of the IOSP of each IO device of the set of N IO devices comprising a set of P Priority E-IOSPQs, wherein packet data in each E-IOSPQ is de-queued using the scheduling algorithm, wherein

each I-FSPQ of the set of M I-FSPQs of the FSP of each IO device of the set of N IO devices comprising a set of P priority I-FSPQs, wherein packet data in each I-FSPQ is de-queued using the scheduling algorithm, wherein

each E-FSPQ of the set of N E-FSPQs of each H-VIQs of the set N H-VIQs of an FSP of each IO device of the set of N IO devices comprising a set of P Priority E-FSPQs, wherein packet data in each E-FSPQ is de-queued using the scheduling algorithm, wherein

the E-VIQs of each H-VIQ of the set N H-VIQs of the FSP of each IO device of the set of N IO devices comprising a set of P priority E-VIQs, wherein packet data in the E-VIQs is de-queued using the scheduling algorithm, and wherein the scheduling algorithm comprises a strict priority algorithm, a weighted fair queuing algorithm, a weighted round robin algorithm, a strict priority and weighted fair queuing algorithm, or a strict priority weighted round robin algorithm.

6. The WACC disaggregated networking switching system claim 1, wherein an IOSP of each IO device of the set of N IO devices configured to:

establish quantize congestion notification (QCN) between each I-IOSPQ of each H-VOQ of the set of M H-VOQs of the IOSP of each IO device of the set of N IO devices and each corresponding E-FSPQ of each H-VIQ of the set of N H-VIQs of an FSP of each IO device of the set of N IO devices, and wherein

packet data in each I-IOSPQ of the set of N I-IOSPQs of each H-VOQ of the set M H-VOQs of the IOSP of each IO device of the set of N IO devices is de-queued using a scheduling algorithm based on the established QCN.

7. The WACC disaggregated networking switching system claim 1, wherein an Ethernet switch of the set of M Ethernet switches configured to:

when the Ethernet switch of the set of M Ethernet switches has congestion within the Ethernet switch, the Ethernet switch of the set of M Ethernet switches provides point-to-point priority-based flow control (PFC) between the switch queues of the Ethernet switch of the set of M Ethernet switches and each corresponding I-FSPQ of the M I-FSPQs of the FSP of each IO device of the set of N IO devices, and wherein

the point-to-point PFC provided by the Ethernet switch of the set of M Ethernet switches back-pressures a specified quality of service (QoS) class on an Ethernet port of the corresponding IO device to cause the corresponding IO device to stop transmission of packets for a specified time on the specified QoS class on the Ethernet port of the corresponding IO device that is being back pressured.

8. The WACC disaggregated networking switching system claim 1, further comprising:

a virtual switch fabric including a set of N virtual line card slots each comprises a logical aggregation of the jth switch port of the set of N switch ports of each of the set of M Ethernet switches, wherein the jth IO device of the set of N IO devices associated with only the jth virtual line card slot of the N virtual line card slots.

9. A wide area common carrier (WACC) disaggregated networking switching system comprising:

an Ethernet fabric including a set of M Ethernet switches each comprising a set of N switch ports, each Ethernet switch to establish switch queues, wherein a variable i having a value ranging from 1 to M to denote the ith Ethernet switch of the set of M Ethernet switches, wherein a variable j having a value ranging from 1 to N to denote the jth switch port of the set of N switch ports; and

a set of N input/output (IO) devices each comprising;

a set of W IO ports, wherein a variable x having a value ranging from 1 to W to denote the xth IO port of the W IO ports;

a set of M Ethernet ports, wherein the ith Ethernet port of the jth IO device is connected to the jth switch port of the ith Ethernet switch;

an IO side packet processor (IOSP) configured to;

establish a set of M hierarchical virtual output queues (H-VOQs) each comprising a set of N ingress-IOSP queues (I-IOSPQs) and I-VOQs, wherein the ith H-VOQ corresponds to the ith Ethernet port of the jth IO device, and wherein the jth I-IOSPQ of the ith H-VOQ corresponds to the jth IO device; and

establish a set of W egress-IOSP queues (E-IOSPQs), wherein the xth E-IOSPQ corresponds to the xth IO port; and

a fabric side packet processor (FSP) configured to;

establish a set of M ingress-FSP queues (I-FSPQs), wherein the ith I-FSPQ corresponds to the ith Ethernet switch; and

establish a set of N E-VIQs, wherein the jth E-VIQ corresponds to the jth IO device.

10. The WACC disaggregated networking switching system of claim 9, wherein an IOSP of an IO device of the set of N IO devices configured to, when an IP packet is received at an IO port of the set of W IO ports:

determine an egress IO device of the set of N IO devices and an egress port number of the egress IO device based on packet header information of the received IP packet and an I-IOSP forwarding information base (FIB) of the IOSP of the IO device;

classify the packet to a flow and a traffic class based on the packet header information, the egress IO device, and the egress port number;

generate an equal cost multi-path forwarding (ECMP) hash key from a 5-Tuple of the packet header information using a hashing algorithm;

identify a micro-flow for the ECMP hash key in a fabric ECMP hash of the IOSP of the IO device;

generate a switch number of a corresponding Ethernet switch based on the micro-flow;

queue packet data of the packet and first metadata to an I-IOSPQ of an H-VOQ corresponding respectively to the egress IO device and the switch number, wherein the first metadata comprising the packet header information, the traffic class, an internal flow identification (ID) corresponding to the flow, the egress port number, an egress IO device ID corresponding to the egress IO device, and the Switch number;

de-queue the packet data and the first metadata from the I-VOQs of the H-VOQ using a scheduling algorithm; and

transmit the de-queued packet data and the first metadata to the FSP of the ingress IO device, wherein the FSP configured to;

queue an Ethernet packet including the packet data, second metadata, and a first media access control (MAC) header to an I-FSPQ corresponding to the switch number of the metadata; and

de-queue the Ethernet packet from the I-FSPQ using the scheduling algorithm; and

transmit the Ethernet packet to the egress IO device via an Ethernet switch corresponding to the switch number.

11. The WACC disaggregated networking switching system of claim 9, wherein an Ethernet switch of the set of M Ethernet switches configured to, when a first Ethernet packet is received from an IO device of the set of N IO devices:

identify an egress port of the set of N switch ports of the Ethernet switch based on an egress port number of first metadata of the received first Ethernet packet;

generate a second MAC header based on the egress port number and an egress IO device of the set of N IO devices of the first metadata;

generate second metadata from the first metadata by removing the egress IO device;

queue a second Ethernet packet including the packet data, the second metadata, and the second MAC header to switch queues of the Ethernet switch;

de-queue the second Ethernet packet from the switch queues using a scheduling algorithm; and

transmit the second Ethernet packet to the egress IO device via the egress port of the Ethernet switch.

12. The WACC disaggregated networking switching system of claim 9, wherein an FSP of an IO device of the set of N IO devices configured to, when a second Ethernet packet is received from an Ethernet switch of the set of M Ethernet switches at an Ethernet port of the IO device of the set of N IO devices:

determine an ingress IO device of the set of N IO devices based on an E-FSP FIB of the FSP and an internal flow ID of second metadata of the second received Ethernet packet;

queue the packet data and the second metadata of the second received Ethernet packet to an E-VIQs corresponding respectively to an egress IO port of second metadata of the received second Ethernet packet;

de-queue the packet data and the second metadata from the E-VIQs using a scheduling algorithm; and

transmit the de-queued packet data and the second metadata to an IOSP of the egress IO device, wherein the IOSP configured to;

queue a second IP packet including the packet data and packet header information of the second metadata to an E-IOSPQ of the set of W E-IOSPQs corresponding to an egress port of the second metadata; and

de-queue the second IP packet from the E-IOSPQ using the scheduling algorithm; and

transmit the second IP packet via the egress port.

13. The WACC disaggregated networking switching system of claim 9, wherein

the switch queues of each Ethernet switch comprise a set of P priority switch queues, wherein a variable k having a value ranging from 1 to P to denote the kth priority switch queue, wherein packet data in the switch queues is de-queued using a scheduling algorithm, wherein

each I-IOSPQ of the set of N I-IOSPQs of each H-VOQ of the set M H-VOQs of an IOSP of each IO device of the set of N IO devices comprising a set of P priority I-IOSPQs, wherein packet data in each I-IOSPQ is de-queued using the scheduling algorithm, wherein

the I-VOQs of each H-VOQ of the set M H-VOQs of the IOSP of each IO device of the set of N IO devices comprising a set of P priority I-VOQs, wherein packet data in the I-VOQs is de-queued using the scheduling algorithm, wherein

each E-IOSPQ of the set of W E-IOSPQs of the IOSP of each IO device of the set of N IO devices comprising a set of P Priority E-IOSPQs, wherein packet data in each E-IOSPQ is de-queued using the scheduling algorithm, wherein

each I-FSPQ of the set of M I-FSPQs of the FSP of each IO device of the set of N IO devices comprising a set of P priority I-FSPQs, wherein packet data in each I-FSPQ is de-queued using the scheduling algorithm, wherein

Each E-VIQs of the set N E-VIQs of the FSP of each IO device of the set of N IO devices comprising a set of P priority E-VIQs, wherein packet data in the E-VIQs is de-queued using the scheduling algorithm, and wherein the scheduling algorithm comprises a strict priority algorithm, a weighted fair queuing algorithm, a weighted round robin algorithm, a strict priority and weighted fair queuing algorithm, or a strict priority weighted round robin algorithm.

14. The WACC disaggregated networking switching system claim 9, wherein an IOSP of each IO device of the set of N IO devices configured to:

establish quantize congestion notification (QCN) between each I-IOSPQ of each H-VOQ of the set of M H-VOQs of the IOSP of each IO device of the set of N IO devices and each E-VIQ of the set of N E-VIQs of an FSP of each IO device of the set of N IO devices, and wherein

packet data in each I-IOSPQ of the set of N I-IOSPQs of each H-VOQ of the set M H-VOQs of the IOSP of each IO device of the set of N IO devices is de-queued using a scheduling algorithm based on the established QCN.

15. The WACC disaggregated networking switching system claim 9, wherein an Ethernet switch of the set of M Ethernet switches configured to:

when the Ethernet switch of the set of M Ethernet switches has congestion within the Ethernet switch of the set of M Ethernet switches, the Ethernet switch of the set of M Ethernet switches provides point-to-point priority-based flow control (PFC) between the switch queues of the Ethernet switch of the set of M Ethernet switches and each corresponding I-FSPQ of the M I-FSPQs of the FSP of each IO device of the set of N IO devices, and wherein

the point-to-point PFC provided by the Ethernet switch of the set of M Ethernet switches back-pressures a specified quality of service (QoS) class on an Ethernet port of the corresponding IO device to cause the corresponding IO device to stop transmission of packets for a specified time on the specified QoS class on the Ethernet port of the corresponding IO device that is being back pressured.

16. The WACC disaggregated networking switching system claim 9, further comprising:

a virtual switch fabric including a set of N virtual line card slots each comprises a logical aggregation of the jth switch port of the set of N switch ports of each of the set of M Ethernet switches, wherein the jth IO device of the set of N IO devices associated with only the jth virtual line card slot of the N virtual line card slots.

17. A wide area common carrier (WACC) disaggregated networking switching system comprising:

an Ethernet fabric including a set of M Ethernet switches each comprising a set of N switch ports, each Ethernet switch to establish switch queues, wherein a variable i having a value ranging from 1 to M to denote the ith Ethernet switch of the set of M Ethernet switches, wherein a variable j having a value ranging from 1 to N to denote the jth switch port of the set of N switch ports; and

a set of N input/output (IO) devices each comprising;

a set of M Ethernet ports, wherein the ith Ethernet port of the jth IO device is connected to the jth switch port of the ith Ethernet switch;

an IO side packet processor (IOSP) configured to;

establish a set of M hierarchical virtual output queues (H-VOQs) each comprising a set of N ingress-IOSP queues (I-IOSPQs) and I-VOQs, wherein the ith H-VOQ corresponds to the ith Ethernet port of the jth IO device, and wherein the jth I-IOSPQ of the ith H-VOQ corresponds to the jth IO device; and

establish a set of W egress-IOSP queues (E-IOSPQs); and

a fabric side packet processor (FSP) configured to;

establish a set of M ingress-FSP queues (I-FSPQs), wherein the ith I-FSPQ corresponds to the ith Ethernet switch; and

establish a set of W E-VIQs.

18. The WACC disaggregated networking switching system of claim 17, wherein an IOSP of an IO device of the set of N IO devices configured to, when an IP packet is received at an IO port of the set of W IO ports:

determine an egress IO device of the set of N IO devices and an egress port number of the egress IO device based on packet header information of the received IP packet and an I-IOSP forwarding information base (FIB) of the IOSP of the IO device;

classify the packet to a flow and a traffic class based on the packet header information, the egress IO device, and the egress port number;

generate an equal cost multi-path forwarding (ECMP) hash key from a 5-Tuple of the packet header information using a hashing algorithm;

identify a micro-flow for the ECMP hash key in a fabric ECMP hash of the IOSP of the IO device;

generate a switch number of a corresponding Ethernet switch based on the micro-flow;

queue packet data of the packet and first metadata to an I-IOSPQ of an H-VOQ corresponding respectively to the egress IO device and the switch number, wherein the first metadata comprising the packet header information, the traffic class, an internal flow identification (ID) corresponding to the flow, the egress port number, an egress IO device ID corresponding to the egress IO device, and the Switch number;

de-queue the packet data and the first metadata from the I-VOQs of the H-VOQ using a scheduling algorithm; and

transmit the de-queued packet data and the first metadata to the FSP of the ingress IO device, wherein the FSP configured to;

queue an Ethernet packet including the packet data, second metadata, and a first media access control (MAC) header to an I-FSPQ corresponding to the switch number of the first metadata; and

de-queue the Ethernet packet from the I-FSPQ using the scheduling algorithm; and

transmit the Ethernet packet to the egress IO device via an Ethernet switch corresponding to the switch number.

19. The WACC disaggregated networking switching system of claim 17, wherein an FSP of an IO device of the set of N IO devices configured to, when a second Ethernet packet is received from an Ethernet switch of the set of M Ethernet switches at an Ethernet port of the IO device of the set of N IO devices:

determine an ingress IO device of the set of N IO devices based on an E-FSP FIB of the FSP and an internal flow ID of second metadata of the second received Ethernet packet;

queue the packet data and the second metadata to an E-VIQs corresponding respectively to an egress IO port of second metadata of the received second Ethernet packet;

de-queue the packet data and the second metadata from the E-VIQs using a scheduling algorithm; and

transmit the de-queued packet data and the second metadata to an IOSP of the egress IO device, wherein the IOSP configured to;

queue a second IP packet including the packet data and packet header information of the second metadata to an E-IOSPQ of the set of W E-IOSPQs corresponding to an egress port of the second metadata; and

de-queue the second IP packet from the E-IOSPQ using the scheduling algorithm; and

transmit the second IP packet via the egress port.

20. The WACC disaggregated networking switching system claim 17, wherein an Ethernet switch of the set of M Ethernet switches configured to:

when the Ethernet switch of the set of M Ethernet switches has congestion within the Ethernet switch of the set of M Ethernet switches, the Ethernet switch of the set of M Ethernet switches provides point-to-point priority-based flow control (PFC) between the switch queues of the Ethernet switch of the set of M Ethernet switches and each corresponding I-FSPQ of the set of M I-FSPQs of the FSP of each IO device of the set of N IO devices, and wherein

the point-to-point PFC provided by the Ethernet switch of the set of M Ethernet switches back-pressures a specified quality of service (QoS) class on an Ethernet port of the corresponding IO device to cause the corresponding IO device to stop transmission of packets for a specified time on the specified QoS class on the Ethernet port of the corresponding IO device that is being back pressured.