Method and apparatus for high speed packet switching using train packet queuing and providing high scalability

• US 20020145974A1
• Filed: 04/06/2001
• Published: 10/10/2002
• Est. Priority Date: 04/06/2001
• Status: Active Grant

• Current Assignee: Amkis Corporation
• Original Assignee: Erlang Technology
• Inventors: Hu, Chunhua, Saidi, Hossein, Min, Paul Seungkyu, Yan, Peter Yifey
• Granted Patent: US 7,106,738 B2
• Time in Patent Office: Days
• US Class Current: 370/230
• CPC Class Codes: H04L 2012/5679 Arbitration or scheduling
• CPC Class Codes: H04L 49/1523 Parallel switch fabric planes
• CPC Class Codes: H04L 49/153 ATM switching fabrics having parallel switch planes
• CPC Class Codes: H04L 49/25 Routing or path finding in a switch fabric
• CPC Class Codes: H04L 49/258 Grouping
• CPC Class Codes: H04L 49/30 Peripheral units, e.g. input or output ports
• CPC Class Codes: H04L 49/3081 ATM peripheral units, e.g. policing, insertion or extraction

Abstract

Disclosed herein is a system architecture capable of processing fixed length and/or variable length data packets. Under the method of the invention, incoming data packets are queued together according to their corresponding switch processing parameters (SPPs), and then the commonly-queued data packets are processed through a switch fabric as a single unit. In one aspect of the invention, the commonly-queued data packets are processed by the switch fabric as a single train packet. In another aspect of the invention, the commonly-queued data packets are sliced into a set of subtrain packets. A switch fabric then processes the set of subtrain packets in parallel using a plurality of switch planes. Both aspects of the invention can be implemented with a plurality of packet formatters and deformatters linked to a switch fabric in various configurations, including multi-path and hierarchical switching systems. a multichannel switching system.

Claims

1. A method of routing data packets through a switch fabric, each data packet having a corresponding switch processing parameter (SPP), said method comprising the steps of:

a) receiving a plurality of data packets;

b) queuing a plurality of data packets into a plurality of data packet queues according to their corresponding SPPs such that data packets sharing a common corresponding SPP are commonly-queued;

c) creating train packets from commonly-queued data packets, each train packet comprising a payload and a header, wherein the train packet creating step includes encapsulating a plurality of commonly-queued data packets within the payload of at least one train packet and encapsulating the SPP corresponding to each data packet encapsulated in the train packet payload within the train packet header; and

d) routing each train packet through the switch fabric as specified by its encapsulated SPP.

2. The method of claim 1 wherein the train packet creating step includes the step of creating a train packet from at least some of the data packets in a data packet queue if the data packets queued therein have an aggregate length greater than or equal to a pre-selected maximum threshold value.

3. The method of claim 1 wherein the train packet creating step includes the step of creating train packets having varying lengths.

4. The method of claim 1 wherein the train packet creating step includes creating train packets having a fixed length.

5. The method of claim 1 wherein the train packet creating step includes the step of creating a train packet from whatever is queued in a non-empty data packet queue upon passage of a pre-selected time period after which that data packet queue became non-empty.

6. The method of claim 1 wherein each train packet payload is comprised of a plurality of payload blocks, and wherein the train packet payload encapsulating step includes encapsulating within each payload block either a plurality of data packets, a single data packet, a portion of a data packet, padding, or some combination thereof.

7. The method of claim 1 further comprising the step of:

once the train packet has been routed through the switch fabric, extracting from the train packet each data packet contained therein.

8. The method of claim 1 further comprising the steps of:

sorting the data packets into a plurality of distribution classes according to a predetermined set of distribution criteria; and

wherein the queuing step is preformed only upon the data packets sorted into less than all but at least one of said distribution classes.

9. The method of claim 1 further comprising the steps of:

sorting the data packets into a plurality of distribution classes according to a predetermined set of distribution criteria; and

wherein the queuing step is performed upon the data packets sorted into each of said distribution classes.

10. A method of routing data packets through a switch fabric having a plurality of switch planes, each data packet having a corresponding switch processing parameter (SPP), said method comprising the steps of:

a) receiving a plurality of data packets;

b) queuing a plurality of data packets into a plurality of data packet queues according to their corresponding SPPs such that data packets sharing a common corresponding SPP are queued within the same data packet queue;

c) creating train packets from commonly-queued data packets, each train packet comprising a payload and a header, wherein the train packet creating step includes encapsulating a plurality of commonly-queued data packets within the payload of at least one train packet and encapsulating the common SPP corresponding to each data packet encapsulated in the train packet payload within the train packet header;

d) for each train packet, creating a set of N subtrain packets, each subtrain packet comprising a subtrain payload and a subtrain header, wherein the step of creating a subtrain packet set includes creating the subtrain payloads by slicing each train packet payload into N slices, wherein each slice comprises a subtrain payload, and encapsulating within each subtrain header the SPP encapsulated within the train packet header of the train packet from which the set of subtrain packets was sliced;

e) routing each subtrain packet within a set of subtrain packets through a different switch plane within the switch fabric as specified by its encapsulated SPP.

11. The method of claim 10 wherein the train packet creating step includes the step of creating a train packet from at least some of the data packets in a data packet queue if the data packets queued therein have an aggregate length greater than or equal to a pre-selected maximum threshold value.

12. The method of claim 10 wherein the train packet creating step includes the step of creating train packets having varying lengths.

13. The method of claim 10 wherein the train packet creating step includes creating train packets having a fixed length.

14. The method of claim 10 wherein the train packet creating step further includes the step of creating a train packet from whatever is queued in a non-empty data packet queue upon passage of a pre-selected time period after which that data packet queue became non-empty.

15. The method of claim 10 wherein each train packet payload is comprised of a plurality of payload blocks, and wherein the train packet payload encapsulating step includes encapsulating within each payload block either a plurality of data packets, a single data packet, a portion of a data packet, padding, or some combination thereof.

16. The method of claim 10 further comprising the steps of:

once each subtrain packet within a set of subtrain packets has been routed through a switch plane, reassembling the train packet from which that set of subtrain packets was created; and

extracting each data packet from the reassembled train packet.

17. The method of claim 10 further comprising the step of sorting the data packets into a plurality of distribution classes according to a predetermined set of distribution criteria, and wherein the queuing step is preformed only upon the data packets sorted into less than all but at least one of said distribution classes.

18. The method of claim 10 further comprising the step of sorting the data packets into a plurality of distribution classes according to a predetermined set of distribution criteria, and wherein the queuing step is preformed upon the data packets sorted into each of said distribution classes.

20. The switch of claim 19 wherein each packet formatter comprises a packet queuer comprised of a plurality of waiting buffers for queuing data packets and a controller configured to (1) queue data packets in the waiting buffers according to their corresponding SPPs such that data packets sharing a common corresponding SPP are commonly-queued, and (2) for each waiting buffer, create a train packet therefrom by encapsulating in a train packet payload at least some of the data packets queued therein and encapsulating the SPP shared by the data packets encapsulated in the train packet payload in a train packet header.

21. The switch of claim 20 wherein each packet queuer further comprises a plurality of backlog buffers for queuing train packets, each backlog buffer corresponding to a waiting buffer, and wherein the controller is further configured to queue each train packet in a backlog buffer corresponding to the waiting buffer from which that train packet was created.

22. The switch of claim 20 wherein each packet queuer is further configured to create a train packet from at least some of the data packets queued in a waiting buffer once the data packets queued therein have an aggregate length greater than or equal to a pre-selected maximum threshold value.

23. The switch of claim 20 wherein each packet queuer is further configured to create a train packet from whatever is queued in a non-empty waiting buffer upon passage of a pre-selected time period after which that waiting buffer became non-empty.

24. The switch of claim 20 wherein at least one packet formatter further comprises a plurality of said packet queuers and a multiplexor for multiplexing the train packets created by said packet queuers.

25. The switch of claim 19 wherein each train packet payload comprises a plurality of payload blocks, each payload block comprising a data portion and a control header portion, wherein the data portion comprises either a plurality of data packets, a single data packet, a portion of a data packet, padding, or some combination thereof, wherein the control header comprises deformatting information, and wherein each packet deformatter is configured to extract data packets from the train packet payloads according to the deformatting information within the control headers of the payload blocks.

26. The switch of claim 19 wherein each packet formatter is configured to create train packets having variable lengths.

27. The switch of claim 19 wherein each packet formatter is configured to create train packets having a fixed length.

29. The switch of claim 28 wherein each packet formatter comprises a packet queuer for creating train packets from commonly-queued data packets and a slicing unit for creating subtrain packet sets from the train packets created by the packet queuer, each packet queuer comprising a plurality of waiting buffers for queuing data packets and a controller configured to (1) queue data packets in the waiting buffers according to their corresponding SPPs such that data packets sharing a common corresponding SPP are commonly-queued, and (2) for each waiting buffer, create a train packet therefrom by encapsulating in a train packet payload at least some of the data packets queued therein and encapsulating the SPP shared by the data packets encapsulated in the train packet payload in a train packet header, and wherein the slicing unit is configured to, for each train packet created by the packet queuer, create N subtrain payloads for the N subtrain packets in a subtrain packet set by slicing the payload of a train packet into N slices.

30. The switch of claim 29 wherein each packet queuer further comprises a plurality of backlog buffers for queuing train packets awaiting slicing by the slicing unit.

31. The switch of claim 29 wherein each packet queuer controller is configured to create a train packet from at least some of the data packets queued in a waiting buffer once the data packets queued in that waiting buffer have an aggregate length equal to or exceeding a pre-selected maximum threshold value.

32. The switch of claim 29 wherein each packet queuer controller is configured to create a train packet from whatever is queued in a non-empty waiting buffer upon passage of a pre-selected amount of time after which that waiting buffer became non-empty.

33. The switch of claim 29 wherein each packet queuer controller is configured to create train packets having a fixed length.

34. The switch of claim 29 wherein each packet queuer controller is configured to create train packets having varying lengths.

35. The switch of claim 29 wherein at least one packet formatter further comprises a plurality of said packet queuers and a multiplexor for multiplexing the train packets created by the plurality of said packet queuers upstream from the slicing unit.

36. The switch of claim 29 wherein the slicing unit of each packet formatter is further configured to, for each subtrain packet set, encapsulate slicing information within the subtrain header of each subtrain packet within a subtrain packet set, wherein the switch planes in the switch fabric are configured to be synchronous with each other, and wherein each packet deformatter comprises a packet merger configured to reassemble the train packet payloads from which each received subtrain packet set was sliced according to the slicing information encapsulated in the subtrain packet headers, and a packet restorer configured to extract each data packet from the reassembled train packet payloads.

37. The switch of claim 29 wherein the slicing unit of each packet formatter is further configured to, for each subtrain packet set, encapsulate slicing information within the subtrain header of each subtrain packet within a subtrain packet set, wherein the switch planes in the switch fabric are configured to be asynchronous with each other, and wherein each packet deformatter comprises a packet merger configured to (1) queue subtrain packets received from the switch planes, wherein the subtrain packets in the same subtrain packet set are commonly-queued, and (2) for each queued subtrain packet set, reassemble the train packet payload from which that subtrain packet set was sliced according to the slicing information encapsulated in the subtrain packet headers, and a packet restorer configured to extract each data packet from the reassembled train packet payloads.

39. The device of claim 38 wherein the packet queuer is further configured to create train packets having variable lengths.

40. The device of claim 38 wherein the packet queuer is further configured to create train packets having a fixed length.

41. The device of claim 38 wherein the packet queuer is further configured to create a train packet from the data packets queued in a waiting buffer if the data packets queued that waiting buffer have an aggregate length equal to or exceeding a pre-selected maximum threshold value.

42. The device of claim 38 wherein the packet queuer is further configured to create a train packet from whatever is queued in a non-empty waiting buffer once a pre-selected threshold amount of time has passed since that waiting buffer became non-empty.

43. The device of claim 38 wherein the packet queuer further comprises a plurality of backlog buffers for queuing train packets.

44. The device of claim 38 further comprising a plurality of said inputs, a plurality of said packet queuers, wherein each packet queuer is receives data packets from a different input, and a multiplexor connected to the plurality of packet queuers for multiplexing the train packets created from the packet queuers.

45. The device of claim 38 further comprising a slicing unit for slicing each train packet created by the controller into a set of N subtrain packets, each subtrain packet comprising a subtrain payload and a subtrain header, wherein each subtrain payload comprises a portion of the train packet payload of the train packet from which the subtrain packet set was sliced, and wherein each subtrain header includes the SPP of the train packet header of the train packet from which the subtrain packet set was sliced.

46. The device of claim 45 wherein the slicing unit is configured to encapsulate slicing information for the train packet within each subtrain header.

49. A multi-path switching system for routing data packets, said multi-path switch comprising a plurality of paths through which data packets are routed, wherein each path is associated with a distribution class and configured to route data packets corresponding to said distribution class, and wherein at least one of the paths is comprised of a switch according to claim 19.

50. The multi-path switching system of claim 49 wherein at least one of the paths is comprised of a switch according to claim 28, wherein the path that is comprised of the switch according to claim 28 is a different path than the path that is comprised of the switch according to claim 19.

51. A multi-path switching system for routing data packets, said multi-path switching system comprising a plurality of paths through which data packets are routed, wherein each path is associated with a distribution class and configured to route data packets corresponding to said distribution class, and wherein at least one of the paths is comprised of a switch according to claim 28.

19. A switch for routing data packets between a plurality of switch inputs and a plurality of switch outputs, each of said data packets having a corresponding switch processing parameter (SPP), said switch comprising:

a) a plurality of packet formatters for queuing together data packets sharing a common corresponding SPP, creating train packets from the commonly-queued data packets, and providing the train packets to a switch fabric, wherein each train packet comprises a payload and a header, wherein the payload of at least one train packet includes a plurality of commonly-queued data packets, and wherein the header of each train packet includes the common SPP corresponding to each data packet included in the payload of that train packet;

b) the switch fabric having a plurality of switch fabric inputs for receiving train packets provided by the packet formatters and a plurality of switch fabric outputs for outputting routed train packets, wherein the switch fabric is configured to route each received train packet to a switch fabric output according to the SPP included in the header of each train packet; and

c) a plurality of packet deformatters for receiving routed train packets outputted from the switch fabric, extracting data packets from the payloads of the received routed train packets, and outputting the extracted data packets.

28. A switch for routing a plurality of data packets between a plurality of switch inputs and a plurality of switch outputs, each of said data packets having a corresponding switch processing parameter (SPP), said switch comprising:

a) a plurality of packet formatters, each of said packet formatters configured to (1) queue data packets according to their corresponding SPPs such that data packets sharing a common corresponding SPP are commonly-queued, (2) create subtrain packet sets from the commonly-queued data packets, each subtrain packet set comprising a plurality N of subtrain packets, each subtrain packet comprising a subtrain payload and a subtrain header, wherein the subtrain payloads of the subtrain packets in at least one subtrain packet set encapsulate a plurality of commonly-queued data packets in the aggregate, and wherein the subtrain header of each subtrain packet in each subtrain packet set includes the SPP shared by each data packet encapsulated in the aggregated subtrain payload of that subtrain packet set, and (3) for each subtrain packet set, output the subtrain packets included in that subtrain packet set in parallel;

b) a switch fabric for routing subtrain packet sets received from the packet formatters, said switch fabric comprising a plurality N of switch planes, each switch plane having a plurality of switch plane inputs for receiving subtrain packets from the packet formatters and a plurality of switch plane outputs for outputting subtrain packets, wherein each switch plane is configured to (1) receive a subtrain packet from each subtrain packet set, and (2) route each received subtrain packet to a switch plane output according to the SPP included in its subtrain header; and

c) a plurality of packet deformatters, each packet deformatter configured to (1) receive routed subtrain packet sets from the switch fabric, and (2) extract from the received subtrain packet sets the data packets encapsulated therein.

38. A device for formatting data packets, said device comprising:

a) an input for receiving a plurality of data packets, each of said data packets having a corresponding switch processing parameter (SPP);

b) a packet queuer connected to said input, said packet queuer comprising a plurality of waiting buffers for queuing data packets therein and a controller configured to (1) queue each data packet in an appropriate waiting buffer according to its SPP such that data packets sharing a common SPP are commonly-queued, and (2) create train packets from the commonly-queued data packets, each train packet having a payload and a header, wherein the payload of at least one train packet is comprised of a plurality of commonly-queued data packets, and wherein the header of each train packet includes the SPP correponding to each data packet within the payload of that train packet.

47. A device for deformatting train packets, each train packet comprising a payload portion and a header portion, the payload portion having a plurality of data packets sharing a common corresponding switch processing parameter (SPP) encapsulated therein, the header portion including boundary information for the data packets encapsulated in the payload portion, said device comprising:

an input for receiving train packets; and

a packet restorer connected to said input, wherein said packet restorer is configured to extract from each received train packet the data packets encapsulated therein according to the boundary information included in the header of the train packet.

48. A device for deformatting sets of N subtrain packets, each subtrain packet having a subtrain payload and a subtrain header, each subtrain payload comprising a portion of a train packet payload, wherein each subtrain packet set has an aggregate subtrain payload comprising an entire train packet payload, wherein at least one train packet payload comprises a plurality of data packets sharing a common switch processing parameter (SPP), and wherein each subtrain header includes slicing information for the subtrain packet set and boundary information for the data packets encapsulated within the train packet payload, said device comprising:

a) a plurality N of inputs for receiving subtrain packets in parallel;

b) a packet merger for reassembling the train packet payload from the aggregate subtrain payload of each set of received subtrain packets according to the slicing information included within the subtrain header of each subtrain packet in the subtrain packet set; and

c) a packet restorer connected to said packet merger, wherein said packet restorer is configured to extract from each recassembled train packet the data packets encapsulated therein according to the boundary information included in the subtrain headers.

52. A multi-path switching system for routing data packets, each data packet having a corresponding switch processing parameter (SPP), said multi-path switching system comprising:

a plurality of packet queuers for creating train packets from data packets sharing a common SPP, each train packet having a payload and a header, wherein the payload of at least one train packet comprises a plurality of data packets sharing a common SPP, and wherein header of each train packet includes the SPP corresponding to each data packet comprising the payload of that train packet;

a plurality of traffic distributors for receiving train packets from the packet queuers and distributing each received train packet to at least one of a plurality of paths according to a predetermined set of distribution criteria;

each path comprising a switch fabric for routing train packets according to their SPPs; and

a plurality of packet deformatters for receiving train packets routed by the switch fabrics and deformatting each routed train packet by extracting data packets from each train packet payload.

53. A multi-path switching system for routing data packets, each data packet having a corresponding switch processing parameter (SPP), said multi-path switching system comprising:

a plurality of packet queuers for creating train packets from data packets sharing a common SPP, each train packet having a payload and a header, wherein the payload of at least one train packet comprises a plurality of data packets sharing a common SPP, and wherein header of each train packet includes the SPP corresponding to each data packet comprising the payload of that train packet;

a plurality of traffic distributors for receiving train packets from the packet queuers and distributing each received train packet to at least one of a plurality of paths according to a predetermined set of distribution criteria;

wherein at least one of said paths comprises a slicing unit for receiving train packets from the traffic distributors and slicing each received train packet into a set of N subtrain packets, and a switch fabric having a plurality N of switch planes, each switch plane for routing a different subtrain packet within a subtrain packet set; and

wherein said switch fabric within said at least one path comprising said slicing unit is linked to a plurality of packet deformatters for receiving subtrain packets, reassembling the train packets from which the subtrain packets were sliced, and extracting data packets from the reassembled train packets.

54. A device for routing data packets, each data packet having a corresponding switch processing parameter (SPP), said device comprising:

a) means for receiving a plurality of data packets;

b) means for queuing said received data packets into a plurality of data packet queues according to their corresponding SPPs such that data packets sharing a common corresponding SPP are commonly-queued;

c) means for creating train packets from commonly-queued data packets, each train packet comprising a payload and a header, wherein the creating means is configured to (1) encapsultate a plurality of commonly-queued data packets within the payload of at least one train packet and (2) encapsulate the SPP corresponding to each data packet encapsulated in the train packet payload within the train packet header, wherein said creating means is in circuit with said queuing means; and

d) means for routing each train packet as specified by its encapsulated SPP, wherein said routing means is in circuit with said creating means.

55. A device for routing data packets, each data packet having a corresponding switch processing parameter (SPP), said device comprising:

a) means for receiving a plurality of data packets;

b) means for queuing said received data packets into a plurality of data packet queues according to their corresponding SPPs such that data packets sharing a common corresponding SPP are queued within the same data packet queue;

c) means for creating train packets from commonly-queued data packets, each train packet comprising a payload and a header, wherein the train packet creating means is configured to (1) encapsulate a plurality of commonly-queued data packets within the payload of at least one train packet and (2) encapsulate the common SPP corresponding to each data packet encapsulated in the train packet payload within the train packet header, wherein said train packet creating means is in circuit with said queuing means;

d) means for creating a set of N subtrain packets from each train packet created by the train packet creating means, each subtrain packet comprising a subtrain payload and a subtrain header, wherein said subtrain packet set creating means is configured to (1) create subtrain payloads by slicing each train packet payload into N slices, wherein each slice comprises a subtrain payload, and (2) encapsulate within each subtrain header the SPP encapsulated within the train packet header of the train packet from which the set of subtrain packets was sliced;

e) means for routing each subtrain packet within a set of subtrain packets through a different switch plane within the routing means as specified by its encapsulated SPP.

56. A switch for routing data packets between a plurality of switch inputs and a plurality of switch outputs, said switch comprising:

a) a plurality of first level SPP mappers for determining a corresponding first level SPP for each data packet and attaching said determined first level SPP thereto;

b) a plurality of first level packet formatters for queuing together data packets sharing a common corresponding first level SPP, creating first level train packets from the commonly-queued data packets, wherein each first level train packet comprises a payload and a header, wherein the payload of at least one first level train packet includes a plurality of commonly-queued data packets, and wherein the header of each first level train packet includes the common first level SPP corresponding to each data packet included in the payload of that first level train packet;

c) a plurality of second level SPP mappers for determining a corresponding second level SPP for each first level train packet and attaching said determined second level SPP thereto;

d) a plurality of second level packet formatters for queuing together first level train packets sharing a common corresponding second level SPP, creating second level train packets from the commonly-queued data packets, wherein each second level train packet comprises a payload and a header, wherein the payload of at least one second level train packet includes a plurality of commonly-queued first level train packets, and wherein the header of each second level train packet includes the common second level SPP corresponding to each first level train packet included in the payload of that second level train packet;

e) a plurality of multiplexors, each multiplexor linking a plurality of first level packet formatters to a second level packet formatter;

f) a switch fabric having a plurality of switch fabric inputs for receiving second level train packets from the second level packet formatters and a plurality of switch fabric outputs for outputting routed second level train packets, wherein the switch fabric is configured to route each received second level train packet to a switch fabric output according to the second level SPP included in the header of each second level train packet;

g) a plurality of second level packet deformatters for receiving routed second level train packets outputted from the switch fabric, extracting first level train packets from the payloads of the received routed second level train packets, and outputting the extracted first level train packets;

h) a plurality of first level packet deformatters for receiving extracted first level train packets outputted from the second level packet deformatters, extracting data packets from the payloads of the received extracted first level train packets, and outputting the extracted data packets; and

i) a plurality of demultiplexors, each demultiplexor linking a second level packet deformatter with a plurality of first level packet deformatters.

57. A switch for routing data packets between a plurality of switch inputs and a plurality of switch outputs, said switch comprising:

a) a plurality of first level SPP mappers for determining a corresponding first level SPP for each data packet and attaching said determined first level SPP thereto;

b) a plurality of first level packet formatters for queuing together data packets sharing a common corresponding first level SPP, creating first level train packets from the commonly-queued data packets, wherein each first level train packet comprises a payload and a header, wherein the payload of at least one first level train packet includes a plurality of commonly-queued data packets, and wherein the header of each first level train packet includes the common first level SPP corresponding to each data packet included in the payload of that first level train packet;

c) a plurality of second level SPP mappers for determining a corresponding second level SPP for each first level train packet and attaching said determined second level SPP thereto;

d) a plurality of second level packet formatters, each of said second level packet formatters configured to (1) queue first level train packets according to their corresponding second level SPPs such that first level train packets sharing a common corresponding second level SPP are commonly-queued, (2) create subtrain packet sets from the commonly-queued first level train packets, each subtrain packet set comprising a plurality N of subtrain packets, each subtrain packet comprising a subtrain payload and a subtrain header, wherein the subtrain payloads of the subtrain packets in at least one subtrain packet set encapsulate a plurality of commonly-queued first level train packets in the aggregate, and wherein the subtrain header of each subtrain packet in each subtrain packet set includes the second level SPP shared by each first level train packet encapsulated in the aggregated subtrain payload of that subtrain packet set, and (3) for each subtrain packet set, outputting the subtrain packets included in that subtrain packet set in parallel;

e) a plurality of multiplexors, each multiplexor linking a plurality of first level packet formatters to a second level packet formatter;

f) a switch fabric for routing subtrain packet sets received from the second level packet formatters, said switch fabric comprising a plurality N of switch planes, each switch plane having a plurality of switch plane inputs for receiving subtrain packets from the second level packet formatters and a plurality of switch plane outputs for outputting subtrain packets, wherein each switch plane is configured to (1) receive a subtrain packet from each subtrain packet set, and (2) route each received subtrain packet to a switch plane output according to the second level SPP included in its subtrain header; and

g) a plurality of second level packet deformatters, each second level packet deformatter configured to (1) receive routed subtrain packet sets from the switch fabric, (2) extract from the received subtrain packet sets the first level train packets encapsulated therein, and (3) output each extracted first level train packet;

h) a plurality of first level packet deformatters for receiving extracted first level train packets outputted from the second level packet deformatters, extracting data packets from the payloads of the received extracted first level train packets, and outputting the extracted data packets; and

i) a plurality of demultiplexors, each demultiplexor linking a second level packet deformatter with a plurality of first level packet deformatters.