Universal routing in PCI-Express fabrics

• US 7,752,346 B2
• Filed: 12/21/2007
• Issued: 07/06/2010
• Est. Priority Date: 12/21/2007
• Status: Active Grant

• Current Assignee: SanDisk Technologies LLC (Western Digital Corporation)
• Original Assignee: Aprius Incorporated
• Inventors: Talayco, Daniel, Ekici, Ali, Akyol, Bora
• Primary Examiner(s): Dang; Khanh
• Application Number:
  US11/963,484
• Publication Number:
  US 20090164694A1
• Time in Patent Office: 928 Days
• Field of Search: 710/9, 710/10, 710/26, 710/104, 710/305, 710/306, 710/310
• US Class Current: 710/9
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence

Abstract

A universal routing identifier (URID) is provided to extend the function space in PCI-Express fabrics. Methods and systems based on the URID are provided for configuring URID capable devices and upgrading PCI-Express bridges and switches having lookup tables with access control functionality. The lookup table entry contains URIDs of destination ports, backup ports, acceptance ports, and permitted ports for downstream and upstream filtering, routing and arbitrating of transaction packets. URID capable devices can be incrementally added to current PCI-Express bridges and switches. A configuration mechanism is added to the current PCI/PCI-Express enumeration software. The URID capabilities can be disabled to maintain system compatibility. A URID capable PCI-Express system is able to address ten of thousands single-function devices. A URID capability segment field is provided in the current PCI-Express configuration space. Each URID capable device contains the URID capability segment implemented in its own set of configuration space registers.

Claims

1. A method for routing a plurality of transaction packets in a data processing system having a plurality of devices coupled to a switched fabric, the method comprising:

assigning a routing identifier that is unique to each device;

inserting a first routing identifier in a first transaction packet that is sent from a first device, wherein the first transaction packet has a modified PCI-Express format that replaces the bus-device-function (BDF) number field in a standard PCI-Express format with a 16-bit universal routing identifier (URID) field, wherein the first routing identifier is inserted in the URID field as the 16-bit URID;

receiving the first transaction packet containing the first routing identifier by a second device;

comparing the first routing identifier against a routing database having a plurality of data entries stored in the second device; and

routing the first transaction packet through a predetermined port in the second device based on the result of the comparing.

2. The method of claim 1 wherein the standard PCI-Express format is a bus-device-function (BDF) format that includes a plurality of number fields comprising an 8-bit bus number, a 5-bit device number, and a 3-bit function number.

3. The method of claim 1 wherein the 16-bit universal routing identifier (URID) enables routing of the plurality of transaction packets to a plurality of device functions independent of distribution of the device functions to devices and buses of the data processing system.

4. The method of claim 1 wherein the data entries of the routing database comprise a plurality of URIDs for destination ports, backup ports, acceptance ports and permitted ports.

5. The method of claim 1 wherein the comparing further identifies whether the first routing identifier is the standard BDF format or the URID format.

6. The method of claim 1 wherein the assigning of the first routing identifier is performed with a Type 0 PCI configuration write in accordance with a PCI standard protocol.

7. A method for assigning a plurality of routing identifiers to a plurality of devices in a PCIe fabric, the method comprising:

a first configuring mechanism which enumerates the plurality of devices according to their order of discovery; and

a second configuring mechanism which assigns the plurality of routing identifiers to the plurality of devices using a specific programming code, each routing identifier of the plurality of routing identifiers comprising a 16-bit universal routing identifier (URID) that replaces the bus-device-function (BDF) number in a PCI-Express transaction packet, the URID enabling routing of the PCI-Express transaction packets to a plurality of device functions independent of distribution of the device functions to devices and buses of a host data processing system.

8. The method of claim 7 wherein each of the routing identifiers comprises an extended field that replaces the bus-device-function (BDF) number field in a standard PCIe format with a 16-bit universal routing identifier (URID) field.

9. The method of claim 7 wherein each of the devices comprises a PCI-Express configuration space having a plurality of configuration registers.

10. The method of claim 7 wherein the first configuring mechanism assigns the plurality of routing identifiers to the plurality of devices by sending Type 0 PCI configuration write requests in accordance with a PCI Bus-based enumeration process.

11. The method of claim 7 wherein the second configuring mechanism writes a base routing identifier to a first base configuration register of a first device, and wherein the specific programming code freezes the first base configuration register of the first device by setting a plurality of bits in a second capabilities register containing in the first device.

12. A system for routing a plurality of modified PCI-Express transaction packets to a destination device in a data processing system having a plurality of devices coupled to a switch fabric, the system comprising:

at least one processor

an agent coupled to the processor and assigning a unique routing identifier to each device;

a comparing mechanism coupled to the processor and determining if the routing identifier is contained within a routing database;

a routing mechanism coupled to the processor and forwarding the transaction packet to the destination device;

wherein the destination identifier is the unique routing identifier assigned by the agent; and

wherein the unique routing identifier conforms to a modified PCI-Express format that replaces the bus-device-function (BDF) number field in a standard PCI-Express format with a 16-bit universal routing identifier (URID) field, wherein the unique routing identifier is inserted in the URID field as the 16-bit URID enabling routing of the transaction packets to the plurality of devices independent of distribution of the plurality of devices and buses of the data processing system.

13. The system of claim 12 wherein the standard PCI-Express format is a bus-device-function (BDF) format that includes a plurality of number fields comprising an 8-bit bus number, a 5-bit device number, and a 3-bit function number.

14. The system of claim 12 wherein the comparing mechanism is based on a lookup table having a plurality of entries for destination ports, backup ports, downstream accepted ports and upstream permitted ports.

15. The system of claim 14 wherein the comparing mechanism verifies the validity of the source identifier associated with a source port.

16. The system of claim 14 wherein the comparing mechanism verifies whether the source identifier has the permission to route the transaction packet to a destination port.

17. The system of claim 14 wherein the comparing mechanism verifies the validity of the routing identifier associated with a destination port.

18. A routing apparatus for routing modified PCI-Express transaction packets among a plurality of devices in a data processing system, the routing apparatus comprising:

a plurality of ports, each port conforming to a PCI-Express physical format which is standardized by PCI-SIG;

a lookup table with a plurality of data entries;

a data processing unit coupled to the lookup table; and

wherein the modified PCI-Express transaction packets conform to a modified PCI-Express format that replaces the bus-device-function (BDF) number field in a standard PCI-Express format with a 16-bit universal routing identifier (URID) field, wherein a 16-bit routing identifier is inserted in the URID field to enable routing of the transaction packets to device functions of the plurality of devices independent of distribution of the device functions to devices and buses of the data processing system.

19. The routing apparatus of claim 18 wherein each device can transmit and receive the modified PCI-Express transaction packets.

20. The routing apparatus of claim 18 wherein each of the modified transaction packets comprises a source identifier and a destination identifier.

21. The routing apparatus of claim 20 wherein the each of the source identifier and the destination identifier is inserted in an extended identifier field.

22. The routing apparatus of claim 21 wherein the standard PCI-Express format is a bus-device-function (BDF) format that includes a plurality of number fields comprising an 8-bit bus number, a 5-bit device number, and a 3-bit function number.

23. The routing apparatus of claim 18 wherein the data processing unit processes the source identifier and the destination identifier of each transaction packet, verifies the validity of the source identifier, determines a destination port associated with the destination identifier and routes each transaction packet having the destination identifier to the destination port which is coupled to a destination device.

24. The routing apparatus of claim 23 wherein the verification of the source identifier and the determination of the destination identifier are performed using the lookup table.