SYSTEM AND METHOD FOR INITIALIZING SHARED MEMORIES FOR SHARING ENDPOINTS ACROSS A PLURALITY OF ROOT COMPLEXES

• US 20080147959A1
• Filed: 12/19/2006
• Published: 06/19/2008
• Est. Priority Date: 12/19/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Freimuth, Douglas M., Recio, Renato J., Thurber, Steven M., Vargas, Jacobo A., Salzberg, Claudia A.
• Granted Patent: US 8,271,604 B2
• Time in Patent Office: Days
• US Class Current: 711/100
• CPC Class Codes: G06F 12/1009 using page tables, e.g. page table structures
• CPC Class Codes: G06F 12/145 the protection being virtual, e.g. for virtual blocks or segments before a translation mechanism

Abstract

A system and method for initializing shared memories for sharing endpoints across a plurality of root complexes is provided. In one illustrative embodiment, a multi-root PCIe manager (MR-PCIM) initializes the shared memory between root complexes and endpoints by discovering the PCIe switch fabric by traversing all the links accessible through the interconnected switches of the PCIe switch fabric. As the links are traversed, the MR-PCIM compares information obtained for each of the root complexes and endpoints to determine which endpoints and root complexes reside on the same blade. A virtual PCIe tree data structure is then generated that ties the endpoints available on the PCIe switch fabric to each root complex. The MR-PCIM, or a single-root PCIe manager (SR-PCIM), may then assign each endpoint and root complex a base and limit within the PCIe memory address space the endpoint belongs to.

Claims

1. A method, in a data processing system, for initializing a shared memory for communication between a plurality root complexes of the data processing system, comprising:

generating a first virtual hierarchy of components of a communication fabric in the data processing system, the first virtual hierarchy being associated with a first root complex of the data processing system;

generating a second virtual hierarchy of components of the communication fabric in the data processing system, the second hierarchy being associated with a second root complex of the data processing system;

assigning an address range in a first shared memory associated with the first root complex, for each endpoint in the first virtual hierarchy;

assigning an address range in a second shared memory associated with the second root complex, for each endpoint in the second virtual hierarchy; and

performing communication between a first host system associated with the first root complex and a second host system associated with the second root complex utilizing the assigned address ranges in the first and second shared memories.

2. The method of claim 1, wherein generating the first virtual hierarchy and generating the second virtual hierarchy, comprises:

traversing links in the communication fabric to discover the components of the communication fabric;

storing identification data for the components of the communication fabric; and

generating the first and second virtual hierarchies based on the stored identification data.

3. The method of claim 2, wherein generating the first and second virtual hierarchies based on the stored identification data comprises:

comparing the identification data for each endpoint component with identification data for each root complex component discovered during traversing of the links in the communication fabric;

setting a co-residency field value associated with an endpoint component if the identification data for the endpoint component matches identification data for a root complex component; and

generating the first and second virtual hierarchies based on co-residency field values associated with endpoint components of the communication fabric.

4. The method of claim 1, wherein the communication fabric is a peripheral component interconnect express (PCIe) fabric and the endpoints are PCIe adapters.

5. The method of claim 1, wherein the generating and assigning operations are performed by single root peripheral component interconnect managers (SR-PCIMs) associated with the respective first and second root complexes.

6. The method of claim 5, wherein the SR-PCIMs assign, for each endpoint of a corresponding virtual hierarchy, a base address and limit within a PCI memory address space to which the endpoint belongs, and wherein the SR-PCIMs write the base address and limit to a Base Address Register (BAR) of the endpoint.

7. The method of claim 5, wherein the SR-PCIMs individually configure shared memories associated with their respective root complexes based on only the virtual hierarchy associated with their respective root complexes, and wherein the SR-PCIMs generate the virtual hierarchies based on information gathered by a multiple root peripheral component interconnect manager (MR-PCIM) that traverses all of the links of the communication fabric and identifies associations between endpoints and root complexes present in the communication fabric.

8. The method of claim 1, wherein the data processing system is a blade server and wherein the host systems are blades of the blade server.

9. The method of claim 1, wherein the communication fabric is a multiple root aware communication fabric comprising one or more multiple root aware (MRA) switches.

10. The method of claim 1, wherein at least one endpoint has an associate address range in both the first shared memory and the second shared memory such that the at least one endpoint is shared between the first root complex and the second root complex.

11. The method of claim 1, wherein performing communication between the first root complex and the second root complex utilizing the assigned address ranges in the first and second shared memories comprises utilizing an address translation and protection table data structure to map addresses associated with one of the first or second shared memories to addresses associated with an other of the first or second shared memories.

12. The method of claim 1, further comprising:

assigning an address range in the first shared memory associated with the first root complex, for each root complex in the first virtual hierarchy; and

assigning an address range in the second shared memory associated with the second root complex, for each root complex in the second virtual hierarchy, wherein communication between the first host system and the second host system is performed directly between the first root complex and the second root complex.

13. The method of claim 1, wherein communication between the first host system and second host system is performed between a root complex of one of the first host system and the second host system and an endpoint of an other of the first host system and the second host system.

14. A computer program product comprising a computer useable medium having a computer readable program, wherein the computer readable program, when executed in a data processing system, causes the data processing system to:

generate a first virtual hierarchy of components of a communication fabric in the data processing system, the first virtual hierarchy being associated with a first root complex of the data processing system;

generate a second virtual hierarchy of components of the communication fabric in the data processing system, the second hierarchy being associated with a second root complex of the data processing system;

assign an address range in a first shared memory associated with the first root complex, for each endpoint in the first virtual hierarchy;

assign an address range in a second shared memory associated with the second root complex, for each endpoint in the second virtual hierarchy; and

perform communication between a first host system associated with the first root complex and a second host system associated with the second root complex utilizing the assigned address ranges in the first and second shared memories.

15. The computer program product of claim 14, wherein the computer readable program causes the data processing system to generate the first virtual hierarchy and generating the second virtual hierarchy, by:

traversing links in the communication fabric to discover the components of the communication fabric;

storing identification data for the components of the communication fabric; and

generating the first and second virtual hierarchies based on the stored identification data.

16. The computer program product of claim 15, wherein the computer readable program causes the data processing system to generate the first and second virtual hierarchies based on the stored identification data by:

comparing the identification data for each endpoint component with identification data for each root complex component discovered during traversing of the links in the communication fabric;

setting a co-residency field value associated with an endpoint component if the identification data for the endpoint component matches identification data for a root complex component; and

generating the first and second virtual hierarchies based on co-residency field values associated with endpoint components of the communication fabric.

17. The computer program product of claim 14, wherein the communication fabric is a peripheral component interconnect express (PCIe) fabric and the endpoints are PCIe adapters.

18. The computer program product of claim 14, wherein the computer readable program causes the data processing system to perform the generate and assign operations using single root peripheral component interconnect managers (SR-PCIMs) associated with the respective first and second root complexes.

19. The computer program product of claim 18, wherein the SR-PCIMs assign, for each endpoint of a corresponding virtual hierarchy, a base address and limit within a PCI memory address space to which the endpoint belongs, and wherein the SR-PCIMs write the base address and limit to a Base Address Register (BAR) of the endpoint.

20. The computer program product of claim 18, wherein the SR-PCIMs individually configure shared memories associated with their respective root complexes based on only the virtual hierarchy associated with their respective root complexes, and wherein the SR-PCIMs generate the virtual hierarchies based on information gathered by a multiple root peripheral component interconnect manager (MR-PCIM) that traverses all of the links of the communication fabric and identifies associations between endpoints and root complexes present in the communication fabric.

21. The computer program product of claim 14, wherein the data processing system is a blade server and wherein the host systems are blades of the blade server.

22. The computer program product of claim 14, wherein the communication fabric is a multiple root aware communication fabric comprising one or more multiple root aware (MRA) switches.

23. The computer program product of claim 14, wherein at least one endpoint has an associate address range in both the first shared memory and the second shared memory such that the at least one endpoint is shared between the first root complex and the second root complex.

24. The computer program product of claim 14, wherein the computer readable program causes the data processing system to perform communication between the first root complex and the second root complex utilizing the assigned address ranges in the first and second shared memories by utilizing an address translation and protection table data structure to map addresses associated with one of the first or second shared memories to addresses associated with an other of the first or second shared memories.

25. The computer program product of claim 14, wherein the computer readable program further causes the data processing system to:

assign an address range in the first shared memory associated with the first root complex, for each root complex in the first virtual hierarchy; and

assign an address range in the second shared memory associated with the second root complex, for each root complex in the second virtual hierarchy, wherein communication between the first host system and the second host system is performed directly between the first root complex and the second root complex.

26. The computer program product of claim 14, wherein communication between the first host system and second host system is performed between a root complex of one of the first host system and the second host system and an endpoint of an other of the first host system and the second host system.

27. A data processing system comprising:

a first host system, the first host system comprising a first root complex;

a second host system, the second host system comprising a second root complex; and

a communication fabric coupling the first host system to the second host system, wherein;

the first host system generates a first virtual hierarchy of components of the communication fabric, the first virtual hierarchy being associated with the first root complex;

the second host system generates a second virtual hierarchy of components of the communication fabric, the second hierarchy being associated with the second root complex;

the first host system assigns an address range in a first shared memory associated with the first root complex, for each endpoint in the first virtual hierarchy;

the second host system assigns an address range in a second shared memory associated with the second root complex, for each endpoint in the second virtual hierarchy; and

communication is performed between the first host system and the second host system utilizing the assigned address ranges in the first and second shared memories.

28. The data processing system of claim 27, wherein the first host system generates the first virtual hierarchy and the second host system generates the second virtual hierarchy, by:

traversing links in the communication fabric to discover the components of the communication fabric;

storing identification data for the components of the communication fabric; and

generating the first and second virtual hierarchies based on the stored identification data.

29. The data processing system of claim 28, wherein the first host system generates the first virtual hierarchy and the second host system generates the second virtual hierarchy based on the stored identification data by:

comparing the identification data for each endpoint component with identification data for each root complex component discovered during traversing of the links in the communication fabric;

setting a co-residency field value associated with an endpoint component if the identification data for the endpoint component matches identification data for a root complex component; and

generating the first and second virtual hierarchies based on co-residency field values associated with endpoint components of the communication fabric.

30. The data processing system of claim 28, wherein the communication fabric is a peripheral component interconnect express (PCIe) fabric and the endpoints are PCIe adapters.

31. The data processing system of claim 28, wherein first host system and the second host system perform the generate and assign operations using single root peripheral component interconnect managers (SR-PCIMs) associated with the respective first and second root complexes.

32. The data processing system of claim 31, wherein the SR-PCIMs assign, for each endpoint of a corresponding virtual hierarchy, a base address and limit within a PCI memory address space to which the endpoint belongs, and wherein the SR-PCIMs write the base address and limit to a Base Address Register (BAR) of the endpoint.

33. The data processing system of claim 31, wherein the SR-PCIMs individually configure shared memories associated with their respective root complexes based on only the virtual hierarchy associated with their respective root complexes, and wherein the SR-PCIMs generate the virtual hierarchies based on information gathered by a multiple root peripheral component interconnect manager (MR-PCIM) that traverses all of the links of the communication fabric and identifies associations between endpoints and root complexes present in the communication fabric.

34. The data processing system of claim 27, wherein the data processing system is a blade server and wherein the host systems are blades of the blade server.

35. The data processing system of claim 27, wherein at least one endpoint has an associate address range in both the first shared memory and the second shared memory such that the at least one endpoint is shared between the first root complex and the second root complex.