Providing remote, reliant and high performance PCI express device in cloud computing environments

• US 10,303,644 B2
• Filed: 01/16/2015
• Issued: 05/28/2019
• Est. Priority Date: 11/21/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Franke, Hubertus, Pasetto, Davide, Penner, Hartmut
• Primary Examiner(s): Meky, Moustafa M
• Assistant Examiner(s): Mendaye, Kidest
• Application Number:
  US14/598,381
• Publication Number:
  US 20160147709A1
• Time in Patent Office: 1,593 Days
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A system architecture, a method, and a computer program product are disclosed for attaching remote physical devices. In one embodiment, the system architecture comprises a compute server and a device server. The compute server includes a system memory, and one or more remote device drivers; and the device server includes a system memory and one or more physical devices, and each of the physical devices includes an associated device memory. The compute server and the device server are connected through an existing network fabric that provides remote direct memory access (RDMA) services. A system mapping function logically connects one or more of the physical devices on the device server to the compute server, including mapping between the system memories and the device memories and keeping the system memories and the device memories in synchronization using the RDMA.

Claims

1. A system architecture for attaching remote physical devices through an existing network fabric providing remote direct memory access (RDMA) services, the system architecture comprising:

a compute server including a compute server system memory, and running an operating system and one or more remote device drivers;

a device server including a device server system memory and one or more physical devices, each of the physical devices including an associated device memory, and wherein the compute server and the device server are connected through the existing network fabric, the device server directly accesses the system memory of the compute server using the RDMA services of the network fabric, and the compute server directly accesses the system memory of the device server using the RDMA services of the network fabric;

a remote interface (RI) component of the compute server to allocate memory space in the system memory of the compute server for each of the physical devices of the device server, and wherein the device server allocates memory space in the system memory of the device server for each of the physical devices of the device server; and

a system mapping function to logically connect one or more of the physical devices on the device server to the compute server by mapping data transfers between the compute server and the physical devices into a set of remote memory access operations, including mapping between the system memories and the device memories and keeping the system memories and the device memories in synchronization using the RDMA; and

wherein;

the system mapping function includes one or more components to shadow and to remap the device memories to main memory copies on the compute server and to keep the device memories and the main memory copies in synchronization using the RDMA, and to shadow and to remap memory buffer spaces on the compute server to copies on the device server and to keep the memory buffer spaces and the copies in synchronization using the RDMA; and

the one or more components of the system mapping function includes;

a device mapping component on the device server to shadow and remap each device memory to a main memory copy on the compute server and to keep the each device memory and the main memory copy in synchronization using the RDMA; and

a client mapping component on the compute server to shadow and remap each memory buffer space on the compute server to a copy on the device server and to keep the each memory buffer space and the copy in synchronization using the RDMA.

2. The system architecture according to claim 1, wherein the physical devices on the device server are PCIe devices.

3. The system architecture according to claim 1, wherein the system mapping function further includes an interrupt request mapping component for re-mapping interrupt vector addresses.

4. The system architecture according to claim 1, further comprising a memory management unit (MMU) based facility on the compute server for detecting and trapping writes inside a shadow MMIO space and io vectors.

5. The system architecture according to claim 1, further comprising an input output memory management unit (IOMMU) based facility on the device server for detecting and trapping writes inside shadow memory buffers and io vectors.

6. The system architecture according to claim 1, wherein:

the compute server runs one or more virtualized operating system images;

a remote interface bus driver is instantiated inside one or more of the virtualized operating system images; and

RDMA facilities are provided to each of the virtualized operating system images that runs one of the remote interface bus drivers.

7. The system architecture according to claim 1, further comprising:

one or more additional compute servers; and

wherein;

the physical devices of the remote device server provide multiple virtual functions; and

the system mapping function attaches different ones of the virtual functions of each of the physical devices to a same one or to different ones of the compute servers.

8. The system architecture according to claim 1, wherein the device server is implemented as a field programmable gate array (FPGA).

9. An article of manufacture comprising:

at least one tangible computer readable hardware device having computer readable program code logic tangibly embodied therein to attach remote physical devices of a device server to a compute server through an existing network fabric providing remote direct memory access (RDMA) services, the compute server including a compute server system memory, and running an operating system and one or more remote device drivers;

the device server including a device server system memory and one or more physical devices, each of the physical devices including an associated device memory; and

wherein the compute server and the device server are connected through the existing network fabric, the device server directly accesses the system memory of the compute server using the RDMA services of the network fabric, and the compute server directly accesses the system memory of the device server using the RDMA services of the network fabric, the computer readable program code logic, when executing, performing the following;

allocating memory space in the system memory of the compute server for each of the physical devices of the device server, and wherein the device server allocates memory space in the system memory of the device server for each of the physical devices of the device server; and

logically connecting one or more of the physical devices on the device server to the compute server, including mapping each of the device memories to the system memory of the compute server, mapping the system memory of the compute server to the system memory of the device server, and keeping the system memories and the device memories in synchronization using the RDMA; and

wherein;

the logically connecting includes;

shadowing and remapping the device memories to main memory copies on the compute server and keeping the device memories and the main memory copies in synchronization using the RDMA, and

shadowing and remapping memory buffer spaces on the compute server to copies on the device server and keeping the memory buffer spaces and the copies in synchronization using the RDMA;

the shadowing and remapping the device memories includes using a device mapping component on the device server to shadow and remap each device memory to a main memory copy on the compute server and to keep the each device memory and the main memory copy in synchronization using the RDMA; and

the shadowing and remapping memory buffer spaces on the compute server includes using a client mapping component on the compute server to shadow and remap each memory buffer space on the compute server to a copy on the device server and to keep the each memory buffer space and the copy in synchronization using the RDMA.

10. The article of manufacture according to claim 9, wherein:

the compute server runs one or more virtualized operating system images;

a remote interface bus driver is instantiated inside one or more of the virtualized operating system images; and

RDMA facilities are provided to each of the virtualized operating system images that runs one of the remote interface bus drivers.

11. The article of manufacture according to claim 9, wherein:

the physical devices of the remote device server provide multiple virtual functions; and

the logically connecting includes attaching different ones of the virtual functions of each of the physical devices to a same one or to different ones of a plurality of compute servers.