Event driven remote direct memory access snapshots

• US 8,996,741 B1
• Filed: 09/25/2013
• Issued: 03/31/2015
• Est. Priority Date: 09/25/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Alanis, Francisco J., Cardona, Omar, Torres, Pedro V., Vageline, Michael P.
• Primary Examiner(s): Mamo, Elias
• Application Number:
  US14/036,543
• Publication Number:
  US 20150089011A1
• Time in Patent Office: 552 Days
• Field of Search: 710/8
• US Class Current: 710/8
• CPC Class Codes: G06F 11/0745 in an input/output transactions management context input/output processing in general G06F13/00
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]

Abstract

Mechanisms are provided, in a data processing system, for generating a snapshot of a remote direct memory access (RDMA) resource. The mechanisms receive, from an Input/Output (IO) adapter associated with the data processing system, an error event notification and store, in response to the error event notification, a snapshot of a RDMA resource associated with the error event notification. The mechanisms tear down the RDMA resource in response to the error even notification and free memory associated with the RDMA resource in response to tearing down the RDMA resource. The snapshot stores contents of the RDMA resource.

Claims

1. A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to:

receive from an Input/Output (IO) adapter associated with the data processing system, an error event notification;

store, in response to the error event notification, a snapshot of a RDMA resource associated with the error event notification;

tear down the RDMA resource in response to the error event notification; and

free memory associated with the RDMA resource in response to tearing down the RDMA resource, wherein the snapshot stores contents of the RDMA resource, wherein;

the snapshot is stored in kernelspace in a flight recorder data structure of an IO adapter device driver corresponding to the IO adapter,

storing the snapshot comprises, performing a lookup operation in a mapping data structure of cross memory map information corresponding to the resource identifier, invoking a cross memory map (xmap) service of a kernel to perform a cross memory read of the userspace RDMA resource, and performing a write of the contents of the userspace RDMA resource into an entry in the flight recorder data structure, and

the error event notification comprises a resource identifier of a resource associated with an error event detected by the IO adapter.

2. The computer program product of claim 1, wherein the snapshot of the RDMA resource associated with the error event notification comprises contents of the RDMA resource obtained from the RDMA resource and a resource descriptor obtained from a kernelspace shadow copy of the RDMA resource.

3. The computer program product of claim 1, wherein the cross memory map information comprises a location of a shadow copy of the userspace RDMA resource, and wherein storing the snapshot comprises:

using metadata stored in the shadow copy to perform a cross memory read operation of the userspace RDMA resource and a write of contents of the userspace RDMA resource to the flight recorder data structure; and

copying the metadata stored in the shadow copy to the flight recorder data structure.

4. The computer program product of claim 1, wherein the computer readable program further causes the data processing system to:

perform a post-mortem process on the stored snapshot of the userspace RDMA resource.

5. The computer program product of claim 4, wherein the post-mortem process is one of a root cause analysis process, a user notification process that notifies a user of the error event, or an output process that outputs contents of the snapshot of the userspace RDMA resource.

6. The computer program product of claim 1, wherein the userspace RDMA resource is one of a Queue Pair associated with the IO adapter and a userspace application for performing RDMA operations of an RDMA connection between the userspace application and the IO adapter, or a Completion Queue for receiving notifications of work request completions in association with the RDMA connection between the IO adapter and the userspace application.

7. A data processing system, comprising:

a processor executing a kernel in a kernelspace and an application in a userspace; and

an Input/Output adapter coupled to the processor, wherein the processor is configured to;

receive from the Input/Output (IO) adapter an error event notification;

store, in response to the error event notification, a snapshot of a RDMA resource associated with the error event notification in a kernelspace flight recorder data structure associated with a kernelspace resident IO adapter device driver;

tear down the RDMA resource in response to the error event notification; and

free memory associated with the RDMA resource in response to tearing down the RDMA resource, wherein the snapshot stores contents of the RDMA resource, wherein;

the snapshot is stored in kernelspace in a flight recorder data structure of an IO adapter device driver corresponding to the IO adapter,

the processor is configured to store the snapshot at least by performing a lookup operation in a mapping data structure of cross memory map information corresponding to the resource identifier, invoking a cross memory map (xmap) service of a kernel to perform a cross memory read of the userspace RDMA resource, and performing a write of the contents of the userspace RDMA resource into an entry in the flight recorder data structure, and

the error event notification comprises a resource identifier of a resource associated with an error event detected by the IO adapter.

8. The data processing system of claim 7, wherein the snapshot of the RDMA resource associated with the error event notification comprises contents of the RDMA resource obtained from the RDMA resource and a resource descriptor obtained from a kernelspace shadow copy of the RDMA resource.

9. The data processing system of claim 7, wherein the cross memory map information comprises a location of a shadow copy of the userspace RDMA resource, and wherein the processor is configured to store the snapshot at least by:

using metadata stored in the shadow copy to perform a cross memory read operation of the userspace RDMA resource and a write of contents of the userspace RDMA resource to the flight recorder data structure; and

copying the metadata stored in the shadow copy to the flight recorder data structure.

10. The data processing system of claim 7, wherein the processor is further configured to:

perform a post-mortem process on the stored snapshot of the userspace RDMA resource.

11. The data processing system of claim 10, wherein the post-mortem process is one of a root cause analysis process, a user notification process that notifies a user of the error event, or an output process that outputs contents of the snapshot of the userspace RDMA resource.

12. The data processing system of claim 7, wherein the userspace RDMA resource is one of a Queue Pair associated with the IO adapter and a userspace application for performing RDMA operations of an RDMA connection between the userspace application and the IO adapter, or a Completion Queue for receiving notifications of work request completions in association with the RDMA connection between the IO adapter and the userspace application.