Event driven remote direct memory access snapshots

• US 8,996,743 B1
• Filed: 11/13/2013
• Issued: 03/31/2015
• Est. Priority Date: 09/25/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Alanis, Francisco J., Cardona, Omar, Torres, Pedro V., Vageline, Michael P.
• Primary Examiner(s): Mamo, Elias
• Application Number:
  US14/078,781
• Publication Number:
  US 20150088829A1
• Time in Patent Office: 503 Days
• Field of Search: None
• US Class Current: 710/8
• CPC Class Codes: G06F 11/0745 in an input/output transactions management context input/output processing in general G06F13/00
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]

Abstract

Mechanisms are provided, in a data processing system, for generating a snapshot of a remote direct memory access (RDMA) resource. The mechanisms receive, from an Input/Output (IO) adapter associated with the data processing system, an error event notification and store, in response to the error event notification, a snapshot of a RDMA resource associated with the error event notification. The mechanisms tear down the RDMA resource in response to the error even notification and free memory associated with the RDMA resource in response to tearing down the RDMA resource. The snapshot stores contents of the RDMA resource.

Claims

1. A method, in a data processing system, for generating a snapshot of a remote direct memory access (RDMA) resource, the method comprising:

receiving, by the data processing system, from an Input/Output (IO) adapter associated with the data processing system, an error event notification;

storing, by the data processing system, in response to the error event notification, a snapshot of a RDMA resource associated with the error event notification;

tearing down, by the data processing system, the RDMA resource in response to the error event notification; and

freeing, by the data processing system, memory associated with the RDMA resource in response to tearing down the RDMA resource, wherein the snapshot stores contents of the RDMA resource, wherein the snapshot is stored in kernelspace in a flight recorder data structure of an IO adapter device driver corresponding to the IO adapter, and wherein storing the snapshot comprises performing invoking a cross memory map (xmap) service of a kernel to perform a cross memory read of the userspace RDMA resource and performing a write of the contents of the userspace RDMA resource into an entry in the flight recorder data structure.

2. The method of claim 1, wherein the snapshot of the RDMA resource associated with the error event notification comprises contents of the RDMA resource obtained from the RDMA resource and a resource descriptor obtained from a kernelspace shadow copy of the RDMA resource.

3. The method of claim 1, wherein the RDMA resource is a kernelspace RDMA resource.

4. The method of claim 1, wherein the error event notification comprises a resource identifier of a resource associated with an error event detected by the IO adapter, and wherein storing the snapshot comprises performing a lookup operation in an mapping data structure, of cross memory map information corresponding to the resource identifier.

5. The method of claim 4, wherein the cross memory map information comprises a location of a shadow copy of the userspace RDMA resource, and wherein storing the snapshot comprises:

using metadata stored in the shadow copy to perform a cross memory read operation of the userspace RDMA resource and a write of contents of the userspace RDMA resource to the flight recorder data structure; and

copying the metadata stored in the shadow copy to the flight recorder data structure.

6. The method of claim 1, further comprising:

performing a post-mortem process on the stored snapshot of the userspace RDMA resource.

7. The method of claim 6, wherein the post-mortem process is one of a root cause analysis process, a user notification process that notifies a user of the error event, or an output process that outputs contents of the snapshot of the userspace RDMA resource.

8. The method of claim 1, wherein the userspace RDMA resource is one of a Queue Pair associated with the IO adapter and a userspace application for performing RDMA operations of an RDMA connection between the userspace application and the IO adapter, or a Completion Queue for receiving notifications of work request completions in association with the RDMA connection between the IO adapter and the userspace application.

9. A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to:

receive, by the data processing system, from an Input/Output (IO) adapter associated with the data processing system, an error event notification;

store, by the data processing system, in response to the error event notification, a snapshot of a RDMA resource associated with the error event notification;

tear down, by the data processing system, the RDMA resource in response to the error event notification; and

free, by the data processing system, memory associated with the RDMA resource in response to tearing down the RDMA resource, wherein the snapshot stores contents of the RDMA resource, wherein the snapshot is stored in kernelspace in a flight recorder data structure of an IO adapter device driver corresponding to the IO adapter, and wherein storing the snapshot comprises invoking a cross memory map (xmap) service of a kernel to perform a cross memory read of the userspace RDMA resource and performing a write of the contents of the userspace RDMA resource into an entry in the flight recorder data structure.

10. The computer program product of claim 9, wherein the snapshot of the RDMA resource associated with the error event notification comprises contents of the RDMA resource obtained from the RDMA resource and a resource descriptor obtained from a kernelspace shadow copy of the RDMA resource.

11. The computer program product of claim 9, wherein the RDMA resource is a kernelspace RDMA resource.

12. The computer program product of claim 9, wherein the computer readable program further causes the data processing system to:

perform a post-mortem process on the stored snapshot of the userspace RDMA resource.

13. The computer program product of claim 12, wherein the post-mortem process is one of a root cause analysis process, a user notification process that notifies a user of the error event, or an output process that outputs contents of the snapshot of the userspace RDMA resource.

14. The computer program product of claim 9, wherein the userspace RDMA resource is one of a Queue Pair associated with the IO adapter and a userspace application for performing RDMA operations of an RDMA connection between the userspace application and the IO adapter, or a Completion Queue for receiving notifications of work request completions in association with the RDMA connection between the IO adapter and the userspace application.

15. A data processing system, comprising:

a processor executing a kernel in a kernelspace and an application in a userspace; and

an Input/Output adapter coupled to the processor, wherein the processor is configured to;

receive, from an Input/Output (IO) adapter associated with the data processing system, an error event notification;

store, in response to the error event notification, a snapshot of a RDMA resource associated with the error event notification;

tear down the RDMA resource in response to the error event notification; and

free memory associated with the RDMA resource in response to tearing down the RDMA resource, wherein the snapshot stores contents of the RDMA resource, wherein the snapshot is stored in kernelspace in a flight recorder data structure of an IO adapter device driver corresponding to the IO adapter, and wherein storing the snapshot comprises invoking a cross memory map (xmap) service of a kernel to perform a cross memory read of the userspace RDMA resource and performing a write of the contents of the userspace RDMA resource into an entry in the flight recorder data structure.

16. The data processing system of claim 15, wherein the snapshot of the RDMA resource associated with the error event notification comprises contents of the RDMA resource obtained from the RDMA resource and a resource descriptor obtained from a kernelspace shadow copy of the RDMA resource.

17. The data processing system of claim 15, wherein the RDMA resource is a kernelspace RDMA resource.

18. The data processing system of claim 15, wherein the computer readable program further causes the data processing system to:

perform a post-mortem process on the stored snapshot of the userspace RDMA resource.

19. The data processing system of claim 18, wherein the post-mortem process is one of a root cause analysis process, a user notification process that notifies a user of the error event, or an output process that outputs contents of the snapshot of the userspace RDMA resource.

20. The data processing system of claim 15, wherein the userspace RDMA resource is one of a Queue Pair associated with the IO adapter and a userspace application for performing RDMA operations of an RDMA connection between the userspace application and the IO adapter, or a Completion Queue for receiving notifications of work request completions in association with the RDMA connection between the IO adapter and the userspace application.