NON-DISRUPTIVE FAILOVER OF RDMA CONNECTION

• US 20120166886A1
• Filed: 12/27/2010
• Published: 06/28/2012
• Est. Priority Date: 12/27/2010
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Shankar, Hari, Liu, Huadong, Li, Hua
• Granted Patent: US 8,627,136 B2
• Time in Patent Office: Days
• US Class Current: 714/43
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437

Abstract

A novel RDMA connection failover technique that minimizes disruption to upper subsystem modules (executed on a computer node), which create requests for data transfer. A new failover virtual layer performs failover of an RDMA connection in error so that the upper subsystem that created a request does not have knowledge of an error (which is recoverable in software and hardware), or of a failure on the RDMA connection due to the error. Since the upper subsystem does not have knowledge of a failure on the RDMA connection or of a performed failover of the RDMA connection, the upper subsystem continues providing requests to the failover virtual layer without interruption, thereby minimizing downtime of the data transfer activity.

Claims

1. A system for performing non-disruptive failover of an RDMA connection between a first computer node and a second computer node, the system comprising:

an upper subsystem module at the first computer node configured to create a request; and

a failover virtual layer in communication with the upper system module at the first computer node, the failover virtual layer configured to establish the RDMA connection between the first computer node and the second computer node, to detect an error on the RDMA connection and perform a failover of the RDMA connection so that the upper subsystem module does not have knowledge of the error on the RDMA connection and the upper subsystem module continues providing requests to the failover virtual layer while the RDMA connection sustained the error.

2. The system of claim 1, wherein the failover virtual layer is further configured to create at least one virtual queue structure (VQS), the VQS accessible by the upper subsystem module and not accessible by an interconnect adapter at the first computer node.

3. The system of claim 2, wherein the failover virtual layer is further configured to create at least two physical queue structures (PQS) associated with the VQS, the PQSs are not accessible by the upper subsystem module.

4. The system of claim 2, wherein the failover virtual layer is further configured to perform the following steps:

receive a completion status from the interconnect adapter at the first computer node;

post the completion status at the PQS; and

responsive to the completion status being successful, move the completion status to the VQS associated with the PQS.

5. A system for performing non-disruptive failover of an RDMA connection between a first computer node and a second computer node, the system comprising:

an upper subsystem module at the first computer node configured to create a request, the request including a source memory address of data to be read over an RDMA connection and a destination memory address to store data to; and

a failover virtual layer configured to create at least one virtual queue structure (VQS), the VQS accessible by the upper subsystem module and not accessible by an interconnect adapter at the first computer node, the failover virtual layer further configured to create at least two physical queue structures (PQS) associated with a VQS, the PQS are not accessible by the upper subsystem module.

6. The system of claim 5, wherein the interconnect adapter is configured to perform data transfer between the first computer node and the second computer node over an RDMA connection.

7. The system of claim 5, further comprising an interconnect driver configured to manage the interconnect adapter.

8. The system of claim 5, wherein the interconnect adapter is further configured not to receive memory addresses of the virtual queue structure created by the failover virtual layer and is configured to receive memory addresses of the physical queue structures

9. The system of claim 5, wherein the upper subsystem module comprises at least a file system and RAID system.

10. The system of claim 5, wherein the failover virtual module further comprises:

a virtual QS management module configured to create the VQS and the PQSs and to perform failover of the PQS in error;

a request scheduling module configured to select an active PQS to post a request, to perform load balancing of requests among active PQSs; and

a monitoring module configured to monitor the PQSs to determine if a PQS is in error.

11. The system of claim 5, wherein each the VQS and PQS at the first computer node further comprise:

a send queue configured to store RDMA read, RDMA write and send requests;

a receive queue configured to store receive requests, each having a list of memory buffers to accommodate data from a send operation at a second computer node; and

a completion queue configured to store a status of the RDMA transmission.

12. The system of claim 5, further comprising a data structure for storing information about VQS and PQSs, the data structure maintaining association between the VQS and two or more PQSs.

13. The system of claim 12, wherein the data structure further maintains information about PQSs in error.

14. A method for performing an RDMA transfer between a first and a second computer node, the method comprising:

creating at least one virtual queue structure (VQS) in a memory, the VQS for storing data transfer requests and completion status;

providing a handle of the created VQS to an upper subsystem module so that the upper subsystem module is adapted to access the VQS using the handle while an interconnect adapter at the first computer node is not adapted to access the VQS, the upper subsystem module is configured to provide data transfer requests;

creating physical queue structures (PQS) in the memory;

registering the created PQS with the interconnect adapter so that the interconnect adapter is adapted to access the PQS and the upper subsystem is adapted not to access the PQS;

associating the created VQS with the two or more created PQS;

connecting the created VQS at the first computer node with a VQS at the second computer node; and

connecting the created PQS at the first computer node with a PQS at the second computer node.

15. The method of claim 14, wherein associating the created VQS with the two or more PQS further comprises selecting a PQS having an active status and providing a reference from the VQS to the active PQSs.

16. A method for performing non-disruptive failover of an RDMA connection between a first computer node and a second computer node, the method comprising:

receiving a request from an upper subsystem module, the request may include source and destination memory addresses;

posting the request to a virtual queue structure (VQS) associated with two or more physical queue structures (PQS) and posting the request to a first PQS;

accessing the first PQS to obtain the source and destination address of the data;

sending the data over the RDMA connection between the first computer node and the second computer node;

responsive to determining that an error occurred on the RDMA connection, identifying a second PQS associated with the VQS, the second PQS not in error;

moving the request from the first PQS to the second PQS associated with the VQS;

using the second PQS to perform data transfer over the RDMA connection without notifying the upper subsystem module of the error so that the upper subsystem module continues posting requests to the VQS;

posting a completion status in the second PQS at the first computer node; and

responsive to the completion status being a successful one, moving the completion status to the VQS, thereby making the upper subsystem module aware of the successful completion.

17. A method for performing non-disruptive failover of an RDMA connection between a first computer node and a second computer node, the method comprising:

receiving a data transfer request from an upper subsystem module; and

providing a failover virtual layer at the first and the second nodes, the failover virtual layer at each node is configured to establish an RDMA connection between the first and the second computer node, to receive the data transfer request and to perform a failover of an RDMA transfer without notifying the upper subsystem module so that the upper subsystem module continues providing data transfer requests to the failover virtual layer.

18. The method of claim 17, further comprising creating at least one virtual queue structure (VQS), the VQS accessible by the upper subsystem module and not accessible by an interconnect adapter at the first computer node.

19. The method of claim 17, further comprising creating at least two physical queue structures (PQS) associated with a VQS, the PQS are not accessible by the upper subsystem module.

20. The method of claim 19, further comprising:

receiving a completion status from the interconnect adapter at the first computer node;

posting the completion status at the PQS; and

responsive to the completion status being successful, moving the completion status to the VQS associated with the PQS, thereby making the completion status available to the upper subsystem module.

21. The method of claim 19, further comprising:

receiving a completion status from the interconnect adapter at the first computer node;

posting the completion status at the PQS;

responsive to the completion status being a failure, identifying a second PQS not in error and moving the requests from the PQS in error to the second PQS without notifying the upper subsystem module.

22. The method of claim 17, wherein the data to be transferred represents write logs to be transferred from the first computer node to the second computer node to maintain high availability functionality between the nodes.

23. The method of claim 17, wherein the data to be transferred represents data blocks to be transferred from the first computer system to the second computer system.

24. A high availability (HA) cluster system for performing non-disruptive failover of an RDMA connection between a first storage node and a second storage node engaged in a transfer of write logs, the system comprising:

an upper subsystem module at the first computer node configured to create a request for transfer of the write logs, the request including a source memory address of the write logs to be transferred over an RDMA connection and a destination memory address at the second computer node; and

a failover virtual layer in communication with the upper system module at the first computer node, the failover virtual layer configured to establish the RDMA connection between the first computer node and the second computer node, obtain the write logs from the memory and store the write logs to a memory of the second storage node, detect an error on the RDMA connection and perform a failover of the RDMA connection so that the upper subsystem module does not have knowledge of the error on the RDMA connection and the upper subsystem module continues providing requests to the failover virtual layer while the RDMA connection sustained the error.

25. A computer-program product comprising a computer-readable medium having computer program code embodied thereon for performing an RDMA transfer between a first and a second computer node, the computer program code adapted to:

create at least one virtual queue structure (VQS) in a memory, the VQS for storing data transfer requests and completion status;

provide a handle of the created VQS to an upper subsystem module so that the upper subsystem module is adapted to access the VQS using the handle while an interconnect adapter at the first computer node is not adapted to access the VQS, the upper subsystem module is configured to provide data transfer requests;

create physical queue structures (PQS) in the memory;

register the created PQS with the interconnect adapter so that the interconnect adapter is adapted to access the PQS and the upper subsystem is adapted not to access the PQS;

associate the created VQS with the two or more created PQS;

connect the created VQS at the first computer node with a VQS at the second computer node; and

connect the created PQS at the first computer node with a PQS at the second computer node.