NON-DISRUPTIVE FAILOVER OF RDMA CONNECTION

• US 20140136880A1
• Filed: 12/19/2013
• Published: 05/15/2014
• Est. Priority Date: 12/27/2010
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Shankar, Hari, Liu, Huadong, Li, Hua
• Granted Patent: US 9,537,710 B2
• Time in Patent Office: Days
• US Class Current: 714/4.11
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437

Abstract

A novel RDMA connection failover technique that minimizes disruption to upper subsystem modules (executed on a computer node), which create requests for data transfer. A new failover virtual layer performs failover of an RDMA connection in error so that the upper subsystem that created a request does not have knowledge of an error (which is recoverable in software and hardware), or of a failure on the RDMA connection due to the error. Since the upper subsystem does not have knowledge of a failure on the RDMA connection or of a performed failover of the RDMA connection, the upper subsystem continues providing requests to the failover virtual layer without interruption, thereby minimizing downtime of the data transfer activity.

Claims

1. A system for performing non-disruptive failover of an RDMA connection between a first computer node and a second computer node, the system comprising:

an upper subsystem module, associated with a first computer node, configured to;

create a request comprising a source memory address of data that is to be read over an RDMA connection and a destination memory address at which the data is to be stored; and

a failover virtual layer configured to;

establish the RDMA connection between the first computer node and a second computer node based upon the request; and

responsive to detecting an error on the RDMA connection;

perform a failover of the RDMA connection; and

during the error on the RDMA connection, accept one or more requests from the upper subsystem module.

2. The system of claim 1, upper subsystem module not provided with notification of the error on the RDMA connection such that the upper subsystem module is configured to be unaware of the error on the RDMA connection.

3. The system of claim 1, the failover virtual layer configured to create a virtual queue structure accessible to the upper subsystem module.

4. The system of claim 3, the failover virtual layer configured to create a physical queue structure associated with the virtual queue structure.

5. The system of claim 4, the physical queue structure not accessible to the upper subsystem module.

6. The system of claim 3, the virtual queue structure not accessible by an interconnect adapter at the first computer node.

7. The system of claim 3, the failover virtual layer configured to create at least two physical queue structures associated with the virtual queue structure.

8. The system of claim 4, the failover virtual layer configured to:

receive a completion status from an interconnect adapter at the first computer node;

post the completion status at the physical queue structure; and

responsive to the completion status being successful, move the completion status to the virtual queue structure.

9. The system of claim 1, comprising:

an interconnect adapter configured to;

facilitate data transfer between the first computer node and the second computer node over the RDMA connection.

10. The system of claim 9, the interconnect adapter configured to not receive a memory address associated with a virtual queue structure.

11. The system of claim 9, the interconnect adapter configured to receive one or more memory addresses associated with a physical queue structure.

12. The system of claim 7, the failover virtual layer configured to perform load balancing between the at least two physical queue structures.

13. The system of claim 4, the failover virtual layer configured to monitor the physical queue structure for an error.

14. The system of claim 3, the virtual queue structure comprising at least one of:

a send queue configured to store RDMA requests;

a receive queue to store a request from the second computer node;

or

a completion queue configured to store an RDMA transmission status.

15. The system of claim 4, the physical queue structure comprising at least one of:

a send queue configured to store RDMA requests;

a receive queue to store a request from the second computer node;

or

a completion queue configured to store an RDMA transmission status.

16. The system of claim 4, the failover virtual layer configured to:

populate a data structure with at least one of;

virtual queue structure information;

physical queue structure information;

or

an association between the virtual queue structure and the physical queue structure.

17. The system of claim 16, the failover virtual layer configured to populate the data structure with error information associated with the physical queue structure.

18. A method for performing non-disruptive failover of an RDMA connection between a first computer node and a second computer node, comprising:

invoking an upper subsystem module, associated with a first computer node, to;

create a request comprising a source memory address of data that is to be read over an RDMA connection and a destination memory address at which the data is to be stored; and

invoking a failover virtual layer to;

establish the RDMA connection between the first computer node and a second computer node based upon the request; and

responsive to detecting an error on the RDMA connection;

perform a failover of the RDMA connection; and

during the error on the RDMA connection, accept one or more requests from the upper subsystem module.

19. The method of claim 18, comprising:

creating a virtual queue structure accessible to the upper subsystem module; and

creating a physical queue structure associated with the virtual queue structure.

20. A computer-program product comprising a non-transitory computer-readable medium having computer program code embodied thereon for performing non-disruptive failover of an RDMA connection between a first computer node and a second computer node, the computer program code adapted to:

invoke an upper subsystem module, associated with a first computer node, to;

create a request comprising a source memory address of data that is to be read over an RDMA connection and a destination memory address at which the data is to be stored; and

invoke a failover virtual layer to;

establish the RDMA connection between the first computer node and a second computer node based upon the request; and

responsive to detecting an error on the RDMA connection;

perform a failover of the RDMA connection; and

during the error on the RDMA connection, accept one or more requests from the upper subsystem module.