System and method for establishing a peer connection using reliable RDMA primitives

• US 7,593,996 B2
• Filed: 07/18/2003
• Issued: 09/22/2009
• Est. Priority Date: 07/18/2003
• Status: Active Grant

• Current Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Original Assignee: NetApp, Inc.
• Inventors: sen Sarma, Joydeep, Gole, Abhijeet
• Primary Examiner(s): Lin; Kenny S
• Assistant Examiner(s): Hussain; Tauqir
• Application Number:
  US10/622,565
• Publication Number:
  US 20050015459A1
• Time in Patent Office: 2,258 Days
• Field of Search: 709/212, 370/214, 370/400, 370/464, 370/471, 710/38, 711/162
• US Class Current: 709/212
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A system and method for reliable peer-to-peer communication is provided. The system and method includes a cluster connection manager that acts as a unified resource for opening, closing and maintaining peer-to-peer communication channels between applications executing on storage systems comprising a storage system cluster. The cluster connection manager communicates with cluster connection manager clients to maintain and manage the communication channels. The cluster connection manager may also be operatively interconnected with a plurality of cluster interconnect devices, thereby enabling failover operation in the event that a cluster interconnect device suffers an error condition.

Claims

1. A method for initiating a peer-to-peer communication session, the method comprising:

initiating a boot process;

initializing a cluster connection manager in the booting process before a storage operating system executing on a cluster partner is fully active;

initiating, by the cluster connection manger, a first remote direct memory access (RDMA) read operation directed to a cluster partner before a storage operating system executing on the cluster partner is fully active, the RDMA read operation bypassing the operating system;

performing, in response to a successful first RDMA read operation, a first RDMA write operation to the cluster partner;

performing, in response to a successful RDMA write operation, a second RDMA read operation directed to the cluster partner; and

performing, in response to a successful second RDMA read operation, a second RDMA write operation to the cluster partner before a storage operating system executing on the cluster partner is fully active, in the booting process.

2. The method of claim 1 wherein the step of attempting a first RDMA read operation further comprises the step of issuing a RDMA read operation to the cluster partner requesting a pre-set memory address location that is associated with a status variable on the cluster partner.

3. The method of claim 1 further comprising:

exchanging a set of peer connection information;

passing a set of client information to the cluster partner;

creating a set of appropriate communication ports;

alerting the cluster partner of a ready status; and

alerting a set of clients that the cluster partner is in a ready state.

4. The method of claim 3 wherein the set of peer connection information comprises a version number.

5. The method of claim 3 wherein the step of passing a set of client information to the cluster partner further comprises:

collecting, from a set of clients, the set of client information; and

transferring the collected set of client information to the cluster partner.

6. The method of claim 5 wherein the client information comprises a number of communication ports required.

7. The method of claim 5 wherein the set of client information further comprises an amount of memory requested by a particular client.

8. The method of claim 1 wherein the cluster partner is a storage system.

9. The method of claim 1 wherein the cluster partner is an application server.

10. A method comprising:

initiating a boot process;

initializing a cluster connection manager in the boot process before a storage operating system executing on a cluster partner is fully active;

initiating, a peer-to-peer communication session, by a cluster connection manager, before a storage operating system executing on the cluster partner is fully active which bypasses an operating system on a storage system by attempting a first remote direct memory access read operation directed to a predefined hardware address and a predefined port number, the predefined hardware address and the predefined port number previously known to support a RDMA operation; and

performing, in the booting process before a storage operation system executing on the cluster partner is fully active, in response to a successful initiating, a first remote direct memory access write operation directed to the predefined hardware address and the predefined port number.

11. The method of claim 10 further comprising:

performing, in response to a successful first remote direct memory access write, a second remote direct memory access read operation directed to the predefined hardware address and the predefined port number.

12. The method of claim 10 wherein the predefined hardware address comprises a fibre channel identifier.

13. The method of claim 10 wherein the predefined port number comprises a virtual interface.

14. The method of claim 10 wherein the first remote direct memory access is delivered to a predefined memory address storing booting status information.

15. A system configured to establish reliable peer-to-peer communication among storage system of a clustered environment, the system comprising:

a booting process executed by a processor;

a peer process executing on each storage system partner having an operating system; and

a cluster connection manager executing on each storage system partner, the cluster connection manager establishing a reliable peer-to-peer connection between each peer process in the booting process before a storage operating system executing on a cluster partner is fully active be connecting to a predetermined port number using a predetermined network address, the reliable peer-to-peer connection bypassing the operating system and initiate a remote direct memory access (RDMA) read operation directed to a cluster partner.

16. The system of claim 15 wherein the reliable peer-to-peer connection is established without requiring a storage operating system executing on each storage system partner to be fully functioning.

17. The system of claim 15 wherein the peer-to-peer connection is a virtual interface connection.

18. The system of claim 15 wherein the peer process is a cluster connection client that requests services from the cluster connection manager.

19. A system configured to open initial peer-to-peer connection over a cluster interconnect, the system comprising:

a storage system having an operating system;

a booting process executed by a processor;

a cluster connection manager executing on the storage system, the cluster connection manager configured to establish a peer connection in the booting process before a storage operating system executing on a cluster partner in fully active on a predetermined port number and using a predetermined network address with in the storage system the peer-to-peer connection bypassing the operating system and initiate a remote direct memory access (RDMA) read operation directed to a cluster partner; and

a process executing on the storage system, the process configured to use the established peer connection for communication.

20. The system of claim 19 wherein the peer-to-peer connection is a virtual interface connection.

21. The system of claim 19 wherein the process executing on the storage system is a cluster connection client that requests services from the cluster connection manager.

22. The system of claim 19 wherein the process executing on the storage system communicates with a cluster partner using the established peer connection.

23. A system configured to accept the initiation of a peer-to-peer connection over a cluster interconnect, the system comprising:

a storage system having an operating system;

a booting process executed by a process;

a cluster connection manager executing on the storage system, the cluster connection manager executing on the storage system, the cluster connection manager configured to accept a peer connection on a predetermined port number and using a predetermined network address within the storage system and initiate a remote direct memory access (RDMA) read operation to a cluster partner in the booting process before a storage operating system executing fully active; and

a process executing on the storage system, the process configured to read information from the established peer connection.

24. The system of claim 23 wherein the peer-to-peer connection is a virtual interface connection.

25. The system of claim 23 wherein the process executing on the storage system is a cluster connection client that requests services from the cluster connection manager.

26. The system of claim 23 wherein the process executing on the storage system reads information from a cluster partner.

27. The system of claim 23 wherein the information comprises heartbeat signals.

28. A computer readable medium containing executable program instructions executed by a processor, comprising:

program instructions that initiate a booting process;

program instructions that initialize a cluster connection manager in the booting process before a storage operating system executing a cluster partner is fully active;

program instructions that initiate in the booting process, a first remote direct memory access (RDMA) read operation before a storage operating system executing on a cluster partner is fully active;

program instructions that perform, in response to a successful RDMA write operation, a second RDMA read operation directed to the cluster partner; and

program instructions that perform in the booting process before a storage cooperating system executing on the cluster partner is fully active, in response to a successful second RDMA read operation, a second RDMA write operation to the cluster partner.