System and method for providing remote direct memory access over a transport medium that does not natively support remote direct memory access operations

• US 8,621,029 B1
• Filed: 04/28/2004
• Issued: 12/31/2013
• Est. Priority Date: 04/28/2004
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Gole, Abhijeet, Kimmel, Jeffrey S., Lent, Arthur F., Grier, James R., Mitchell, David W.
• Primary Examiner(s): Zele, Krista
• Assistant Examiner(s): HENDERSON, ESTHER BENOIT
• Application Number:
  US10/833,405
• Time in Patent Office: 3,534 Days
• Field of Search: 709/212, 709/231
• US Class Current: 709/212
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A system and method provides a remote direct memory access over a transport medium that does not natively support remote direct memory access operations. An emulated VI module of a storage operating system emulates RDMA operations over such a medium, e.g., conventional Ethernet, thereby enabling storage. Storage appliances in a cluster configuration utilize the non-RDMA compatible transport medium as a cluster interconnect.

Claims

1. A storage system for use in a storage system cluster with a partner storage system, comprising:

a processor and a memory;

a network adapter of the storage system configured to be coupled to a transport medium that does not support remote direct memory access operations;

a network driver of the storage system configured to control the network adapter; and

an emulated protocol module, stored in the memory, operatively interconnected with the network driver configured to emulate a protocol supporting remote direct memory access operations, the emulated protocol module further configured to generate a protocol packet that is supported by the network adapter and the transport medium, wherein the protocol packet comprises the remote direct memory access operations, the emulated protocol module further configured to implement reliable reception of the packets over the transport medium.

2. The storage system of claim 1 wherein the transport medium comprises a SCSI transport that includes support for zero copy bulk data transfers.

3. The storage system of claim 2 wherein the SCSI transport comprises FCP.

4. The storage system of claim 2 wherein the SCSI transport comprises iSCSI.

5. The storage system of claim 1 wherein the transport medium comprises a cluster interconnect operatively connecting the storage system to the partner storage system.

6. The storage system of claim 1 wherein the network adapter comprises an Ethernet media access controller.

7. The storage system of claim 1 wherein the network driver comprises a media access layer of a storage operating system of the storage system.

8. The storage system of claim 1 wherein the protocol comprises a virtual interface protocol.

9. The storage system of claim 1 wherein the emulated protocol module implements a cluster specific modification to the protocol.

10. The storage system of claim 9 wherein the cluster specific modification comprises supporting a kill packet within the protocol.

11. The storage system of claim 1 wherein the emulated protocol module is further configured to detect a missing sequence number of a set of sequence numbers assigned to the packets, the emulated protocol module further configured to, in response to detecting the missing sequence number, respond with a NACK message and ignore packets received over the transport medium until a received packet contains the missing sequence number.

12. The storage system of claim 1 wherein the emulated protocol module is integrated with high layer modules of a storage operating system of the storage system so that emulation of the protocol is transparent to the high layer modules of the storage operating system.

13. A storage system cluster, comprising:

a processor and a memory;

a first storage system comprising a network adapter operatively connected to a transport medium that does not support remote direct memory access operations;

a second storage system operatively connected to the first storage system via the transport medium, wherein the second storage system is a partner to the first storage system; and

the first storage system comprising an emulated protocol module configured to emulate a protocol supporting remote direct memory access operations, the emulated protocol module further configured to generate a protocol packet that is supported by the network adapter and the transport medium, wherein the protocol packet comprises the remote direct memory access operations and wherein the emulated protocol module is further configured to implement reliable reception of the packets over the transport medium.

14. The storage system cluster of claim 13 wherein the transport medium comprises a SCSI transport.

15. The storage system cluster of claim 14 wherein the SCSI transport comprises iSCSI.

16. The storage system cluster of claim 13 wherein each of the first and the second storage systems further comprises a network driver operatively connected with the emulated protocol module.

17. The storage system cluster of claim 13 wherein the first and second storage systems are configured as storage appliances.

32. The storage system cluster of claim 13 wherein the emulated protocol module is further configured to detect a missing sequence number of a set of sequence numbers assigned to the packets, the emulated protocol module further configured to, in response to detecting the missing sequence number, respond with a NACK message and ignore packets received over the transport medium until a received packet contains the missing sequence number.

18. A method, comprising:

receiving an operation at a protocol emulation module of a first storage system;

emulating a protocol that supports a remote direct memory access operation at the protocol emulation module;

generating, by the protocol emulation module, a packet that is supported by a network adapter of the first storage system and a transport medium operatively connected to the network adapter that does not support remote direct memory access operations, the packet comprising the remote direct memory access operation;

sending the packet from the first storage system to a second storage system over the transport medium; and

implementing reliable reception of the packets over the transport medium.

19. The method of claim 18 wherein the transport medium comprises a SCSI transport.

20. The method of claim 19 wherein the SCSI transport comprises iSCSI.

21. The method of claim 18 wherein the packet comprises a data packet.

22. The method of claim 18 wherein the packet comprises a control packet.

33. The method of claim 18 further comprising:

detecting a missing sequence number of a set of sequence numbers assigned to the packets; and

in response to detecting the missing sequence number, responding with a NACK message and ignoring packets received over the transport medium until a received packet contains the missing sequence number.

23. A computer readable storage device containing executable program instructions executed by a processor, comprising:

program instructions that receive an operation at a protocol emulation module of a first storage system;

program instructions that emulate a protocol that supports a remote direct memory access operation at the protocol emulation module and further configured to implement reliable reception of the packets over the transport medium;

program instructions that generate a packet that is supported by a network adapter of the first storage system and a transport medium operatively connected to the network adapter that does not support remote direct memory access operations, the packet comprising the remote direct memory access operation; and

program instructions that send the packets from the first storage system to a second storage system over the transport medium, wherein the second storage system comprises a partner storage system.

24. An apparatus, comprising:

a memory including instructions for

receiving an operation at a protocol emulation module of a first storage system;

emulating a protocol that supports a remote direct memory access operation at the protocol emulation module;

generating, by the protocol emulation module, a packet that is supported by a network adapter of the first storage system and a transport medium operatively connected to the network adapter that does not support remote direct memory access operations, the packet comprising the remote direct memory access operation;

sending the packets from the first storage system to a second storage system over the transport medium; and

implementing reliable reception of the packets over the transport medium.

25. The apparatus of claim 24 wherein the transport medium comprises a SCSI transport.

26. The apparatus of claim 25 wherein the SCSI transport comprises iSCSI.

34. The apparatus of claim 24 wherein the means for implementing reliable reception comprises:

means for detecting a missing sequence number of a set of sequence numbers assigned to the packets; and

means for responding, in response to detecting the missing sequence number, with a NACK message and means for ignoring packets received over the transport medium until a received packet contains the missing sequence number.

27. A method, comprising:

emulating a protocol that supports a remote direct memory access operation over a transport medium operatively connected to the first storage system that does not support remote direct memory access operations, the protocol emulated by a module configured to execute on a storage operating system of a first storage system;

generating by the module the remote direct memory access operation within packets that are supported by a network adapter of the first storage system and the transport medium;

connecting the first storage system to a second storage system using the transport medium;

receiving the packets from the first storage system by the second storage system using the transport medium;

performing the remote direct memory access operation within the received packets in response to a take over operation received by the first storage system;

is implementing reliable reception of the packets over the transport medium; and

generate a protocol packet that is supported by the network adapter and the transport medium.

28. The method of claim 27, further comprising providing the second storage system with a second module for emulating remote direct memory access functionality.

29. The method of claim 27, wherein the transport medium comprises an Ethernet connection.

30. The method of claim 27, wherein the transport medium comprises a SCSI transport.

31. The method of claim 30, wherein the SCSI transport comprises iSCSI.

35. The method of claim 27 wherein implementing reliable reception comprises:

detecting a missing sequence number of a set of sequence numbers assigned to the packets; and

in response to detecting the missing sequence number, responding with a NACK message and ignoring packets received over the transport medium until a received packet contains the missing sequence number.