NVM express controller for remote access of memory and I/O over ethernet-type networks

• US 9,785,356 B2
• Filed: 09/02/2015
• Issued: 10/10/2017
• Est. Priority Date: 06/26/2013
• Status: Active Grant

• Current Assignee: Point Financial, Inc.
• Original Assignee: Cnex Labs, Inc.
• Inventors: Huang, Yiren
• Primary Examiner(s): Christensen, Scott B
• Assistant Examiner(s): DO, LAM T
• Application Number:
  US14/843,891
• Publication Number:
  US 20150378606A1
• Time in Patent Office: 769 Days
• Field of Search: None
• CPC Class Codes: G06F 13/00 Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units interface circuits for specific input/output devices G06F3/00 ; multiprogram control therefor G06F9/46; multiprocessor systems G06F15/16
• CPC Class Codes: G06F 13/387 for adaptation of different data processing systems to different peripheral devices, e.g. protocol converters for incompatible systems, open system
• CPC Class Codes: G06F 13/40 Bus structure for computer networks G06F15/163; for optical bus networks H04B10/25
• CPC Class Codes: G06F 2213/0026 PCI express
• CPC Class Codes: G06F 2213/3852 Converter between protocols
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: H04L 67/1004 Server selection for load balancing

Abstract

A method and system for enabling Non-Volatile Memory express (NVMe) for accessing remote solid state drives (SSDs) (or other types of remote non-volatile memory) over the Ethernet or other networks. An extended NVMe controller is provided for enabling CPU to access remote non-volatile memory using NVMe protocol. The extended NVMe controller is implemented on one server for communication with other servers or non-volatile memory via Ethernet switch. The NVMe protocol is used over the Ethernet or similar networks by modifying it to provide a special NVM-over-Ethernet frame.

Claims

1. An extended NVMe storage network comprising:

an Ethernet network;

a plurality of extended NVMe controllers, each respective extended NVMe controller of the plurality of extended NVMe controllers comprising;

a host interface adapted to couple the extended NVMe controller to a host processor;

a direct network interface adapted to couple the extended NVMe controller to the Ethernet network;

hardware logic configured to receive from the host processor NVMe commands directed to a remote namespace with remote non-volatile memory that is coupled to the external network, and to convert the NVMe commands to a format suitable for transmission over the Ethernet network to a remote extended NVMe controller coupled to the remote namespace, wherein the converted NVMe commands are passed to the direct network interface to be transmitted over the Ethernet network; and

a directory server comprising;

a network interface adapted to couple the directory server to the Ethernet network;

a memory adapted to store a mapping between network addresses and assigned NVMe identifiers that identify extended NVMe controllers of the plurality of extended NVMe controllers;

a processor adapted to assign an available NVMe identifier to an extended NVMe controller that requests an NVMe identifier and store the assignment as a mapping in the memory; and

wherein the hardware logic in the extended NVMe controllers is further configured to transmit requests for NVMe identifiers to the directory server via the direct network interface in the extended NVMe controllers, and the processor in the directory server is configured to receive requests for NVMe identifiers from the plurality of extended NVMe controllers via the network interface in the directory server.

2. The extended NVMe storage network of claim 1, wherein the memory is further adapted to store a mapping between assigned NVMe identifiers and indications of the type of device coupled with the associated extended NVMe controller.

3. The extended NVMe storage network of claim 1, wherein the memory is further adapted to store a mapping between assigned NVMe identifiers and indications of whether the associated extended NVMe controller is active.

4. The extended NVMe storage network of claim 1, wherein the memory is further adapted to store a mapping between assigned NVMe identifiers and identifiers indicating whether the assigned NVMe identifier was learned based on network discovery.

5. The extended NVMe storage network of claim 1, wherein the memory is further adapted to store, when the assigned NVMe identifier is associated with a local namespace for a local non-volatile memory, identifiers for extended NVMe controllers that are registered with the local namespace.

6. The extended NVMe storage network of claim 5, wherein the processor is further adapted to send a shutdown notification message to any extended NVMe controllers that are registered with the local namespace responsive to receiving a message from a remote NVMe controller for the local namespace that the local namespace is to be shut down.

7. The extended NVMe directory network of claim 1, wherein the one or more messages include at least a destination network address, a source network address, a message type indicator indicating a NMVe address resolution message type, a client network address, a globally unique identifier (GUID), a client type indicator, a client identifier, a server network address, a server GUID, and a frame data.

8. The extended NVMe directory network of claim 1, wherein the network address is a Media Access Control (MAC) address.

9. The extended NVMe directory network of claim 1, wherein the network address is a Layer 3 address.

10. The extended NVMe directory network of claim 1, wherein the stored mapping in the memory is persistent across a shutdown and startup of the extended NVMe directory server.

11. An extended Non-Volatile Memory Express (NVMe) directory server, comprising:

a network interface adapted to couple the directory server to an Ethernet network;

a memory adapted to store a mapping between network addresses and assigned NVMe identifiers that identify extended NVMe controllers; and

a processor adapted to send one or more messages via the Ethernet network to assign an available NVMe identifier to an extended NVMe controller and to store the assignment as a mapping in the memory, wherein the extended NVMe controller is configured to transmit a request for an NVMe identifier to the directory server.

12. The directory server of claim 11, wherein the memory is further adapted to store a mapping between assigned NVMe identifiers and indications of the type of device coupled with the associated extended NVMe controller.

13. The directory server of claim 11, wherein the memory is further adapted to store, when the assigned NVMe identifier is associated with a local namespace for a local non-volatile memory, identifiers for extended NVMe controllers that are registered with the local namespace.

14. The directory server of claim 13, wherein the processor is further adapted to send a shutdown notification message to any extended NVMe controllers that are registered with the local namespace responsive to receiving a message from a remote NVMe controller for the local namespace that the local namespace is to be shut down.

15. The directory server of claim 11, wherein the one or more messages include at least a destination network address, a source network address, a message type indicator indicating a NMVe address resolution message type, a client network address, a globally unique identifier (GUID), a client type indicator, a client identifier, a server network address, a server GUID, and a frame data.

16. The directory server of claim 11, wherein the network address is a Layer 3 address.

17. A computer-implemented method in an extended Non-Volatile Memory Express (NMVe) directory server, comprising:

receiving a broadcasted discovery message from an extended NVMe controller via an Ethernet network;

responsive to receiving the broadcasted discovery message, sending an offer message to the extended NVMe controller with an available NVMe identifier via the Ethernet network;

receiving a request message from the extended NVMe controller requesting the available NVMe identifier via the Ethernet network; and

storing a mapping between a network address of the extended NVMe controller and the available NVMe identifier.

18. The computer-implemented method of claim 17, further comprising:

responsive to receiving the request message, sending an acknowledgement message to the extended NVMe controller indicating acknowledgement of the request and storing a mapping of the associated network address of the extended NVMe controller and the available NVMe identifier.

19. The computer-implemented method of claim 17, further comprising:

responsive to receiving the request message, sending a negative acknowledgement message to the extended NVMe controller indicating a failure to allocate the NVMe identifier to the extended NVMe controller.

20. The computer-implemented method of claim 17, further comprising:

responsive to receiving a release message from the extended NVMe controller, removing an existing mapping between the network address of the extended NVMe controller and the NVMe identifier for the extended NVMe controller.

21. The computer-implemented method of claim 17, further comprising:

responsive to receiving an identify message from the extended NVMe controller, sending a reply message to the extended NVMe controller including a plurality of existing mappings between extended NVMe controllers in the external network and associated NVMe identifiers.

22. The computer-implemented method of claim 17, further comprising:

sending a notify message to the extended NVMe controller indicating an error.

23. The computer-implemented method of claim 22, wherein the error indicated is a missed heartbeat signal from the extended NVMe controller.

24. The computer-implemented method of claim 17, further comprising:

responsive to receiving a reserve message from the extended NVMe controller to reserve a local namespace, storing a mapping of the NVMe identifier for a remote NVMe controller for the local namespace with an indication that the extended NVMe controller is a registered extended NVMe controller for the local namespace.

25. The computer-implemented method of claim 17, further comprising:

receiving a tag message from the extended NMVe controller indicating a status of the extended NVMe controller.