NVM express controller for remote access of memory and I/O over ethernet-type networks

• US 9,785,355 B2
• Filed: 09/02/2015
• Issued: 10/10/2017
• Est. Priority Date: 06/26/2013
• Status: Active Grant

• Current Assignee: Point Financial, Inc.
• Original Assignee: Cnex Labs, Inc.
• Inventors: Huang, Yiren
• Primary Examiner(s): Christensen, Scott B
• Assistant Examiner(s): DO, LAM T
• Application Number:
  US14/843,884
• Publication Number:
  US 20150378605A1
• Time in Patent Office: 769 Days
• Field of Search: None
• CPC Class Codes: G06F 13/00 Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units interface circuits for specific input/output devices G06F3/00 ; multiprogram control therefor G06F9/46; multiprocessor systems G06F15/16
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: H04L 67/1004 Server selection for load balancing

Abstract

A method and system for enabling Non-Volatile Memory express (NVMe) for accessing remote solid state drives (SSDs) (or other types of remote non-volatile memory) over the Ethernet or other networks. An extended NVMe controller is provided for enabling CPU to access remote non-volatile memory using NVMe protocol. The extended NVMe controller is implemented on one server for communication with other servers or non-volatile memory via Ethernet switch. The NVMe protocol is used over the Ethernet or similar networks by modifying it to provide a special NVM-over-Ethernet frame.

Claims

1. An extended Non-Volatile Memory Express (NVMe) controller device, comprising:

a host interface adapted to couple the extended NVMe controller to a host processor;

a direct network interface adapted to couple the extended NVMe controller to an external network, the direct network interface having a plurality of local ports connecting to the external network;

hardware logic configured to receive from the host processor NVMe commands directed to a remote namespace with remote non-volatile memory that is coupled to the external network, and to convert the NVMe commands to a format suitable for transmission over the external network to a remote extended NVMe controller coupled to the remote namespace, wherein the converted NVMe commands are passed to the direct network interface to be transmitted over the external network;

a local storage interface adapted to couple the extended NVMe controller to a local namespace for a local non-volatile memory via a plurality of memory channels of the local non-volatile memory;

wherein the hardware logic is further configured to assign each memory channel to one of the plurality of local ports of the direct network interface;

wherein the remote extended NVMe controller includes a remote direct network interface having a plurality of destination ports connecting to the external network, and the hardware logic is further configured to assign each memory channel to one of the plurality of destination ports; and

wherein the hardware logic is further configured to be able to reassign the plurality of memory channels to the plurality of local ports of the extended NVMe controller and to reassign the plurality of memory channels to the plurality of destination ports of the remote extended NVMe controller.

2. The extended NVMe controller device of claim 1, wherein the plurality of memory channels are assigned and reassignable to the plurality of local ports in a distribution such that an amount of traffic travelling through each port of the plurality of local ports is within a certain range of an amount of traffic travelling through each one of the other ports of the plurality of local ports.

3. The extended NVMe controller device of claim 1, wherein the plurality of local ports are adapted to transmit data to the remote extended NVMe controller.

4. The extended NVMe controller device of claim 1, wherein the plurality of local ports are adapted to receive data from the remote extended NVMe controller.

5. The extended NVMe controller device of claim 1, wherein the plurality of memory channels are dynamically reassigned to remaining ports of the plurality of local ports responsive to failure of one or more of the plurality of local ports.

6. The extended NVMe controller device of claim 1, wherein the plurality of memory channels are dynamically reassigned to remaining ports of the plurality of local ports responsive to the shutdown of one or more of the plurality of local ports.

7. The extended NVMe controller device of claim 1, wherein each of the plurality of memory channels is assigned to one of the plurality of local ports based on a lower number of bits of the physical page address field associated with that memory channel.

8. The extended NVMe controller device of claim 1, wherein each of the plurality of memory channels is assigned to one of the plurality of local ports based on a lower number of bits of the logical block address field associated with that memory channel.

9. The extended NVMe controller device of claim 1, wherein each of the plurality of memory channels is assigned to one of the plurality of local ports based on a port mask associated with the one of the plurality of local ports.

10. The extended NVMe controller device of claim 1, wherein the external network includes an Ethernet network, each of the plurality of local ports is an Ethernet port, and each of the plurality of destination ports is an Ethernet port.

11. The extended Non-Volatile Memory Express (NVMe) controller device of claim 1, wherein:

the host interface is a host PCI Express (PCIe) interface adapted to couple the extended NVMe controller to the host processor;

the external network includes an Ethernet network and the direct network interface is a direct Ethernet interface adapted to couple the extended NVMe controller to the external Ethernet network; and

the hardware logic includes a command translator configured to convert the NVMe commands to NVM-over-Ethernet (NVMoE) commands that are suitable for transmission over the external Ethernet network.

12. The extended Non-Volatile Memory Express (NVMe) controller device of claim 11, further comprising:

a submission queue manager that manages NVMe command submission queues for the host processor.

13. A computer-implemented method in an extended Non-Volatile Memory Express (NVMe) controller device for load balancing, comprising:

receiving from a host processor, NVMe commands directed to a remote namespace for a remote non-volatile memory coupled to an external network via a remote direct network interface, the extended NVMe controller also coupled to the external network via a local direct network interface;

retrieving data indicated in the NVMe commands from a plurality of memory channels of a local non-volatile memory of a local namespace;

assigning each of the memory channels to one of a plurality of local ports of the local direct network interface, wherein the memory channels are reassignable to the plurality of local ports;

assigning each of the memory channels to one of a plurality of destination ports of the remote direct network interface, wherein the memory channels are reassignable to the plurality of destination ports;

converting the received NVMe commands to a format suitable for transmission over the external network a remote extended NVMe controller coupled to the remote namespace; and

transmitting the converted NVMe commands including the data over the external network to the remote extended NVMe controller, said transmission using the local ports and destination ports assigned to the memory channels for the converted NVMe commands.

14. The method of claim 13, wherein the plurality of memory channels are assigned and reassignable to the plurality of local ports in a distribution such that an amount of traffic travelling through each port of the plurality of local ports is within a certain range of an amount of traffic travelling through each one of the other ports of the plurality of local ports.

15. The method of claim 13, wherein the plurality of local ports are adapted to transmit data to the remote extended NVMe controller.

16. The method of claim 13, wherein the plurality of local ports are adapted to receive data from the remote extended NVMe controller.

17. The method of claim 13, wherein the plurality of memory channels are dynamically reassigned to remaining ports of the plurality of local ports responsive to failure of one or more of the plurality of local ports.

18. The method of claim 13, wherein the plurality of memory channels are dynamically reassigned to remaining ports of the plurality of local ports responsive to the shutdown of one or more of the plurality of local ports.

19. The method of claim 13, wherein each of the plurality of memory channels is assigned to one of the plurality of local ports based on a lower number of bits of the physical page address field associated with that memory channel.

20. The method of claim 13, wherein each of the plurality of memory channels is assigned to one of the plurality of local ports based on a lower number of bits of the logical block address field associated with that memory channel.

21. The method of claim 13, wherein each of the plurality of memory channels is assigned to one of the plurality of local ports based on a port mask associated with the one of the plurality of local ports.

22. The method of claim 13, wherein the external network includes an Ethernet network, each of the plurality of local ports is an Ethernet port, and each of the plurality of destination ports is an Ethernet port.