Mechanism for disaggregated storage class memory over fabric

• US 10,277,677 B2
• Filed: 09/12/2016
• Issued: 04/30/2019
• Est. Priority Date: 09/12/2016
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Nachimuthu, Murugasamy K., Kumar, Mohan J.
• Primary Examiner(s): Rossiter, Sean D
• Application Number:
  US15/262,473
• Publication Number:
  US 20180077235A1
• Time in Patent Office: 960 Days
• Field of Search: 711147
• CPC Class Codes: G06F 12/00 Accessing, addressing or allocating within memory systems or architectures digital input from, or digital output to record carriers, e.g. to disk storage units, G06F3/06
• CPC Class Codes: G06F 13/00 Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units interface circuits for specific input/output devices G06F3/00 ; multiprogram control therefor G06F9/46; multiprocessor systems G06F15/16
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 49/00 Packet switching elements
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Mechanisms for disaggregated storage class memory over fabric and associated methods, apparatus, and systems. A rack is populated with pooled system drawers including pooled compute drawers and pooled storage class memory (SCM) drawers, also referred to as SCM nodes. Optionally, a pooled memory drawer may include a plurality of SCM nodes. Each SCM node provides access to multiple storage class memory devices. Compute nodes including one or more processors and local storage class memory devices are installed in the pooled compute drawers, and are enabled to be selectively-coupled to access remote storage class memory devices over a low-latency fabric. During a memory access from an initiator node (e.g., a compute node) to a target node including attached disaggregated memory (e.g., an SCM node), a fabric node identifier (ID) corresponding to the target node is identified, and an access request is forwarded to that target node over the low-latency fabric. The memory access request is then serviced on the target node, and corresponding data is returned to the initiator. During compute node composition, the compute nodes are configured to access disaggregated memory resources in the SCM nodes.

Claims

1. A method, comprising:

composing memory resources for a compute node in a first pooled system drawer installed in a rack, the compute node including at least one processor coupled to at least one local memory device, the memory resources for the compute node including the at least one local memory device and at least one remote memory device accessed via a disaggregated memory node in a second pooled system drawer installed in the rack;

configuring the compute node to detect a first memory access request originating from a processor in the compute node and corresponding to a memory region in a first remote memory device from among the at least one remote memory device to forward data relating to the first memory access request to the disaggregated memory node via a first fabric packet sent over a fabric coupled between the compute node and the disaggregated memory node;

receiving the first fabric packet at the disaggregated memory node, extracting the data relating to the first memory access request and performing a first memory access of the first remote memory device corresponding to the first memory access request; and

returning data corresponding to the first memory access from the disaggregated memory node to the compute node via a second fabric packet sent over the fabric.

2. The method of claim 1, wherein the at least one remote memory device comprises storage class memory (SCM), and the disaggregated memory node comprises an SCM node in which a plurality of SCM memory devices are installed.

3. The method of claim 2, wherein the remote memory device comprises a byte addressable three-dimensional crosspoint memory.

4. The method of claim 1, wherein the first memory access request includes a system physical address (SPA) of memory to be accessed and a plurality of nodes are coupled to the fabric, each node including a fabric node identifier (fabric node ID), the method further comprising:

determining, based on the SPA, a fabric node ID of the node on the fabric corresponding to the disaggregating memory node; and

generating, at the compute node, the first fabric packet to include the fabric node ID that is determined as a destination fabric node ID.

5. The method of claim 4, further comprising:

determining, based on the SPA, a device physical address (DPA) corresponding to a physical address in the first remote memory device at which the first memory access is to be performed.

6. The method of claim 5, wherein the disaggregated memory node includes a memory controller coupled to a plurality of memory devices via a plurality of channels, each channel having a channel ID and coupled to one or more memory devices, each memory device coupled to a given channel having a respective memory device ID, the method further comprising:

determining, using data contained in the first fabric packet and via logic on the disaggregated memory node, the channel ID and memory device ID corresponding to a memory device to be accessed to perform the first memory access, the memory device to be accessed comprising the first memory device; and

accessing the first memory device via a channel of the memory controller corresponding to the channel ID that is determined.

7. The method of claim 6, wherein the first fabric packet includes a source fabric node ID corresponding to the compute node, further comprising:

constructing a local SPA using the source fabric node ID and the DPA;

determining, via the local SPA, the channel ID, memory device ID, and a local DPA; and

using the channel ID, memory device ID, and local DPA to perform the first memory access.

8. The method of claim 7, further comprising:

implementing a DPA range table in memory on the disaggregated memory node, the DPA range table including a fabric node ID column containing a fabric node ID for each row, a DPA range column including a DPA range for each row, and an SPA base address column containing an SPA base address for each row; and

determining an SPA base address using the fabric node ID of the compute node and the DPA as lookup values for the DPA range table; and

constructing the local SPA as a function of the SPA base address for a given DPA range.

9. The method of claim 7, further comprising:

implementing an access permission lookup table in memory on the disaggregated memory node, access permission lookup table including information defining Read and Write access rights for each of a plurality of compute nodes, for respective SPA ranges; and

determining, using the fabric node ID of the compute node and the local SPA as inputs to the access permission lookup table, whether a memory access corresponding to the memory access request is permitted.

10. The method of claim 1, further comprising employing data compression for at least one of data contained in the first fabric packet and data contained in the second fabric packet.

11. A disaggregated memory node, comprising:

a fabric controller, configured to be communicatively coupled to a fabric;

a memory controller, having a plurality of channels;

a plurality of memory devices coupled to the memory controller via the plurality of channels, wherein each channel is coupled to at least one memory device;

wherein the disaggregated memory node either comprises a pooled memory drawer configured to be installed in a rack or is configured to be installed in a pooled memory drawer including a plurality of disaggregated memory nodes that is configured to be installed in the rack, and wherein the disaggregated memory node is configured to,

receive a first fabric packet sent over the fabric from one of a plurality of compute nodes in a pooled compute drawer in the rack containing data relating to a memory access request originating from a processor in the compute node, the data including memory address information;

determine, based on the memory address information, a memory device in the disaggregated memory node containing data corresponding to the memory access request;

access the memory device that is determined to service the memory access request; and

return data corresponding to the memory access request in a second fabric packet sent via the fabric controller and destined for the compute node.

12. The disaggregated memory node of claim 11, wherein the plurality of memory devices comprise storage class memory (SCM), and the disaggregated memory node comprises an SCM node.

13. The disaggregated memory node of claim 11, wherein each channel has a channel identifier (ID) and each memory device connected to a respective channel has a memory device ID, and wherein the disaggregated memory node is further configured to determine, using data contained in the first fabric packet, the channel ID and memory device ID corresponding to the memory device to be accessed to service the memory access request.

14. The disaggregated memory node of claim 13, wherein the first fabric packet includes a source fabric node ID corresponding to the compute node and a device physical address (DPA), and wherein the disaggregated memory node is further configured to:

construct a local system physical address (SPA) using the source fabric node ID and the DPA;

determine, via the local SPA, the channel ID, memory device ID, and a local DPA; and

access the memory device via the memory controller using the channel ID, memory device ID, and local DPA.

15. The disaggregated memory node of claim 14, further comprising:

memory containing a DPA range table including a fabric node ID column containing a fabric node ID for each row, a DPA range column including a DPA range for each row, and an SPA base address column containing an SPA base address for each row,

and wherein the disaggregated memory node is further configured to,

determine a SPA base address using the fabric node ID of the compute node and the DPA as lookup values for the DPA range table; and

construct the local SPA as a function of the SPA base address.

16. The disaggregated memory node of claim 15, further comprising:

a remote DDR-based direct memory access (RDDRx-DMA) engine, coupled to the fabric controller, wherein the RDDRx-DMA engine comprises circuitry and logic configured to construct the local SPA using the source fabric node ID, the DPA, and the DPA range table.

17. The disaggregated memory node of claim 14, further comprising:

a target address decoder (TAD), coupled to the memory controller, comprising circuitry and logic to decode the local SPA and output the channel ID, memory device ID, and local DPA and forward the channel ID, memory device ID, and local DPA to the memory controller.

18. The disaggregated memory node of claim 14, further comprising:

memory containing an access permission lookup table including information defining, for each of a plurality of SPA ranges, Read and Write access rights for each of a plurality of compute nodes configured to access memory devices on the disaggregated memory node,

wherein the disaggregated memory node is further configured to determine, using the fabric node ID of the compute node and the local SPA as inputs to the access permission lookup table, whether a memory access corresponding to the memory access request is permitted.

19. A compute node, comprising:

a processor;

a memory controller, having at least one channel and operatively coupled to the processor;

one or more local memory devices, each coupled to the memory controller via a channel; and

a fabric controller, configured to be communicatively coupled to a fabric,

wherein the compute node is configured to be installed in a pooled compute drawer in a rack including at least one disaggregated memory node, each disaggregated memory node including a plurality of remote memory devices, coupled to the fabric configured to provide access to memory stored in the plurality of remote memory devices, and wherein to compute node is further configured to,

generate a first memory access request;

determine the first memory access request is to be serviced by accessing a memory region in a first remote memory device in a first disaggregated memory node;

generate a first fabric packet containing data relating to the first memory access request and second the first fabric packet outbound from the fabric controller onto the fabric to be delivered to the first disaggregated memory node; and

receive a second fabric packet returned from the disaggregated memory node comprising a response packet containing data returned by the disaggregated memory node after servicing the first memory access request.

20. The compute node of claim 19, wherein each of the one or more local memory devices and the first remote memory device comprises storage class memory (SCM), and the first disaggregated memory node comprises an SCM node in which a plurality of SCM memory devices are installed.

21. The compute node of claim 20, wherein at least one of the one or more local memory devices comprises a byte addressable three-dimensional crosspoint memory DIMM (Dual Inline Memory Module).

22. The compute node of claim 19, wherein the first memory access request includes a system physical address (SPA) of memory to be accessed and wherein a plurality of nodes are coupled to the fabric, each node including a fabric node identifier (fabric node ID), and wherein the compute node is further configured to:

determine, based on the SPA, a fabric node ID of the node on the fabric corresponding to the first disaggregating memory node; and

generate the first fabric packet to include the fabric node ID that is determined as a destination fabric node ID.

23. The compute node of claim 22, further configured to determine, based on the SPA, a device physical address (DPA) corresponding to a physical address in the first remote memory device to be accessed to service the first memory access request.

24. The compute node of claim 19, further configured to:

generate a second memory access request;

determine the second memory access request is to be serviced by accessing a local memory device; and

service the second memory access request by accessing the local memory device via the memory controller.

25. The compute node of claim 24, wherein each of the first and second memory access requests includes a respective system physical address (SPA) of memory to be accessed to service the memory access request, and wherein the compute node is further configured to:

determine, for each of the first and second memory access requests as a function of the SPA for that memory access request, whether the memory access request is to be serviced by accessing a local memory device on the compute node or accessing a remote memory device on a disaggregated memory node.