Direct data move between DRAM and storage on a memory module

• US 10,248,328 B2
• Filed: 07/31/2017
• Issued: 04/02/2019
• Est. Priority Date: 11/07/2013
• Status: Active Grant

• Current Assignee: Netlist, Inc.
• Original Assignee: Netlist, Inc.
• Inventors: Lee, Hyun, Wang, Sheng
• Primary Examiner(s): Lo, Ann J
• Assistant Examiner(s): Taeuber, Michelle L
• Application Number:
  US15/665,246
• Publication Number:
  US 20180113614A1
• Time in Patent Office: 610 Days
• CPC Class Codes: G06F 12/0246 in block erasable memory, e.g. flash memory
• CPC Class Codes: G06F 12/0638 Combination of memories, e.g. ROM and RAM such as to permit replacement or supplementing of words in one module by words in another module address formation of the next microinstruction G06F9/26; masking faults in memories by using spares or by reconfiguring G11C29/70
• CPC Class Codes: G06F 12/08 in hierarchically structured memory systems, e.g. virtual memory systems
• CPC Class Codes: G06F 12/0897 with two or more cache hierarchy levels with multilevel cache hierarchies G06F12/0811
• CPC Class Codes: G06F 13/1668 Details of memory controller
• CPC Class Codes: G06F 13/4068 Electrical coupling
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0646 Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0679 Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

A computer system comprises a processor, a memory module and input/output devices. The memory module includes a circuit board, a volatile memory unit mounted on the circuit board, a non-volatile memory unit mounted on the circuit board and a control circuit mounted on the circuit board. The volatile memory unit comprises DRAM devices, and the non-volatile memory unit comprises flash memory. The processor is configured to execute an operating system (OS) and an application program and to present a memory address space to the application program. The memory address space including a memory mapped input/output (MMIO) space mapped to the I/O devices, a pseudo MMIO (PMMIO) space mapped to the non-volatile memory unit, and a DRAM space mapped to the volatile memory unit, the PMMIO space including a system main memory local storage (MMLS) area and a memory channel storage area, wherein the DRAM space is partitioned into memory pages, and the MCS space is partitioned into storage blocks.

Claims

1. A computer system, comprising:

a system memory bus providing a memory channel, the memory channel comprising a control/address (C/A) bus and a data bus;

a processor coupled to the system memory bus;

a memory module including a circuit board having electrical contacts coupled to the system memory bus, a volatile memory unit mounted on the circuit board, a non-volatile memory unit mounted on the circuit board and a control circuit mounted on the circuit board, the volatile memory unit comprising DRAM devices, and the non-volatile memory unit comprising flash memory;

input/output devices coupled to the system bus;

wherein the processor is configured to execute an operating system (OS) and an application program and to present a memory address space to the application program, the memory address space including a memory mapped input/output (MMIO) space mapped to the I/O devices, a pseudo MMIO (PMMIO) space mapped to the non-volatile memory unit, and a DRAM space mapped to the volatile memory unit, the PMMIO space including a system main memory local storage (MMLS) area and a memory channel storage (MCS) area, wherein the DRAM space is partitioned into memory pages, and the MCS area is partitioned into storage blocks;

wherein the operating system is configured to request direct data transfer for one or more storage blocks in the MCS area between the volatile memory unit and the non-volatile memory unit to store or retrieve files associated with the application program by causing a direct data transfer command to be sent to the memory module via the memory channel, the direct data transfer command including an address in the DRAM space and an address in the MCS space; and

wherein the control circuit is configured to receive the direct data transfer command from the memory channel, and to move the data for the one or more storage blocks between the volatile memory unit and the non-volatile memory unit without any of the data for the one or more storage blocks going through the memory channel;

wherein the processor is further configured to respond to a request from the application program to access a memory page not loaded in the volatile memory by causing a page-in command to be transmitted via the data bus to the memory module, the page-in command including information for the memory page and one or more address locations in the MMLS space where the requested data is to be retrieved;

wherein the control circuit is configured to receive the page-in command from the memory channel, to read the data for the memory page from the MMLS space without any of the data for the memory page going through the memory channel;

wherein the processor is further configured to cause a dummy write command to be transmitted to the volatile memory unit, the dummy write command including an address associated with the memory page;

wherein the volatile memory unit further comprises a registered control device (RCD) that is configured to receive the dummy write command from the memory channel, and wherein the DRAM devices are configured to receive the data for the memory page from the control circuit in response to control/address signals from the RCD, the control/address signals being derived from the dummy write command.

2. The computer system of claim 1, wherein the memory module further comprises a serial presence detect (SPD) device coded with a memory density bigger than a memory density of the volatile memory unit so as to allow the processor to present the PMMIO space in addition to the DRAM space and the MMIO space.

3. The computer system of claim 1, further comprising a router circuit controlled by the control circuit, wherein the direct data transfer command include read or write command transmitted over the C/A bus and a command packet transmitted over the data bus, wherein the control circuit is configured to recognize the read or write command as part of the direct data transfer command and set up the router circuit to route the command packet to the control circuit.

4. The computer system of claim 1, wherein processor is further configured to perform one or more memory read or write operations after transmitting the page-in command and before transmitting the dummy write command.

5. The computer system of claim 4, wherein the control circuit is further configured to notify the processor when the data for the memory page is ready to be written into the volatile memory unit.

6. The computer system of claim 1, wherein the DRAM space is divided into a user space accessible by the application program and a kernel space accessible by the OS but not by the application program, and wherein the address in the DRAM space is in the user space.

7. The computer system of claim 1, wherein the control circuit comprises status registers configured to store status data about an estimated wait time for the direct data transfer command, wherein the memory address space further includes a status register space, and wherein the OS is further configured to initiate a memory read operation addressed to the status registers after the direct data transfer command is sent to the memory module.

8. The computer system of claim 7, wherein the status registers include ECC bits for storing an ECC code associated with the status data.

9. The computer system of claim 7, further comprising a memory controller configured to write pre-defined status codes corresponding to possible status situations into the memory module.

10. The computer system of claim 1, wherein the memory module further includes buffer memory coupled to the control circuit via a set of internal C/A signal lines and a set of internal data lines, wherein the direct data transfer command includes a read command, and wherein the control circuit is configured to store the data for the one or more storage blocks into the buffer memory after read the data for the one or more storage blocks from the non-volatile memory unit, and wherein the control circuit is further configured to perform error correction on the data for the one or more storage blocks by reading each portion of the data for the one or more storage blocks from the buffer memory and writing the portion of the data for the one or more storage blocks back into the buffer memory.

11. A computer system, comprising:

a system memory bus providing a memory channel, the memory channel comprising a control/address (C/A) bus and a data bus;

a memory controller coupled to the system memory bus;

a processor coupled to the memory controller;

a memory module coupled to the memory controller via the system memory bus, the memory module comprising;

a volatile memory unit coupled to the memory channel, the volatile memory unit including a registered control device (RCD) configured to receive input C/A signals from the C/A bus and memory devices configured to perform memory operations in response to output C/A signals from the RCD that are derived from the input C/A signals;

a non-volatile (NV) memory unit; and

a module control device coupled to the volatile memory unit, the non-volatile memory unit, and the memory channel;

wherein the processor is configured to execute an operating system (OS) including a memory module driver configured to cause the memory controller to issue an NV access request to the memory module via the memory channel to transfer first data from the non-volatile memory unit to the volatile memory unit, and to issue a series of dummy write memory commands via the C/A bus after issuing the NV access request, the series of dummy write memory commands including memory addresses related to the NV access operation;

wherein the module control device is configured to perform an NV access operation in response to the NV access request, the NV access operation including reading the first data from the non-volatile memory unit in response to the NV access request, and providing a respective portion of the first data to the volatile memory unit in response to a respective dummy write memory command of the series of dummy write memory commands, the respective dummy write memory command being received by both the module control device and the RCD via the C/A bus;

wherein the volatile memory unit is configured to receive the respective portion of the first data in response to the respective dummy write memory command.

12. The memory module of claim 11, wherein the volatile memory unit, the non-volatile memory unit, and the module control device are mounted on a printed circuit board.

13. The memory module of claim 12, where the memory module further comprises buffer memory mounted on the printed circuit board and configured to buffer the first data, wherein the module control device is further configured to, before receiving the series of dummy write memory commands, send a first read command to the buffer memory to output a first portion of the first data.

14. The memory module of claim 13, wherein the module control device is further configured to send a second read command to the buffer memory to cause the buffer memory to output a second portion of the first data after receiving a first one of the series of dummy write memory commands, and wherein the module control device is further configured to monitor memory commands on the C/A bus while the first data is being transferred to the volatile memory unit, and to send a pause command to the buffer memory in response to receiving a normal memory command for system access to the volatile memory unit via the C/A bus.

15. The memory module of claim 13, wherein the module control device is configured to perform error correction on the first data after storing the first data into the buffer memory and before sending the read command to the buffer memory to output the at least a portion of the first data in response to the dummy write memory command, and wherein the module control device is configured to perform the error correction on the first data stored in the buffer memory by reading data from the buffer memory, performing error detection and correction on data read from the buffer memory, and storing corrected data back into the buffer memory.

16. The memory module of claim 11, wherein the NV access request includes a set of C/A signals received via the C/A bus and a command packet received via the data bus, the command packet including information about one or more locations in the non-volatile memory sub system.

17. The computer system of claim 16, wherein the memory module further comprises a router circuit controlled by the module control device, wherein the module control device is configured to recognize the set of C/A signals as part of the NV access request and to set up the router circuit to route the command packet to the module control device.

18. The computer system of claim 11, wherein the memory module further comprises a serial presence detect (SPD) device coded with a memory density bigger than a memory density of the volatile memory unit.