Apparatus and method for implementing a multi-level memory hierarchy over common memory channels

• US 9,317,429 B2
• Filed: 09/30/2011
• Issued: 04/19/2016
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Ramanujan, Raj K, Ziakas, Dimitrios, Zimmerman, David J, Kumar, Mohan J, Coury, Bassam N, Swaminathan, Muthukumar P
• Primary Examiner(s): Gu, Shawn X
• Application Number:
  US13/994,729
• Publication Number:
  US 20130275682A1
• Time in Patent Office: 1,663 Days
• Field of Search: 711103-105, 711/113, 711/118, 711/122, 711/129, 711/173
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0238 Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory
• CPC Class Codes: G06F 12/08 in hierarchically structured memory systems, e.g. virtual memory systems
• CPC Class Codes: G06F 12/0802 Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
• CPC Class Codes: G06F 12/0811 with multilevel cache hierarchies
• CPC Class Codes: G06F 12/0848 Partitioned cache, e.g. separate instruction and operand caches
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 12/0897 with two or more cache hierarchy levels with multilevel cache hierarchies G06F12/0811
• CPC Class Codes: G06F 13/1684 using multiple buses
• CPC Class Codes: G06F 2212/2024 Rewritable memory not requiring erasing, e.g. resistive or ferroelectric RAM
• CPC Class Codes: G06F 2212/2515 being configurable for different purposes, e.g. as cache or non-cache memory
• CPC Class Codes: G06F 2212/304 In main memory subsystem

Abstract

A system and method are described for integrating a memory and storage hierarchy including a non-volatile memory tier within a computer system. In one embodiment, PCMS memory devices are used as one tier in the hierarchy, sometimes referred to as “far memory.” Higher performance memory devices such as DRAM placed in front of the far memory and are used to mask some of the performance limitations of the far memory. These higher performance memory devices are referred to as “near memory.”

Claims

1. A computer system comprising:

a) a processor comprising a plurality of cores to execute instructions and process data and one or more processor caches to cache instructions and data according to a first cache management policy;

b) a first memory channel comprising a first set of address and data lines coupled to the processor;

c) a second memory channel comprising a second set of address and data lines coupled to the processor;

d) a first first-level memory and a second first-level memory each comprising a first set of characteristics associated therewith, the first set of characteristics including a first read access speed and a first write access speed, the first first-level memory coupled to the first memory channel and the second first-level memory coupled to the second memory channel; and

e) a first second-level memory communicatively coupled to the first memory channel, and a second second-level memory communicatively coupled to the second memory channel, the first and second second-level memories including a second set of characteristics associated therewith, the second set of characteristics including;

i) second read and write access speeds wherein at least one of the second read and write access speeds is relatively lower than either the first read access speed or first write access speed, respectively,

ii) non-volatility such that the first and second second-level memories are to maintain content if power to the first and second-level memories is removed, wherein at least a portion of the first first-level memory is configured as a first cache for instructions and/or data stored in the first second-level memory and at least a portion of the second first-level memory is configured as a second cache for instructions and/or data stored in the second second-level memory;

f) first-level memory controller circuitry to implement said at least a portion of the first first-level memory and said at least a portion of the second first-level memory as the first and second caches;

g) second-level memory interface circuitry to generate memory channel addresses to access the first second-level memory and the second second-level memory.

2. The system as in claim 1 wherein one of the first set of characteristics comprises a first power consumption level and the second set of characteristics comprises a second power consumption level which is relatively lower than the first power consumption level.

3. The system as in claim 1 wherein one of the first set of characteristics comprises a first density and the second set of characteristics comprises a second density which is relatively higher than the first density.

4. The system as in claim 1 wherein one of the second set of characteristics comprise the first and second second-level memories being directly writable so as to not require erasing of existing data prior to writing.

5. The system as in claim 1 wherein the first and second first-level memories comprise dynamic random access memories (DRAMs) and wherein the one or more processor caches comprise static random access memories (SRAMs).

6. The system as in claim 1 wherein the first and second second-level memories comprise any of:

phase change memories (PCMs);

resistive memories;

ferroelectric memories;

ferromagnetic memories;

dielectric memories.

7. The system as in claim 1 wherein the first and second second-level memories comprise vertically stacked storage cells.

8. The system as in claim 1 further comprising:

a mass storage device for persistently storing instructions and data, the mass storage device communicatively coupled to the first and second first-level memories and the first and second second-level memories through an interface.

9. The system as in claim 1 wherein the first and second first-level memories are logically subdivided into a first portion and a second portion, the first portion being allocated as system memory and the second portion being allocated as a cache for instructions and data stored in the first and second second-level memories, respectively, according to a second cache management policy.

10. The system as in claim 1 wherein the first write access speed is relatively higher than the second write access speed but the first read access speed approximates the second read access speed.

11. The system as in claim 10 wherein the first write access speed is at least an order of magnitude higher than the second write access speed.

12. The system as in claim 1 wherein the first set of characteristics includes a first read access latency and a first write access latency and the second set of characteristics includes a second read access latency and a second write access latency at least one of which is relatively higher than either the first read access latency or first write access latency, respectively.

13. The system as in claim 1 wherein the first and second second-level memories are cheaper to manufacture per unit size than the first and second first-level memories.

14. The system as in claim 9 wherein the first cache management policy operates independently from the second cache management policy.

15. The system as in claim 1 wherein the first and second memory channels comprise double data rate (DDR) memory channels.

16. The system as in claim 15 wherein the first and second first-level memories comprise first and second dual in-line memory modules (DIMMs) and the first and second second-level memories comprise third and fourth DIMMs, the first and third DIMMs coupled to individual slots on the first memory channel and the second and fourth DIMMs coupled to individual slots on the second memory channel.

17. A computer system comprising:

a) a processor comprising a plurality of cores to execute instructions and process data and one or more processor caches to cache instructions and data according to a first cache management policy;

b) a first memory channel comprising a set of address and data lines coupled to the processor;

c) a first level memory comprising a first set of characteristics associated therewith, the first set of characteristics including a first read access speed and a first write access speed, the first level memory communicatively coupled to the first memory channel; and

d) a second level memory communicatively coupled to the first memory channel, the second level memory comprising a second set of characteristics associated therewith, the second set of characteristics including;

i) second read and write access speeds wherein at least one of the second read and write access speeds is relatively lower than either the first read access speed or first write access speed, respectively,

ii) non-volatility such that the second level memory is to maintain its content if power to the second level memory is removed,

iii) random access and byte addressability such that instructions and/or data stored therein may be randomly accessed at a granularity equivalent to that used by a memory subsystem of the system;

e) first-level memory controller circuitry to implement at least a portion of the first first-level memory as a cache for the second level memory;

f) second-level memory interface circuitry to communicate with second-level memory controller circuitry.

18. The system as in claim 17 wherein the second-level memory controller circuitry is located on a same card with the first-level memory and the second-level memory.

19. The system as in claim 18 wherein the card is one of:

a DIMM card;

a riser card.

20. The system as in claim 19 wherein the first-level memory controller is also on the card.

21. The system as in claim 17 wherein one of the second set of characteristics comprises the second level memory being directly writable so as to not require erasing of existing data prior to writing.

22. The system as in claim 17 wherein the first level memory comprises a dynamic random access memory (DRAM) and wherein the one or more processor caches comprise static random access memories (SRAMs).

23. The system as in claim 22 wherein the second level memory comprises any of:

a phase change memory (PCM);

a resistive memory;

a ferroelectric memory;

a ferromagnetic memory;

a dielectric memory.

24. The system as in claim 23 wherein the second level memory comprises vertically stacked storage cells.

25. The system as in claim 17 further comprising:

a mass storage device to persistently store instructions and data, the mass storage device communicatively coupled to the first level memory and the second level memory through an interface.

26. The system as in claim 17 wherein the first level memory is logically subdivided into a first portion and a second portion, the first portion being allocated as system memory and the second portion being allocated as a cache for instructions and data stored in the second level memory according to a second cache management policy.

27. The system as in claim 17 wherein the first write access speed is relatively higher than the second write access speed but the first read access speed approximates the second read access speed.

28. The system as in claim 27 wherein the first write access speed is at least an order of magnitude higher than the second write access speed.

29. The system as in claim 17 wherein the first set of characteristics includes a first read access latency and a first write access latency and the second set of characteristics includes a second read access latency and a second write access latency at least one of which is relatively higher than either the first read access latency or first write access latency, respectively.

30. The system as in claim 17 wherein the second level memory is cheaper to manufacture per unit size than the first level memory.

31. An apparatus comprising:

a memory controller comprising;

a) a first memory channel interface to couple to a first first-level memory and a first second-level memory;

b) a second memory channel interface to couple to a second first-level memory and a second second-level memory, wherein, the first and second first-level memories comprise a first set of characteristics associated therewith, the first set of characteristics including a first read access speed and a first write access speed, and wherein the first and second second-level memories comprise a second set of characteristics associated therewith, the second set of characteristics including;

i) second read and write access speeds wherein at least one of the second read and write access speeds is relatively lower than either the first read access speed or first write access speed, respectively,

ii) non-volatility such that the second-level memories are to maintain their content if power is removed from the second-level memories,

iii) random access and byte addressability such that instructions or data stored therein may be randomly accessed at a granularity equivalent to that used by a memory subsystem of the system,

c) a first controller coupled to the first and second memory channel interfaces, the first controller to communicate to the first first-level memory through the first memory channel interface and to communicate to the second first-level memory through the second memory channel interface; and

,

d) second-level memory interface circuitry to communicate to a first second-level memory controller through the first memory channel interface and communicate to a second second-level memory controller through the second memory channel interface.

32. The apparatus of claim 31 wherein the first controller is a cache controller.

33. The apparatus of claim 31 wherein one of the first set of characteristics comprises a first power consumption level and the second set of characteristics comprises a second power consumption level which is relatively lower than the first power consumption level.

34. The apparatus of claim 31 wherein one of the first set of characteristics comprises a first density and the second set of characteristics comprises a second density which is relatively higher than the first density.

35. The apparatus of claim 31 wherein one of the second set of characteristics comprises the first and second second-level memories being directly writable so as to not require erasing of existing data prior to writing.

36. The apparatus of claim 31 wherein the first and second first-level memories comprises dynamic random access memories (DRAMs).

37. The apparatus of claim 36 wherein the first and second second-level memories comprise vertically stacked memories.

38. The apparatus of claim 31 wherein the first and second second-level memories comprise vertically stacked memory cell memories.

39. The apparatus of claim 31 wherein the first and second second-level memories comprise any of:

phase change memories;

resistive memories;

ferroelectric memories;

ferromagnetic memories;

dielectric memories.