Private memory table for reduced memory coherence traffic

• US 9,760,490 B2
• Filed: 07/15/2016
• Issued: 09/12/2017
• Est. Priority Date: 04/02/2015
• Status: Active Grant

• Current Assignee: Beijing Zitiao Network Technology Company Limited (ByteDance Ltd.)
• Original Assignee: International Business Machines Corporation
• Inventors: Daly, David M., Srinivasan, Vijayalakshmi
• Primary Examiner(s): Yi, David X
• Assistant Examiner(s): GOLDSCHMIDT, CRAIG S
• Application Number:
  US15/211,064
• Publication Number:
  US 20160328324A1
• Time in Patent Office: 424 Days
• Field of Search: None
• CPC Class Codes: G06F 12/0815 Cache consistency protocols
• CPC Class Codes: G06F 12/0817 using directory methods
• CPC Class Codes: G06F 12/0831 using a bus scheme, e.g. with bus monitoring or watching means
• CPC Class Codes: G06F 12/084 with a shared cache
• CPC Class Codes: G06F 2212/1008 Correctness of operation, e.g. memory ordering
• CPC Class Codes: G06F 2212/1016 Performance improvement
• CPC Class Codes: G06F 2212/1021 Hit rate improvement
• CPC Class Codes: G06F 2212/1048 Scalability
• CPC Class Codes: G06F 2212/151 Emulated environment, e.g. virtual machine
• CPC Class Codes: G06F 2212/251 Local memory within processor subsystem
• CPC Class Codes: G06F 2212/283 Plural cache memories
• CPC Class Codes: G06F 2212/608 Details relating to cache mapping
• CPC Class Codes: G06F 2212/621 Coherency control relating to peripheral accessing, e.g. from DMA or I/O device
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers

Abstract

A mechanism is provided for memory coherence in a multiple processor system. Responsive to a memory operation from a processing core of the multiple processor system resulting in a cache miss, the mechanism checks a private region table associated with the processing core. The memory operation attempts to access a memory region. Responsive to determining the memory region corresponds to an entry in the private region table, the mechanism performs a remote memory controller snoop of a remote memory controller without snooping the multiple processor system.

Claims

1. A method for memory coherence in a multiple processor system, the method comprising:

generating by a virtual machine manager of a first processing node in the multiple processor system, a private region table associated with a processing core of the first processing node, wherein;

the private region table stores an entry identifying a memory region, associated with a virtual machine executing on the first processing node, which is stored remotely in memory of a second processing node,

the entry includes a bitmask having a bit for each cache line of the memory region, and

the bit for a cache line of the memory region is set to a first value if the corresponding cache line is private to the virtual machine executing on the first processing node, or a second value if the cache line of the memory region is not private to the virtual machine;

responsive to a memory operation from the processing core in the first processing node of the multiple processor system resulting in a cache miss, checking the private region table associated with the processing core, wherein the memory operation attempts to access the memory region stored in memory of the second processing node of the multiple processor system;

responsive to determining the memory region corresponds to an entry in the private region table and the memory region is node-contained in the second processing node, performing the memory operation on the second processing node without snooping the entire multiple processor system;

responsive to another processing core accessing a cache line of the memory region, setting a value of a corresponding bit in the bitmask of the en to the second value; and

invalidating the entry in the private region table in response to all bits in the bitmask for the memory region being set to the second value.

2. The method of claim 1, further comprising performing the memory operation using snoopy protocol responsive to determining the memory region does not correspond to an entry in the private region table.

3. The method of claim 1, further comprising performing the memory operation using snoopy protocol responsive to determining the memory region is not node-contained in the second processing node.

4. The method of claim 1, wherein each entry of the private region table comprises a tag, a size, and least recently used (LRU) information.

5. The method of claim 1, wherein the virtual machine requests memory from the virtual machine manager, the virtual machine manager allocates a remote memory region to the virtual machine, and the virtual machine manager installs an entry in the private region table corresponding to the remote memory region.

6. The method of claim 1, wherein the multiple processor system comprises a plurality of nodes, wherein each node comprises a plurality of chips, wherein each chip comprises a plurality of chiplets, and wherein each chiplet comprises a processing core and a cache associated with the processing core.

7. The method of claim 6, wherein the private region table comprises a chiplet-level private region table and a chip-level private region table.

8. The method of claim 7, further comprising:

responsive to the chiplet-level private region table running out of space, moving a selected entry in the chiplet-level private region table to the chip-level private region table and adding a chiplet identifier to the selected entry.

9. The method of claim 1, further comprising responsive to migrating the virtual machine to a new processing core, migrating entries of the private region table to a private region table associated with the new processing core.

10. The method of claim 9, wherein migrating entries of the private region table to the private region table associated with the new processing core comprises performing a data cache block flush of all cache lines in all regions for the virtual machine.

11. The method of claim 9, wherein migrating entries of the private region table to the private region table associated with the new processing core comprises:

setting a flag on all private region table entries for the virtual machine;

installing entries in the private region table associated with the new processing core for the memory regions allocated to the virtual machine;

for each given memory region to migrate to the new processing core;

walk the given memory region and issue a data cache block move to cache inject into the new processing core;

invalidate a private region table entry for the given memory region in the private region table associated with the old processing core; and

clear the flag in the corresponding entry in the private region table associated with the new processing core.