Method and apparatus for tracking cached addresses for maintaining cache coherency in a computer system having multiple caches

• US 20050160226A1
• Filed: 01/20/2004
• Published: 07/21/2005
• Est. Priority Date: 01/20/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hoover, Russell Dean, Shedivy, David Alan, Averill, Duane Arlyn, Voytovich, Martha Ellen
• Granted Patent: US 7,788,452 B2
• Time in Patent Office: Days
• US Class Current: 711/133
• CPC Class Codes: G06F 12/0817 using directory methods
• CPC Class Codes: G06F 12/0891 using clearing, invalidating or resetting means
• CPC Class Codes: G06F 2212/2542 Non-uniform memory access [NUMA] architecture

Abstract

A computer system includes multiple caches and a cache line state directory structure, having at least a portion dedicated to a particular device cache within a particular device, and contains a fixed number of entries having a one-to-one correspondence to the cache lines of the cache to which it corresponds. The cache line state directory is used to determine whether it is necessary to send an invalidation message to the device cache. In the preferred embodiment, a dedicated portion of the cache line state directory structure corresponds to an I/O bridge device cache. Preferably, the cache line state directory also maintains state for one or more processor caches in a different format. The computer system preferably uses a NUMA architecture, the directories being maintained by node servers in each node.

Claims

1. A digital data processing system, comprising:

a memory;

at least one processor having at least one associated cache for temporarily caching data from said memory;

at least one device having a device cache, said device cache having a fixed number of slots for caching data, each slot caching a cache line of data; and

a cache coherency mechanism, said cache coherency mechanism including a cache line state directory structure, said cache coherency mechanism selectively determining whether to send cache line invalidation messages to said at least one device using state information in said cache line state directory structure, wherein at least a portion of said cache line state directory structure contains a plurality of cache line entries, each entry corresponding to a respective one of said plurality of slots for caching data of said device cache.

2. The digital data processing system of claim 1, wherein said device is an I/O bridge device.

3. The digital data processing system of claim 1, wherein a processor portion of said cache line state directory structure contains cache line state for at least one said cache associated with a processor, said cache coherency mechanism further selectively determining whether to send cache line invalidation messages to the processor with which the cache is associated using state information in said processor portion of said cache line directory structure.

4. The digital data processing system of claim 3, wherein said processor portion of said cache line state directory structure contains cache line state for a plurality of caches associated with a plurality of processors, said cache coherency mechanism further selectively determining whether to send cache line invalidation messages to any of said plurality of processors using state information in said processor portion of said cache line directory structure.

5. The digital data processing system of claim 1, wherein said digital data processing system comprises a plurality of nodes, each node containing at least one processor, a respective portion of said memory, and a respective portion of said cache coherency mechanism.

6. The digital data processing system of claim 5, wherein each said respective portion of said cache coherency mechanism in each respective node maintains cache line state information for cached data having a real address in the respective portion of said memory contained in the node.

7. The digital data processing system of claim 5, wherein each said respective portion of said cache coherency mechanism in each respective node maintains cache line state information for data cached in devices contained in the node.

8. The digital data processing system of claim 1,

wherein said digital data processing system comprises a plurality of devices having respective device caches, each said device cache having a respective fixed number of slots for caching data, each slot caching a cache line of data;

and

wherein said cache line state directory structure includes a plurality of portions, each portion corresponding to a respective one of said plurality of devices, each portion containing a plurality of cache line entries, each entry corresponding to a respective one of said plurality of slots for caching data of the device cache to which the respective portion corresponds.

9. A method for maintaining cache coherency in a digital data processing system, comprising the steps of:

maintaining a cache line state directory structure, said cache line state directory structure having at least a portion corresponding to a device cache in a device of said digital data processing system, said portion containing a plurality of cache line entries, each entry corresponding to a respective one of a plurality of slots for caching lines of data in said device cache;

responsive to each of a plurality of data access requests, accessing said cache line state directory structure to determine whether data having a data address referenced by the request is contained in said device cache;

for each of said plurality of data access requests, determining whether to send an invalidation message to said device based on whether said step of accessing said cache line state directory determines that data having a data address referenced by the request is contained in said device cache; and

for each of said plurality of data access requests, sending an invalidation message to said device responsive to the determination made by said step of determining whether to send an invalidation message.

10. The method of claim 9, wherein said device is an I/O bridge device.

11. The method of claim 9, further comprising the steps of:

receiving a plurality of data access requests for cache lines of data from said device, each data access request from said device including data identifying a slot of said device cache in which the cache line will be stored; and

responsive to receiving each said data access request from said device, updating said cache line state directory structure by writing cache line identifying information corresponding to the data access request at the entry corresponding to the slot in which the cache line requested by the data access request will be stored.

12. The method of claim 9, wherein said step of maintaining a cache line state directory structure comprises maintaining a first portion of said cache line state directory structure corresponding to said device cache, and a second portion of said cache line state directory structure corresponding to a plurality of caches associated with a plurality of processors, said method further comprising the steps of:

responsive to each of said plurality of data access requests, accessing said cache line state directory structure to determine whether data having a data address referenced by the request is contained in any of said plurality of processors;

for each of said plurality of data access requests, determining whether to send an invalidation message to any of said plurality of processors based on whether said step of accessing said cache line state directory structure determines that data having a data address referenced by the request is contained in any of said plurality of processors; and

for each of said plurality of data access requests, sending an invalidation message to at least one of said plurality of processors responsive to the determination made by said step of determining whether to send an invalidation message to any of said plurality of processors.

13. The method of claim 9, wherein said digital data processing system comprises a plurality of nodes, each node containing at least one processor, a respective portion of said memory, and a respective portion of said cache coherency mechanism.

14. A digital data processing system, comprising:

a memory;

a plurality of processors controlling a plurality of caches for temporarily caching data from said memory;

at least one device having a device cache, said device cache having a fixed number of slots for caching data, each slot corresponding to a cache line; and

a cache line state directory structure having a first portion for maintaining cache line state for lines of data cached in said plurality of caches controlled by said plurality of processors, and a second portion for maintaining cache line state for lines of data cached in said device cache;

wherein said first portion of said cache line state directory structure contains a plurality of cache line entries, each entry corresponding to a respective real address of cached data;

wherein said second portion of said cache line state directory structure contains a plurality of cache line entries, each entry corresponding to a respective one of said plurality of slots for caching data of said device cache.

15. The digital data processing system of claim 14, wherein said device is an I/O bridge device.

16. The digital data processing system of claim 14, wherein said digital data processing system comprises a plurality of nodes, each node containing at least one processor, a respective portion of said memory, and a respective portion of said cache line state directory structure.

17. The digital data processing system of claim 16, wherein each said respective portion of said cache line state directory structure each respective node contains cache line state information for cached data having a real address in the respective portion of said memory contained in the node.

18. The digital data processing system of claim 16, wherein each said respective portion of said cache line state directory structure each respective node contains cache line state information for data cached in devices contained in the node.

19. A cache coherency apparatus for a digital data processing system:

a communications interface for communicating with a plurality of devices;

a cache line state directory structure, wherein at least a portion of said cache line state directory structure corresponds to a cache having a plurality of slots for caching data in a first device of said plurality of devices, said at least a portion containing a plurality of cache line entries, each entry corresponding to a respective one of said plurality of slots for caching data of said cache in said first device; and

cache coherence control logic which selectively generates invalidation messages responsive to events affecting the validity of cached data, said cache coherence control logic determining whether to send cache line invalidation messages to said first device using state information in said at least a portion of said cache line state directory structure corresponding to said cache in said first device.

20. The cache coherency apparatus of claim 19, wherein said first device is an I/O bridge device.

21. The cache coherency apparatus of claim 19, wherein said cache line state directory structure contains a plurality of portions, including a first a portion corresponding to said cache in said first device, and a second portion corresponding to at least one cache associated with a processor, said cache coherence control logic further selectively determining whether to send cache line invalidation messages to said processor using state information in said second portion of said cache line state directory structure.

22. The cache coherency apparatus of claim 19, wherein said cache coherency apparatus is embodied in a single integrated circuit chip, said integrated circuit chip being separate from said first device.