SYSTEM AND METHOD FOR EVENT MONITORING IN CACHE COHERENCE PROTOCOLS WITHOUT EXPLICIT INVALIDATIONS

• US 20190324910A1
• Filed: 06/28/2019
• Published: 10/24/2019
• Est. Priority Date: 01/03/2014
• Status: Active Grant

• Current Assignee: Eta Scale AB
• Original Assignee: Eta Scale AB
• Inventors: KAXIRAS, Stefanos, ROS, Alberto
• Granted Patent: US 11,237,966 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 12/0811 with multilevel cache hierarchies
• CPC Class Codes: G06F 12/0815 Cache consistency protocols
• CPC Class Codes: G06F 12/082 Associative directories G06F12/0822 takes precedence
• CPC Class Codes: G06F 12/0831 using a bus scheme, e.g. with bus monitoring or watching means
• CPC Class Codes: G06F 12/084 with a shared cache
• CPC Class Codes: G06F 12/128 adapted to multidimensional cache systems, e.g. set-associative, multicache, multiset or multilevel
• CPC Class Codes: G06F 13/1663 Access to shared memory
• CPC Class Codes: G06F 2212/314 In storage network, e.g. network attached cache
• CPC Class Codes: G06F 2212/621 Coherency control relating to peripheral accessing, e.g. from DMA or I/O device

Abstract

Synchronization events associated with cache coherence are monitored without using invalidations. A callback-read is issued to a memory address associated with the synchronization event, which callback-read either reads the last value written in the memory address or blocks until a next write takes place in the memory address and reads a newly written value.

Claims

1. A computer system comprising:

multiple processor cores;

at least one local cache memory associated with and operatively coupled to each core for storing one or more cache lines accessible only by the associated core;

a shared memory, the shared memory being operatively coupled to the local cache memories and accessible by the cores, the shared memory being capable of storing a plurality of cache lines; and

wherein a core issuing a callback-read to a memory address either reads the last value written in the memory address or is blocked from reading the memory address until the next write takes place in the memory address and then reads a new value of said next write such that the callback-read enables event monitoring for coherence of the at least one local cache and the shared memory without using explicit invalidations.

2. The computer system of claim 1, further comprising:

a callback directory containing a set of callback (CB) bits associated with the memory address, wherein each CB bit in the set corresponds to a core.

3. The computer system of claim 2, wherein:

when a CB bit corresponding to the core that issued the callback-read is set, the callback-read is completed by the core reading the last value in the memory address; and

when the CB bit corresponding to the core that issued the callback-read is unset, the callback-read triggers setting of the CB bit and the callback-read is completed when the new value is written in the memory address and the new value is forwarded to the core that issued the callback-read; and

when the new value is written in the memory address, the new value is forwarded to all of the cores that have their corresponding CB bit set for the memory address, CB bits previously set for the memory address are cleared, and CB bits previously unset for the memory address are set.

4. The computer system of claim 1, further comprising:

a callback directory containing a set of callback (CB) bits and full/empty (F/E) bits associated with the memory address, wherein each CB bit and each F/E bit of the set corresponds to a different one of the cores.

5. The computer system of claim 4, wherein:

when an F/E bit corresponding to the core that issued the callback-read is set, the callback-read from the core to the memory address with the set of CB and F/E bits is completed by the core reading the last value in the memory address;

when the F/E bit corresponding to the core that issued the callback-read is unset, the callback-read triggers setting of a CB bit which corresponds to the memory address, and the callback-read is completed when the new value is written in the memory address and the new value is forwarded to the core that issued the callback-read; and

when the new value is written in the memory address, the new value is forwarded to all of the cores that have a corresponding CB bit set for the memory address, corresponding CB bits previously set are cleared, and F/E bits that correspond to the CB bits previously unset are set.

6. The computer system of claim 4, wherein a read_through read of the memory address issued by a core bypasses the at least one local cache memory associated with the core and resets an F/E bit of the memory address that corresponds to the core.

7. The computer system of claim 4, wherein the callback directory also contains an All/One bit associated with the memory address;

and wherein when the All/One bit is set to one the F/E bits for the memory address operate in unison such that the F/E bits for the memory address act as a single F/E bit; and

wherein when the All/One bit is set to all the F/E bits for the memory address operate independently of one another such that each F/E bit has its value changed independently by a corresponding core.

8. The computer system of claim 7, wherein a write callback one write instruction to the memory address sets the All/One bit to one;

wherein the write callback one write instruction selects a single core that has its corresponding CB bit set for the memory address and forwards a written value to only this single core and resets the corresponding CB bit; and

further wherein when no core has its corresponding CB bit set for the memory address, the write callback one write instruction sets all the F/E bits which correspond to the memory address.

9. The computer system of claim 7, wherein a write callback zero write instruction to the memory address sets the All/One bit which corresponds to the memory address to one.

10. The computer system of claim 7, wherein a write callback all write instruction to the memory address sets the All/One bit to all;

forwards a written value to all of the cores that have their corresponding CB bit set for the memory address, clears the corresponding CB bits previously set and sets the F/E bits that correspond to CB bits previously unset.

11. The computer system of claim 2, wherein each of the multiple processor cores is configured to issue at least one of a callback read and a callback write to access the callback directory using one of a corresponding special load/store or atomic instruction.

12. The computer system of claim 3, wherein the callback directory is located at one or more cache levels where writes become globally visible in the computer system.

13. The computer system of claim 4, wherein when there is no free space in the callback directory, the memory address is evicted to accommodate a new memory address and a current data value of the evicted memory address is forwarded to all the cores that have the corresponding CB bit set for the evicted memory address;

and

wherein the CB bits for the new memory address that replaces the evicted address are initialized to unset and the F/E bits for the new memory are initialized to set.

14. The computer system of claim 2, where the CB bits correspond to program threads of the cores.

15. The computer system of claim 1, wherein the callback read to the memory address is issued by a compare-and-callback instruction wherein the core either continues to a next instruction or issues the callback read to the memory address.

16. The computer system of claim 15, wherein the compare-and-callback instruction takes the core to a power-saving state when the compare-and-callback instruction is awaiting a response to a callback read.

17. The computer system of claim 1, further comprising a callback queue which stores pending callback requests issued for the memory address.

18. The computer system of claim 1, further comprising a callback directory which contains at least one callback bit associated with the memory address, wherein the at least one callback bit is used by the computer system to determine whether the core reads the last value written in the memory address or is blocked from reading the memory address until the next write takes place in the memory address and then reads a new value.

19. The computer system of claim 18, wherein the callback directory also contains at least one full/empty (F/E) bit associated with the memory address wherein the at least one callback bit and the at least one F/E bit are used by the computer system to determine whether the core reads the last value written in the memory address or is blocked from reading the memory address until the next write takes place in the memory address and then reads a new value.

22. The method of claim 2, wherein:

when a CB bit corresponding to the core that issued the callback-read is set, the callback-read is completed by the core reading the last value in the memory address; and

when the CB bit corresponding to the core that issued the callback-read is unset, the callback-read triggers setting of the CB bit and the callback-read is completed when the new value is written in the memory address and the new value is forwarded to the core that issued the callback-read; and

when the new value is written in the memory address, the new value is forwarded to all of the cores that have their corresponding CB bit set for the memory address, CB bits previously set for the memory address are cleared, and CB bits previously unset for the memory address are set.

31. The method of claim 22, wherein a callback directory containing the CB bits is located at one or more cache levels where writes become globally visible in a computer system.

20. A method comprising:

monitoring synchronization events associated with cache coherence without using explicit invalidations by issuing a callback-read to a memory address, which callback-read either enables a processor core to read the last value written in the memory address, or blocks the processor core from reading the memory address until a next write takes place in the memory address and read a newly written value of said next write.

21. The method of claim 20 further, comprising:

associating a set of callback (CB) bits with the memory address, wherein each CB bit in the set corresponds to a core.

23. The method of claim 20, further comprising:

associating a set of callback (CB) bits and full/empty (F/E) bits with the memory address, wherein each CB bit and each F/E bit of the set corresponds to a different one of the cores.

24. The method of claim 23, wherein:

when an F/E bit corresponding to the core that issued the callback-read is set, the callback-read from the core to the memory address with the set of CB and F/E bits is completed by the core reading the last value in the memory address;

when the F/E bit corresponding to the core that issued the callback-read is unset, the callback-read triggers setting of a CB bit which corresponds to the memory address, and the callback-read is completed when the new value is written in the memory address and the new value is forwarded to the core that issued the callback-read; and

when the new value is written in the memory address, the new value is forwarded to all of the cores that have a corresponding CB bit set for the memory address, corresponding CB bits previously set are cleared, and F/E bits that correspond to the CB bits previously unset are set.

25. The method of claim 23, wherein a read_through read of the memory address issued by a core bypasses the at least one local cache memory associated with the core and resets an F/E bit of the memory address that corresponds to the core.

26. The method of claim 23, further comprising associating an All/One bit with the memory address;

and wherein when the All/One bit is set to one the F/E bits for the memory address operate in unison such that the F/E bits for the memory address act as a single F/E bit; and

wherein when the All/One bit is set to all the F/E bits for the memory address operate independently of one another such that each F/E bit has its value changed independently by a corresponding core.

27. The method of claim 26, wherein a write callback one write instruction to the memory address sets the All/One bit to one;

wherein the write callback one write instruction selects a single core that has its corresponding CB bit set for the memory address and forwards a written value to only this single core and resets the corresponding CB bit; and

further wherein when no core has its corresponding CB bit set for the memory address, the write callback one write instruction sets all the F/E bits which correspond to the memory address.

28. The method of claim 26, wherein a write callback zero write instruction to the memory address sets the All/One bit which corresponds to the memory address to one.

29. The method of claim 26, wherein a write callback all write instruction to the memory address sets the All/One bit to all;

forwards a written value to all of the cores that have their corresponding CB bit set for the memory address, clears the corresponding CB bits previously set and sets the F/E bits that correspond to the corresponding CB bits previously unset.

30. The method of claim 21, wherein each of the multiple processor cores is configured to issue at least one of a callback read and a callback write to access the callback directory using one of a corresponding special load/store or atomic instruction.

32. The method of claim 23, wherein when there is no free space in the callback directory, the memory address is evicted to accommodate a new memory address and a current data value of the evicted memory address is forwarded to all the cores that have the corresponding CB bit set for the evicted memory address;

and

wherein the CB bits for the new memory address that replaces the evicted address are initialized to unset and the F/E bits for the new memory are initialized to set.

33. The method of claim 21, where the CB bits correspond to program threads of the cores.

34. The method of claim 20, wherein the callback read to the memory address is issued by a compare-and-callback instruction wherein the core either continues to a next instruction or issues the callback read to the memory address.

35. The method of claim 34, wherein the compare-and-callback instruction takes the core to a power-saving state when the compare-and-callback instruction is awaiting a response to a callback read.

36. The method of claim 20, wherein a callback queue which stores pending callback requests issued for the memory address.

37. The method of claim 20, further comprising:

associating at least one callback bit with the memory address, wherein the at least one callback bit is used by a computer system to determine whether the core reads the last value written in the memory address or is blocked from reading the memory address until the next write takes place in the memory address and then reads a new value.

38. The method of claim 37, further comprising:

associating at least one full/empty (F/E) bit associated with the memory address wherein the at least one callback bit and the at least one F/E bit are used by the computer system to determine whether the core reads the last value written in the memory address or is blocked from reading the memory address until the next write takes place in the memory address and then reads a new value.