Memory sharing across distributed nodes

• US 10,223,116 B2
• Filed: 03/14/2013
• Issued: 03/05/2019
• Est. Priority Date: 10/02/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Loewenstein, Paul N., Johnson, John G., Aingaran, Kathirgamar, Radovic, Zoran
• Primary Examiner(s): Warren, Tracy A
• Application Number:
  US13/828,555
• Publication Number:
  US 20140095810A1
• Time in Patent Office: 2,182 Days
• Field of Search: 711147, 711148, 711141, 707634, 707 8, 707655, 707656
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/082 Associative directories G06F12/0822 takes precedence
• CPC Class Codes: G06F 12/12 Replacement control
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 12/1475 in a virtual system, e.g. with translation means
• CPC Class Codes: G06F 13/102 where the programme performs an interfacing function, e.g. device driver G06F13/105 takes precedence; scheduling within device drivers G06F9/52; contention policies within device drivers G06F9/4881
• CPC Class Codes: G06F 13/362 with centralised access control
• CPC Class Codes: G06F 13/4068 Electrical coupling
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 15/7889 Reconfigurable logic implemented as a co-processor instruction execution using a coprocessor G06F9/3877
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24553 of query operations
• CPC Class Codes: G06F 16/24558 Binary matching operations
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/24569 Query processing with adaptation to specific hardware, e.g. adapted for using GPUs or SSDs
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 21/6209 : to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 9/30065 : Loop control instructions; iterative instructions, e.g. LOOP, REPEAT
• CPC Class Codes: G06F 9/381 : Loop buffering
• CPC Class Codes: G06F 9/3851 : from multiple instruction streams, e.g. multistreaming
• CPC Class Codes: G06F 9/3867 : using instruction pipelines
• CPC Class Codes: G06F 9/546 : Message passing systems or structures, e.g. queues
• CPC Class Codes: H04L 69/14 : Multichannel or multilink protocols
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A method and apparatus are disclosed for enabling nodes in a distributed system to share one or more memory portions. A home node makes a portion of its main memory available for sharing, and one or more sharer nodes mirrors that shared portion of the home node'"'"'s main memory in its own main memory. To maintain memory coherency, a memory coherence protocol is implemented. Under this protocol, load and store instructions that target the mirrored memory portion of a sharer node are trapped, and store instructions that target the shared memory portion of a home node are trapped. With this protocol, valid data is obtained from the home node and updates are propagated to the home node. Thus, no “dirty” data is transferred between sharer nodes. As a result, the failure of one node will not cause the failure of another node or the failure of the entire system.

Claims

1. In a distributed system comprising a first node and a second node, wherein the first node has a first main memory and the second node has a second main memory, wherein the first main memory and the second main memory comprise random access memory, a method performed by the first node, comprising:

generating and storing, in the first main memory, a mapping that maps one or more memory addresses in a first memory location in the first main memory to one or more virtual memory addresses corresponding to a second memory location in the second main memory;

executing, by a processor on the first node, a set of program instructions pertaining to a particular thread of execution, wherein the set of program instructions includes a load instruction to load data from the first memory location of the first main memory, and a store instruction to store updated data into the first memory location of the first main memory;

wherein executing the load instruction comprises;

determining, by the processor, whether the data in the first memory location is valid;

in response to a determination that the data in the first memory location is invalid, causing the load instruction to trap, which causes the processor to suspend execution of the set of program instructions and to begin execution of a first set of trap handling instructions;

wherein executing the first set of trap handling instructions causes;

based on the mapping, obtaining valid data from the second memory location of the second main memory, and storing the valid data into the first memory location of the first main memory; and

updating a validity indicator to indicate that the data in the first memory location is valid; and

resuming, by the processor, execution of the set of program instructions;

wherein executing the store instruction comprises;

causing the store instruction to trap, which causes the first processor to suspend execution of the set of program instructions and to begin execution of a second set of trap handling instructions;

wherein the second set of trap handling instructions causes, based on the mapping, propagating the updated data to the second node to be stored within the second memory location of the second main memory; and

resuming, by the first processor, execution of the set of program instructions.

2. The method of claim 1, wherein the first node comprises a coprocessor, and wherein causing valid data to be obtained from the second memory location of the second main memory, and stored into the first memory location of the first main memory comprises:

the processor providing one or more instructions to the coprocessor to cause the coprocessor to obtain the valid data from the second memory location of the second main memory, and to store the valid data into the first memory location of the first main memory.

3. The method of claim 1, wherein the first node comprises a first coprocessor and the second node comprises a second coprocessor, and wherein causing valid data to be obtained from the second memory location of the second main memory, and stored into the first memory location of the first main memory comprises:

the processor providing one or more instructions to the first coprocessor to cause the first coprocessor to obtain, through interaction with the second coprocessor, the valid data from the second memory location of the second main memory, and to store the valid data into the first memory location of the first main memory.

4. The method of claim 1, wherein:

while executing the set of trap handling instructions, the processor further causing;

a directory entry associated with the second memory location of the second main memory of the second node to be locked, wherein the directory entry is stored on the second node;

information in the directory entry to be updated to indicate that the first node is a valid sharer of the second memory location of the second main memory; and

the directory entry to be unlocked.

5. The method of claim 1, wherein:

while executing the set of trap handling instructions, the processor further causing;

updated data to be obtained from a store buffer on the first node, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet; and

the updated data to be stored into the first memory location of the first main memory.

6. The method of claim 1, wherein:

while executing the set of trap handling instructions, and prior to causing the valid data to be obtained from the second memory location of the second main memory, and stored into the first memory location of the first main memory, the processor causing;

updated data to be obtained from a store buffer on the first node, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet; and

the updated data to be propagated to the second node and stored within the second memory location of the second main memory to give rise to an updated set of data which represents valid data.

7. The method of claim 1, wherein the method further comprises:

while executing the set of trap handling instructions, the first processor;

storing the updated data into the first memory location of the first main memory; and

wherein causing the updated data to eventually be propagated to the second node comprises;

storing the updated data into a store buffer for eventual propagation to the second node.

8. The method of claim 1,

wherein causing the updated data to be propagated to the second node comprises:

storing the updated data into a store buffer for propagation to the second node.

9. The method of claim 8, wherein storing the updated data into a store buffer for propagation to the second node comprises:

determining whether the store buffer contains existing updated data that is destined for the second memory location of the second main memory; and

in response to a determination that the store buffer contains existing updated data that is destined for the second memory location of the second main memory, merging the updated data with the existing updated data in the store buffer.

10. The method of claim 8, further comprising:

after resuming execution of the set of program instructions, and while executing the set of program instructions, the first processor;

retrieving the updated data from the store buffer; and

causing the updated data to be propagated to the second node and stored within the second memory location of the second main memory.

11. The method of claim 8, further comprising:

a second processor retrieving the updated data from the store buffer; and

the second processor causing the updated data to be propagated to the second node and stored within the second memory location of the second main memory;

wherein the second processor may be the first processor or another processor on the first node.

12. The method of claim 11, wherein the first node comprises a first coprocessor and the second node comprises a second coprocessor, and wherein causing the updated data to be propagated to the second node and stored within the second memory location of the second main memory comprises:

the second processor providing one or more instructions to the first coprocessor to cause the first coprocessor to interact with the second coprocessor to cause the updated data to be propagated to the second node and to be stored within the second memory location of the second main memory.

13. The method of claim 8, wherein the distributed system further comprises a third node, wherein the third node has a third main memory, wherein the mapping also maps the one or more virtual memory addresses to one or more memory addresses in a third memory location in the third main memory, and wherein the method further comprises:

a second processor retrieving the updated data from the store buffer; and

the second processor causing;

the updated data to be propagated to the second node and stored within the second memory location of the second main memory; and

based on the mapping, data currently stored in the third memory location of the third main memory of the third node to be invalidated;

wherein the second processor may be the first processor or another processor on the first node.

14. The method of claim 13, wherein the first node comprises a first coprocessor and the third node comprises a second coprocessor, and wherein causing data currently stored in the third memory location of the third main memory of the third node to be invalidated comprises:

the second processor providing one or more instructions to the first coprocessor to cause the first processor to interact with the second coprocessor to cause the data currently stored in the third memory location of the third main memory to be invalidated.

15. The method of claim 8, further comprising:

a second processor retrieving the updated data from the store buffer; and

the second processor causing;

a directory entry associated with the second memory location of the second main memory to be locked, wherein the directory entry is stored on the second node; and

the updated data to be propagated to the second node and stored within the second memory location of the second main memory; and

the directory entry to be unlocked;

wherein

the second processor may be the first processor or another processor on the first node.

16. The method of claim 8, wherein the distributed system further comprises a third node, wherein the third node has a third main memory, wherein the mapping also maps the one or more memory addresses in the second memory location of the second main memory to one or more memory addresses in a third memory location in the third main memory, and wherein the method further comprises:

a second processor retrieving the updated data from the store buffer; and

the second processor causing;

a directory entry associated with the second memory location of the second main memory to be locked, wherein the directory entry is stored on the second node, and wherein information in the directory entry indicates that the third node is currently a valid sharer of the second memory location of the second main memory;

the updated data to be propagated to the second node and stored within the second memory location of the second main memory;

data stored in the third memory location of the third main memory of the third node to be invalidated;

the information in the directory entry to be updated such that the information no longer indicates the third node as a valid sharer of the second memory location of the second main memory; and

the directory entry to be unlocked;

wherein the second processor may be the first processor or another processor on the first node.

17. The method of claim 1,

wherein the method further comprises:

while executing the set of trap handling instructions, the first processor;

storing the updated data into the first memory location of the first main memory; and

wherein causing the updated data to be propagated to the second node comprises;

causing the updated data to be propagated to the second node and stored within the second memory location of the second main memory.

18. The method of claim 1,

wherein the method further comprises:

while executing the set of trap handling instructions, the first processor;

invalidating data currently stored in the first memory location of the first main memory; and

wherein causing the updated data to be propagated to the second node comprises;

causing the updated data to be propagated to the second node and stored within the second memory location of the second main memory.

19. The method of claim 1 wherein executing the store instruction further comprises:

determining whether the first memory location of the first main memory is writeable;

in response to determining the first memory location is not writeable, causing the store instruction to trap.

20. A first node for use in a distributed computing system, the first node comprising:

a first main memory, wherein the first main memory stores a mapping that maps one or more memory addresses in a first memory location in the first main memory to one or more virtual memory addresses corresponding to a second memory location in a second main memory on a second node of the distributed system, wherein the first main memory and the second main memory comprise random access memory;

a first set of trap handling instructions and a second set of trap handling instructions; and

one or more processors including a first processor, the first processor operable to execute a load instruction to load data from the first memory location of the first main memory, and a store instruction to store data into the first memory location of the first main memory, wherein the load instruction and the store instruction are part of a set of program instructions pertaining to a particular thread of execution;

wherein executing the load instruction includes determining whether data in the first memory location of the first main memory is valid, and in response to a determination that the data in the first memory location of the first main memory is invalid, to cause the load instruction to trap, which would cause the first processor to suspend execution of the set of program instructions and to begin execution of the first set of trap handling instructions; and

wherein the first set of trap handling instructions, when executed by the first processor, would cause the first processor to cause;

valid data to be obtained from the second memory location of the second main memory based on the mapping, and stored into the first memory location of the first main memory; and

a validity indicator to be updated to indicate that the data in the first memory location is valid; and

execution of the set of program instructions to be resumed;

wherein the first processor comprises circuitry operable to cause the store instruction to trap, which would cause the first processor to suspend execution of the set of program instructions and to begin execution of the second set of trap handling instructions; and

wherein the second set of trap handling instructions, when executed by the first processor, would cause the first processor to;

cause the updated data to be propagated to the second node to be stored within the second memory location of the second main memory based on the mapping; and

resume execution of the set of program instructions.

21. The first node of claim 20,

wherein the first node further comprises a first coprocessor;

and

wherein the set of trap handling instructions, when executed by the first processor, would cause the first processor to cause valid data to be obtained from the second memory location of the second main memory, and stored into the first memory location of the first main memory by providing one or more instructions to the first coprocessor; and

wherein the first coprocessor is operable to respond to the one or more instructions by obtaining, through interaction with a second coprocessor on the second node, the valid data from the second memory location of the second main memory, and storing the valid data into the first memory location of the first main memory.

22. The first node of claim 20, wherein the set of trap handling instructions, when executed by the first processor, would cause the first processor to further cause:

a directory entry associated with the second memory location of the second main memory of the second node to be locked, wherein the directory entry is stored on the second node;

information in the directory entry to be updated to indicate that the first node is a valid sharer of the second memory location of the second main memory; and

the directory entry to be unlocked.

23. The first node of claim 20,

wherein the first node further comprises a store buffer;

and

wherein the set of trap handling instructions, when executed by the first processor, would cause the first processor to further cause;

updated data to be obtained from the store buffer, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet; and

the updated data to be stored into the first memory location of the first main memory.

24. The first node of claim 20,

wherein the first node further comprises a store buffer;

and

wherein the set of trap handling instructions, when executed by the first processor, would cause the first processor, prior to causing the valid data to be obtained from the second memory location of the second main memory, and stored into the first memory location of the first main memory, to cause;

updated data to be obtained from the store buffer, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet; and

the updated data to be propagated to the second node and stored within the second memory location of the second main memory to give rise to an updated set of data which represents valid data.

25. The first node of claim 20,

wherein the first node further comprises a store buffer;

wherein the trap handling instructions, when executed by the first processor, would further cause the first processor to;

store the updated data into the first memory location of the first main memory; and

wherein the trap handling instructions, when executed by the first processor, would cause the first processor to cause the updated data to be propagated to the second node by storing the updated data into the store buffer for propagation to the second node.

26. The first node of claim 20,

wherein the first node further comprises a store buffer;

and

wherein the trap handling instructions, when executed by the first processor, would cause the first processor to cause the updated data to be propagated to the second node by storing the updated data into the store buffer for propagation to the second node.

27. The first node of claim 26,

wherein storing the updated data into the store buffer for propagation to the second node comprises:

determining whether the store buffer contains existing updated data that is destined for the second memory location of the second main memory; and

in response to a determination that the store buffer contains existing updated data that is destined for the second memory location of the second main memory, merging the updated data with the existing updated data in the store buffer.

28. The first node of claim 26,

wherein the first node further comprises a set of update propagation instructions, which may be part of the set of program instructions pertaining to the particular thread of execution, or separate therefrom;

wherein the set of update propagation instructions, when executed by the first processor or another processor on the first node, would cause the first processor or another processor to;

retrieve the updated data from the store buffer; and

cause the updated data to be propagated to the second node and stored within the second memory location of the second main memory.

29. The first node of claim 26,

wherein the first node further comprises a first coprocessor;

wherein the first node further comprises a set of update propagation instructions, which may be part of the set of program instructions pertaining to the particular thread of execution, or separate therefrom;

wherein the set of update propagation instructions, when executed by the first processor or another processor on the first node, would cause the first processor or another processor to;

retrieve the updated data from the store buffer; and

provide one or more instructions to the first coprocessor to cause the first coprocessor to propagate the updated data to the second node and to cause the updated data to be stored in the second memory location of the second main memory of the second node; and

wherein the first coprocessor is operable to respond to the one or more instructions by interacting with a second coprocessor on the second node to cause the updated data to be propagated to the second node and stored within the second memory location of the second main memory.

30. The first node of claim 26,

wherein the mapping also maps the one or more memory addresses in the second memory location in the second main memory of the second node to one or more memory addresses in a third memory location in a third main memory of a third node;

wherein the first node further comprises a set of update propagation instructions, which may be part of the set of program instructions pertaining to the particular thread of execution, or separate therefrom;

wherein the set of update propagation instructions, when executed by the first processor or another processor on the first node, would cause the first processor or another processor to;

retrieve the updated data from the store buffer; and

cause;

the updated data to be propagated to the second node and stored within the second memory location of the second main memory; and

based on the mapping, data currently stored in the third memory location of the third main memory of the third node to be invalidated.

31. The first node of claim 26,

wherein the mapping also maps the one or more memory addresses in the second memory location in the second main memory of the second node to one or more memory addresses in a third memory location in a third main memory of a third node;

wherein the first node further comprises a first coprocessor;

wherein the first node further comprises a set of update propagation instructions, which may be part of the set of program instructions pertaining to the particular thread of execution, or separate therefrom;

wherein the set of update propagation instructions, when executed by the first processor or another processor on the first node, would cause the first processor or another processor to;

retrieve the updated data from the store buffer;

cause the updated data to be propagated to the second node and stored within the second memory location of the second main memory; and

based on the mapping, provide one or more instructions to the first coprocessor to cause the first coprocessor to cause data currently stored in the third memory location of the third main memory of the third node to be invalidated; and

wherein the first coprocessor is operable to respond to the one or more instructions by interacting with a second coprocessor on the third node to cause the data currently stored in the third memory location of the third main memory of the third node to be invalidated.

32. The first node of claim 26,

wherein the first node further comprises a set of update propagation instructions, which may be part of the set of program instructions pertaining to the particular thread of execution, or separate therefrom;

wherein the set of update propagation instructions, when executed by the first processor or another processor on the first node, would cause the first processor or another processor to;

retrieve the updated data from the store buffer; and

cause;

a directory entry associated with the second memory location of the second main memory of the second node to be locked, wherein the directory entry is stored on the second node;

the updated data to be propagated to the second node and stored within the second memory location of the second main memory; and

the directory entry to be unlocked.

33. The first node of claim 26,

wherein the mapping also maps the one or more memory addresses in the second memory location in the second main memory of the second node to one or more memory addresses in a third memory location in a third main memory of a third node;

wherein the first node further comprises a set of update propagation instructions, which may be part of the set of program instructions pertaining to the particular thread of execution, or separate therefrom;

wherein the set of update propagation instructions, when executed by the first processor or another processor on the first node, would cause the first processor or another processor to;

retrieve the updated data from the store buffer; and

cause;

a directory entry associated with the second memory location of the second main memory of the second node to be locked, wherein the directory entry is stored on the second node, and wherein information in the directory entry indicates that the third node is currently a valid sharer of the second memory location of the second main memory;

the updated data to be propagated to the second node and stored within the second memory location of the second main memory;

data currently stored in the third memory location of the third main memory of the third node to be invalidated;

the information in the directory entry to be updated such that the information no longer indicates the third node as a valid sharer of the second memory location of the second main memory; and

the directory entry to be unlocked.

34. The first node of claim 20,

wherein the trap handling instructions, when executed by the first processor, would further cause the first processor to:

store the updated data into the first memory location of the first main memory; and

wherein the trap handling instructions, when executed by the first processor, would cause the first processor to cause the updated data to be propagated to the second node by causing the updated data to be propagated to the second node and stored within the second memory location of the second main memory.

35. The first node of claim 20,

wherein the trap handling instructions, when executed by the first processor, would further cause the first processor to:

invalidate data currently stored in the first memory location of the first main memory; and

wherein the trap handling instructions, when executed by the first processor, would cause the first processor to cause the updated data to be propagated to the second node by causing the updated data to be propagated to the second node and stored within the second memory location of the second main memory.

36. The first node of claim 20 wherein executing the store instruction further comprises:

determining whether the first memory location of the first main memory is writeable;

in response to determining the first memory location is not writeable, causing the store instruction to trap.

37. In a distributed system comprising a first node and a second node, wherein the first node has a first main memory and the second node has a second main memory, wherein the first main memory and the second main memory comprise random access memory, a method performed by the second node, comprising:

generating and storing, in the second main memory, a mapping that maps one or more memory addresses in a first memory location in the first main memory to one or more virtual memory addresses corresponding to a second memory location in the second main memory;

executing, by a first processor on the second node, a store instruction to store updated data into the second memory location of the second main memory, wherein the store instruction is part of a set of program instructions pertaining to a particular thread of execution;

causing the store instruction to trap, which causes the first processor to suspend execution of the set of program instructions and to begin execution of a set of trap handling instructions;

executing, by the first processor, the set of trap handling instructions, wherein executing the set of trap handling instructions causes;

based on the mapping, storing the updated data into the second memory location of the second main memory; and

resuming, by the first processor, execution of the set of program instructions.

38. The method of claim 37, further comprising:

while executing the set of trap handling instructions, the first processor;

causing data currently stored in the first memory location of the first main memory to be invalidated.

39. The method of claim 37, further comprising:

while executing the set of trap handling instructions, the first processor;

prior to storing the updated data into the second memory location of the second main memory, accessing and locking a directory entry associated with the second memory location of the second main memory; and

after storing the updated data into the second memory location of the second main memory, unlocking the directory entry.

40. The method of claim 37, further comprising:

while executing the set of trap handling instructions, the first processor;

prior to storing the updated data into the second memory location of the second main memory, accessing and locking a directory entry associated with the second memory location of the second main memory, wherein information in the directory entry indicates that the first node is currently a valid sharer of the second memory location of the second main memory;

causing data currently stored in the first memory location of the first main memory to be invalidated;

updating the information in the directory entry such that the information no longer indicates the first node as being a valid sharer of the second memory location of the second main memory; and

after storing the updated data into the second memory location of the second main memory, unlocking the directory entry.

41. The method of claim 37, further comprising:

while executing the set of trap handling instructions, the first processor;

storing information into a buffer indicating that data in the second memory location of the second main memory has been updated;

a second processor retrieving the information from the buffer; and

the second processor causing data currently stored in the first memory location of the first main memory to be invalidated;

wherein the second processor may be the first processor or another processor on the second node.

42. The method of claim 37, further comprising:

while executing the set of trap handling instructions, the first processor;

storing information into a buffer indicating that data in the second memory location of the second main memory has been updated;

a second processor retrieving the information from the buffer;

the second processor accessing and locking a directory entry associated with the second memory location of the second main memory, wherein information in the directory entry indicates that the first node is currently a valid sharer of the second memory location of the second main memory;

the second processor causing data currently stored in the first memory location of the first main memory to be invalidated;

the second processor updating the information in the directory entry such that the information no longer indicates the first node as being a valid sharer of the second memory location of the second main memory; and

the second processor unlocking the directory entry;

wherein the second processor may be the first processor or another processor on the second node.

43. A second node for use in a distributed computing system comprising a first node and the second node, the second node comprising:

a second main memory, wherein the second main memory stores a mapping that maps one or more virtual memory addresses corresponding to a second memory location in the second main memory to one or more memory addresses in a first memory location in a first main memory on the first node, wherein the first main memory and the second main memory comprise random access memory;

a set of trap handling instructions; and

one or more processors including a first processor, the first processor operable to execute a store instruction to store updated data into the second memory location of the second main memory, wherein the store instruction is part of a set of program instructions pertaining to a particular thread of execution, and wherein the first processor comprises circuitry operable to cause the store instruction to trap, which would cause the first processor to suspend execution of the set of program instructions and to begin execution of the set of trap handling instructions; and

wherein the set of trap handling instructions, when executed by the first processor, would cause the first processor to;

store the updated data into the second memory location of the second main memory based on the mapping; and

resume execution of the set of program instructions.

44. The second node of claim 43,

wherein the trap handling instructions, when executed by the first processor, would further cause the first processor to:

cause data currently stored in the first memory location of the first main memory to be invalidated.

45. The second node of claim 43,

wherein the trap handling instructions, when executed by the first processor, would further cause the first processor to:

prior to storing the updated data into the second memory location of the second main memory, access and lock a directory entry associated with the second memory location of the second main memory; and

after storing the updated data into the second memory location of the second main memory, unlock the directory entry.

46. The second node of claim 43,

wherein the trap handling instructions, when executed by the first processor, would further cause the first processor to:

prior to storing the updated data into the second memory location of the second main memory, access and lock a directory entry associated with the second memory location of the second main memory, wherein information in the directory entry indicates that the first node is currently a valid sharer of the second memory location of the second main memory;

cause data currently stored in the first memory location of the first main memory to be invalidated;

update the information in the directory entry such that the information no longer indicates the first node as being a valid sharer of the second memory location of the second main memory; and

after storing the updated data into the second memory location of the second main memory, unlock the directory entry.

47. The second node of claim 43,

wherein the second node further comprises a set of invalidation instructions, which may be part of the set of program instructions pertaining to the particular thread of execution, or separate therefrom;

wherein the trap handling instructions, when executed by the first processor, would further cause the first processor to;

store information into a buffer indicating that data in the second memory location of the second main memory has been updated; and

wherein the set of invalidation instructions, when executed by the first processor or another processor on the second node, would cause the first processor or another processor to;

retrieve the information from the buffer; and

cause data currently stored in the first memory location of the first main memory to be invalidated.

48. The second node of claim 43,

wherein the second node further comprises a set of invalidation instructions, which may be part of the set of program instructions pertaining to the particular thread of execution, or separate therefrom;

wherein the trap handling instructions, when executed by the first processor, would further cause the first processor to;

store information into a buffer indicating that data in the second memory location of the second main memory has been updated; and

wherein the set of invalidation instructions, when executed by the first processor or another processor on the second node, would cause the first processor or another processor to;

retrieve the information from the buffer;

access and lock a directory entry associated with the second memory location of the second main memory, wherein information in the directory entry indicates that the first node is currently a valid sharer of the second memory location of the second main memory;

cause data currently stored in the first memory location of the first main memory to be invalidated;

update the information in the directory entry such that the information no longer indicates the first node as being a valid sharer of the second memory location of the second main memory; and

unlock the directory entry.