Memory sharing across distributed nodes

• US 9,679,084 B2
• Filed: 03/14/2013
• Issued: 06/13/2017
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Loewenstein, Paul N.
• Primary Examiner(s): Nguyen, Phuoc
• Application Number:
  US13/828,983
• Publication Number:
  US 20140279894A1
• Time in Patent Office: 1,552 Days
• Field of Search: 709216
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957

Abstract

A method and apparatus are disclosed for enabling nodes in a distributed system to share one or more memory portions. A home node makes a portion of its main memory available for sharing, and one or more sharer nodes mirrors that shared portion of the home node'"'"'s main memory in its own main memory. To maintain memory coherency, a memory coherence protocol is implemented. Under this protocol, a special data value is used to indicate that data in a mirrored memory location is not valid. This enables a sharer node to know when to obtain valid data from a home node. With this protocol, valid data is obtained from the home node and updates are propagated to the home node. Thus, no “dirty” data is transferred between sharer nodes. Consequently, the failure of one node will not cause the failure of another node or the failure of the entire system.

Claims

1. A first node for use in a distributed computing system, the first node comprising:

a first main memory, wherein a first memory location in the first main memory is usable to mirror a second memory location in a second main memory on a second node of the distributed computing system, wherein said first memory location stores first data that includes a first data value;

one or more sets of program instructions; and

one or more processors;

where the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to perform the operations of;

loading said first data value from the first memory location of the first main memory;

determining whether the first data value is the same as a special data value, wherein the special data value is a predetermined value that is used to indicate invalid data; and

in response to a determination that the first data value is the same as the special data value;

causing valid data to store in place of said first data in the first memory location of the first main memory to be obtained from the second memory location of the second main memory.

2. The first node of claim 1, wherein the first node further comprises a storage buffer, and wherein the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to further perform the operations of:

in response to a determination that the first data value is the same as the special data value;

storing the valid data from the second memory location of the second main memory into an interim memory location in the first main memory;

obtaining updated data from the storage buffer, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet;

updating the valid data in the interim memory location with the updated data to derive a set of modified valid data; and

storing the modified valid data from the interim memory location into the first memory location of the first main memory.

3. The first node of claim 1, wherein the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to further perform the operation of:

during initialization, storing the special data value into the first memory location of the first main memory.

4. The first node of claim 1, wherein the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to further perform the operations of:

in response to a determination that the first data value is the same as the special data value;

prior to causing valid data to be obtained from the second memory location of the second main memory;

obtaining a local lock on the first memory location of the first main memory;

loading data again from the first memory location of the first main memory, wherein the data has a second data value which may be the same as or different from the first data value; and

determining whether the second data value is the same as the special data value;

wherein the operation of causing valid data to be obtained from the second memory location of the second main memory is performed only if the second data value is the same as the special data value.

5. The first node of claim 1,

wherein causing valid data to be obtained from the second memory location of the second main memory comprises:

causing a directory entry associated with the second memory location of the second main memory to be locked, wherein the directory entry is stored in the second main memory; and

wherein the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to further perform the operation of;

in response to a determination that the first data value is the same as the special data value;

causing, after the valid data is obtained from the second memory location of the second main memory;

(a) the directory entry to be updated to indicate that the first node is now a valid sharer of the second memory location of the second main memory; and

(b) the directory entry to be unlocked.

6. The first node of claim 1,

wherein the first node further comprises a storage buffer;

wherein causing valid data to be obtained from the second memory location of the second main memory comprises;

causing a directory entry associated with the second memory location of the second main memory to be locked, wherein the directory entry is stored in the second main memory; and

wherein the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to further perform the operations of;

in response to a determination that the first data value is the same as the special data value;

storing the valid data from the second memory location of the second main memory into an interim memory location in the first main memory;

obtaining updated data from the storage buffer, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet;

updating the valid data in the interim memory location with the updated data to derive a set of modified valid data;

storing the modified valid data from the interim memory location into the first memory location of the first main memory; and

causing;

(a) the directory entry to be updated to indicate that the first node is now a valid sharer of the second memory location of the second main memory; and

(b) the directory entry to be unlocked.

7. The first node of claim 1,

wherein the first node further comprises a first input/output (I/O) interface;

wherein causing valid data to be obtained from the second memory location of the second main memory comprises;

the one or more processors providing one or more instructions to the first I/O interface to cause the I/O interface to obtain valid data from the second memory location of the second main memory; and

wherein the first I/O interface is configured to respond to the one or more instructions from the one or more processors by interacting with a second I/O interface on the second node to obtain from the second I/O interface the valid data from the second memory location of the second main memory.

8. The first node of claim 1,

wherein the first node further comprises a first input/output (I/O) interface;

wherein causing valid data to be obtained from the second memory location of the second main memory comprises;

the one or more processors providing one or more instructions to the first I/O interface to cause the I/O interface to obtain valid data from the second memory location of the second main memory; and

wherein the first I/O interface is configured to respond to the one or more instructions from the one or more processors by interacting with a second I/O interface on the second node to;

(a) cause a directory entry associated with the second memory location of the second main memory to be locked, wherein the directory entry is stored in the second main memory;

(b) obtain the valid data from the second memory location of the second main memory;

(c) cause the directory entry to be updated to indicate that the first node is now a valid sharer of the second memory location of the second main memory; and

(d) cause the directory entry to be unlocked.

9. The first node of claim 1,

wherein the first node further comprises a storage buffer;

wherein the first node further comprises a first input/output (I/O) interface;

wherein causing valid data to be obtained from the second memory location of the second main memory comprises;

the one or more processors providing one or more instructions to the first I/O interface to cause the I/O interface to obtain valid data from the second memory location of the second main memory; and

wherein the first I/O interface is configured to respond to the one or more instructions from the one or more processors by;

(a) interacting with a second I/O interface on the second node;

(i) to cause a directory entry associated with the second memory location of the second main memory to be locked, wherein the directory entry is stored in the second main memory; and

(ii) to obtain the valid data from the second memory location of the second main memory;

(b) storing the valid data from the second memory location of the second main memory into an interim memory location in the first main memory;

(c) obtaining updated data from the storage buffer, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet;

(d) updating the valid data in the interim memory location with the updated data to derive a set of modified valid data;

(e) storing the modified valid data from the interim memory location into the first memory location of the first main memory; and

(f) interacting with the second I/O interface;

(i) to cause the directory entry to be updated to indicate that the first node is now a valid sharer of the second memory location of the second main memory; and

(ii) to cause the directory entry to be unlocked.

10. The first node of claim 1,

wherein the first node further comprises a storage buffer;

and

wherein the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to further perform the operations of;

in response to a determination that the first data value is the same as the special data value;

prior to causing valid data to be obtained from the second memory location of the second main memory;

obtaining updated data from the storage buffer, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet; and

causing the updated data to be propagated to the second node and stored into the second memory location of the second main memory to give rise to an updated set of data which represents valid data.

11. A first node for use in a distributed computing system, the first node comprising:

a first main memory, wherein a first memory location in the first main memory is usable to mirror a second memory location in a second main memory on a second node of the distributed computing system, wherein said first memory location stores first data that includes a first data value;

a store buffer;

one or more sets of program instructions; and

one or more processors;

where the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to perform the operations of;

loading said first data value from the first memory location of the first main memory;

determining whether the first data value is the same as a special data value, wherein the special data value is a predetermined value that is used to indicate invalid data;

in response to a determination that the first data value is not the same as the special data value, storing updated data into the first memory location of the first main memory;

in response to a determination that the first data value is the same as the special data value, foregoing storing the updated data into the first memory location of the first main memory; and

storing the updated data into the store buffer for eventual propagation to the second node to be stored within the second memory location of the second main memory.

12. The first node of claim 11, wherein storing the updated data into the store buffer comprises:

determining whether the store buffer contains existing updated data that is destined for the second memory location of the second main memory; and

in response to a determination that the store buffer contains existing updated data that is destined for the second memory location of the second main memory, merging the updated data with the existing updated data in the store buffer.

13. The first node of claim 11,

wherein the first node further comprises a first input/output (I/O) interface;

and

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause the updated data to be propagated to the second node and stored into the second memory location of the second main memory.

14. The first node of claim 11,

wherein the first node further comprises a first input/output (I/O) interface;

and

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause;

(a) a directory entry associated with the second memory location of the second main memory to be locked, wherein the directory entry is stored on the second node;

(b) the updated data to be stored into the second memory location of the second main memory; and

(c) the directory entry to be unlocked.

15. The first node of claim 11,

wherein the second memory location of the second main memory is also mirrored in a third memory location of a third main memory of a third node;

wherein the first node further comprises a first input/output (I/O) interface;

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause the updated data to be stored into the second memory location of the second main memory; and

wherein the first I/O interface is configured to interact with a third I/O interface on the third node to cause data in the third memory location of the third main memory to be invalidated.

16. The first node of claim 11,

wherein the second memory location of the second main memory is also mirrored in a third memory location of a third main memory of a third node;

wherein the first node further comprises a first input/output (I/O) interface;

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause the updated data to be stored into the second memory location of the second main memory; and

wherein the first I/O interface is configured to interact with a third I/O interface on the third node to cause the special data value, or another special data value, to be stored into the third memory location of the third main memory to indicate that data in the third memory location of the third main memory is invalid.

17. The first node of claim 11,

wherein the second memory location of the second main memory is also mirrored in a third memory location of a third main memory of a third node;

wherein the first node further comprises a first input/output (I/O) interface;

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause;

a directory entry associated with the second memory location of the second main memory to be accessed and locked, wherein the directory entry is stored on the second node, and wherein information in the directory entry indicates that the third node is currently a valid sharer of the second memory location of the second main memory;

the updated data to be stored into the second memory location of the second main memory;

the information in the directory entry to be updated such that the information no longer indicates the third node as a valid sharer of the second memory location of the second main memory; and

the directory entry to be unlocked; and

wherein the first I/O interface is configured to interact with a third I/O interface on the third node to cause data in the third memory location of the third main memory to be invalidated.

18. The first node of claim 11,

wherein the second memory location of the second main memory is also mirrored in a third memory location of a third main memory of a third node;

wherein the first node further comprises a first input/output (I/O) interface;

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause;

a directory entry associated with the second memory location of the second main memory to be accessed and locked, wherein the directory entry is stored on the second node, and wherein information in the directory entry indicates that the third node is currently a valid sharer of the second memory location of the second main memory;

the updated data to be stored into the second memory location of the second main memory;

the information in the directory entry to be updated such that the information no longer indicates the third node as a valid sharer of the second memory location of the second main memory; and

the directory entry to be unlocked; and

wherein the first I/O interface is configured to interact with a third I/O interface on the third node to cause the special data value, or another special data value, to be stored into the third memory location of the third main memory to indicate that data in the third memory location of the third main memory is invalid.

19. The first node of claim 11,

wherein the second memory location of the second main memory is also mirrored in a third memory location of a third main memory of a third node;

wherein the first node further comprises a first input/output (I/O) interface; and

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause the second I/O interface to;

(a) access and lock a directory entry that is associated with the second memory location of the second main memory, wherein the directory entry is stored on the second node, and wherein information in the directory entry indicates that the third node is currently a valid sharer of the second memory location of the second main memory;

(b) store the updated data into the second memory location of the second main memory;

(c) interact with a third I/O interface on the third node to cause the special data value, or another special data value, to be stored into the third memory location of the third main memory to indicate that data in the third memory location of the third main memory is invalid;

(d) update the information in the directory entry to no longer indicate the third node as a valid sharer of the second memory location of the second main memory; and

(e) unlock the directory entry.

20. A second node for use in a distributed computing system comprising a first node and the second node, the second node comprising:

a second main memory, wherein a second memory location in the second main memory is usable to be mirrored in a first memory location in a first main memory on the first node;

one or more sets of program instructions; and

one or more processors;

wherein the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to perform the operations of;

storing updated data into the second memory location of the second memory; and

causing data currently stored in the first memory location of the first main memory to be a predetermined special data value, wherein the special data value is used to indicate invalid data.

21. The second node of claim 20, wherein the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to further perform the operations of:

accessing a directory entry associated with the second memory location of the second main memory, wherein information in the directory entry indicates that the first node is currently a valid sharer of the second memory location of the second main memory; and

updating the information in the directory entry such that the information no longer indicates the first node as being a valid sharer of the second memory location of the second main memory.

22. The second node of claim 20, wherein the one or more sets of program instructions, when executed by the one or more processors, would cause the one or more processors to further perform the operations of:

accessing and locking a directory entry associated with the second memory location of the second main memory, wherein information in the directory entry indicates that the first node is currently a valid sharer of the second memory location of the second main memory;

updating the information in the directory entry such that the information no longer indicates the first node as being a valid sharer of the second memory location of the second main memory; and

unlocking the directory entry.

23. The second node of claim 20,

wherein the second node further comprises a second input/output (I/O) interface;

wherein causing data currently stored in the first memory location of the first main memory to be the predetermined special value comprises;

providing one or more instructions to the second I/O interface to instruct the second I/O interface to invalidate data currently stored in the first memory location of the first main memory; and

wherein the second I/O interface is configured to respond to the one or more instructions by interacting with a first I/O interface on the first node to cause the first I/O interface to invalidate the data currently stored in the first memory location of the first main memory.

24. The second node of claim 20,

wherein the second node further comprises a second input/output (I/O) interface;

wherein causing data currently stored in the first memory location of the first main memory to be the predetermined special value comprises;

providing one or more instructions to the second I/O interface to instruct the second I/O interface to invalidate data currently stored in the first memory location of the first main memory; and

wherein the second I/O interface is configured to respond to the one or more instructions by interacting with a first I/O interface on the first node to cause the first I/O interface to store a predetermined special data value into the first memory location of the first main memory, wherein the special data value is used to indicate that data in the first memory location of the first main memory is invalid.

25. The second node of claim 20, wherein the second node further comprises a store buffer, and wherein causing data currently stored in the first memory location of the first main memory to be the predetermined special value comprises:

storing information into a store buffer, wherein the information indicates, or is usable to determine, that data currently stored in the first memory location of the first main memory is to be invalidated;

retrieving the information from the store buffer; and

causing data currently stored in the first memory location of the first main memory to be invalidated.

26. A method comprising:

loading said first data value from a first memory location of a first main memory on a first node of a distributed computing system, wherein the first memory location in the first main memory is usable to mirror a second memory location in a second main memory on a second node of the distributed computing system, wherein said first memory location stores first data that includes a first data value;

determining whether the first data value is the same as a special data value, wherein the special data value is a predetermined value that is used to indicate invalid data; and

in response to a determination that the first data value is the same as the special data value;

causing valid data to store in place of said first data in the first memory location of the first main memory to be obtained from the second memory location of the second main memory.

27. The method of claim 26, wherein the first node further comprises a storage buffer, the method further comprising:

in response to a determination that the first data value is the same as the special data value;

storing the valid data from the second memory location of the second main memory into an interim memory location in the first main memory;

obtaining updated data from the storage buffer, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet;

updating the valid data in the interim memory location with the updated data to derive a set of modified valid data; and

storing the modified valid data from the interim memory location into the first memory location of the first main memory.

28. The method of claim 26 further comprising:

during initialization, storing the special data value into the first memory location of the first main memory.

29. The method of claim 26 further comprising:

wherein the first node further comprises a first input/output (I/O) interface;

wherein causing valid data to be obtained from the second memory location of the second main memory comprises;

providing one or more instructions to the first I/O interface to cause the I/O interface to obtain valid data from the second memory location of the second main memory; and

wherein the first I/O interface is configured to respond to the one or more instructions by interacting with a second I/O interface on the second node to obtain from the second I/O interface the valid data from the second memory location of the second main memory.

30. The method of claim 26, wherein the first node further comprises a storage buffer, the method further comprising:

in response to a determination that the first data value is the same as the special data value;

prior to causing valid data to be obtained from the second memory location of the second main memory;

obtaining updated data from the store buffer, wherein the updated data is intended to be stored in the second memory location of the second main memory of the second node but has not been propagated to the second node yet; and

causing the updated data to be propagated to the second node and stored into the second memory location of the second main memory to give rise to an updated set of data which represents valid data.

31. A method comprising:

loading a first data value from a first memory location of a first main memory on a first node of a distributed computing system, wherein the first memory location in the first main memory is usable to mirror a second memory location in a second main memory on a second node of the distributed computing system, wherein said first memory location stores first data that includes a first data value;

determining whether the first data value is the same as a special data value, wherein the special data value is a predetermined value that is used to indicate invalid data;

in response to a determination that the first data value is not the same as the special data value, storing updated data into the first memory location of the first main memory;

in response to a determination that the first data value is the same as the special data value, foregoing storing the updated data into the first memory location of the first main memory; and

storing the updated data into a store buffer of the first node for eventual propagation to the second node to be stored within the second memory location of the second main memory.

32. The method of claim 31, wherein storing the updated data into the store buffer comprises:

determining whether the store buffer contains existing updated data that is destined for the second memory location of the second main memory; and

in response to a determination that the store buffer contains existing updated data that is destined for the second memory location of the second main memory, merging the updated data with the existing updated data in the store buffer.

33. The method of claim 31,

wherein the first node further comprises a first input/output (I/O) interface;

and

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause the updated data to be propagated to the second node and stored into the second memory location of the second main memory.

34. The method of claim 31,

wherein the first node further comprises a first input/output (I/O) interface;

and

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause;

(a) a directory entry associated with the second memory location of the second main memory to be locked, wherein the directory entry is stored on the second node;

(b) the updated data to be stored into the second memory location of the second main memory; and

(c) the directory entry to be unlocked.

35. The method of claim 31,

wherein the second memory location of the second main memory is also mirrored in a third memory location of a third main memory of a third node;

wherein the first node further comprises a first input/output (I/O) interface;

wherein the first I/O interface is configured to interact with a second I/O interface on the second node to cause the updated data to be stored into the second memory location of the second main memory; and

wherein the first I/O interface is configured to interact with a third I/O interface on the third node to cause data in the third memory location of the third main memory to be invalidated.

36. A method comprising:

storing updated data into a second memory location of a second main memory on a second node of a distributed computing system, wherein the second memory location in the second main memory is usable to be mirrored in a first memory location in a first main memory on a first node; and

causing data currently stored in the first memory location of the first main memory to be a predetermined special data value, wherein the special data value is used to indicate invalid data.

37. The method of claim 36 further comprising:

accessing a directory entry associated with the second memory location of the second main memory, wherein information in the directory entry indicates that the first node is currently a valid sharer of the second memory location of the second main memory; and

updating the information in the directory entry such that the information no longer indicates the first node as being a valid sharer of the second memory location of the second main memory.

38. The method of claim 36 further comprising:

accessing and locking a directory entry associated with the second memory location of the second main memory, wherein information in the directory entry indicates that the first node is currently a valid sharer of the second memory location of the second main memory;

updating the information in the directory entry such that the information no longer indicates the first node as being a valid sharer of the second memory location of the second main memory; and

unlocking the directory entry.

39. The method of claim 36,

wherein the second node further comprises a second input/output (I/O) interface;

wherein causing data currently stored in the first memory location of the first main memory to be the predetermined special data value comprises;

providing one or more instructions to the second I/O interface to instruct the second I/O interface to invalidate data currently stored in the first memory location of the first main memory; and

wherein the second I/O interface is configured to respond to the one or more instructions by interacting with a first I/O interface on the first node to cause the first I/O interface to invalidate the data currently stored in the first memory location of the first main memory.