Atomic remote memory operations in cache mirroring storage systems

• US 7,020,743 B2
• Filed: 02/24/2003
• Issued: 03/28/2006
• Est. Priority Date: 02/24/2003
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Rao, Raghavendra J., Lee, Whay Sing
• Primary Examiner(s): Verbrugge, Kevin
• Application Number:
  US10/373,922
• Publication Number:
  US 20040168032A1
• Time in Patent Office: 1,128 Days
• Field of Search: 711/3, 711/113, 711/114, 711/141, 711/118, 711/162, 714/6
• US Class Current: 711/113
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064

Abstract

Embodiments of the present invention provide methods and apparatuses for accessing MD atomically. For one embodiment, the local cache and the remote cache are each provided with an MBE. To perform an atomic access operation, the local processing system allocates a burst descriptor buffer. The burst descriptor buffer is used to specify the addresses to be atomically accessed. For write operations, the burst descriptor buffer also specifies the update data. The addresses, and data, if any, are then transmitted to the remote cache as a series of packets constituting a single data access request. The MBE of the remote cache then performs the operations contained in the request. Upon completion of all of the operations the remote MBE transmits an acknowledgement. For read operations the remote MBE also transmits the requested data.

Claims

1. A method, comprising:

(a) allocating a memory buffer for use in a data mirroring operation;

(b) updating a local copy of data stored in a local cache while storing, in said memory buffer, addresses of memory locations of the local cache that are modified during said updating;

(c) performing a burst update to a remote cache communicatively coupled to the local cache by (i) fetching data words identified by the addresses of the memory locations of the local cache, (ii) transmitting the data words to the remote cache, and (iii) receiving and storing the data words in a temporary location accessible by the remote cache until an end-of-burst signal is received;

(d) following receipt of the end-of-burst signal, updating, using the data words, memory locations in the remote cache that correspond to the addresses of the memory locations of the local cache, said updating taking place atomically so as not to permit intervening access to the remote cache; and

(e) sending an acknowledgement indicating the updating of the remote cache is complete.

2. The method of claim 1, wherein allocating the memory buffer comprises allocating storage locations of a scatter burst descriptor buffer (SBDB) for atomic memory operation status indicators and said address of memory locations of the local cache.

3. The method of claim 2, wherein allocating the memory buffer further comprises allocating storage locations of the SBDB for a count of memory locations corresponding to data words to be transmitted to the remote cache.

4. The method of claim 1, wherein the addresses of the local cache memory locations are placed in the memory buffer along with a count of said local cache memory locations and a remote cache identification indicator.

5. The method of claim 1, further comprising and prior to performing the burst update, requesting the local cache to perform a burst update to the remote cache subsequent to a time when the local cache has achieved a consistent data state.

6. The method of claim 2, wherein requesting the local cache to perform a burst update comprises issuing a request packet for an atomic scatter operation to the local cache, the request packet including a series of memory address/data value pairs.

7. The method of claim 6, wherein the memory address/data value pairs include respective ones of the memory addresses of the local cache, each followed by a corresponding one of the data values.

8. The method of claim 6, wherein the request packet includes a count indicator to indicate the number of memory addresses.

9. The method of claim 6, wherein the request packet also includes one or more of:

a source cache indicator, a destination cache indicator, a burst identification, and/or a cyclic redundancy check.

10. The method of claim 2, wherein sending an acknowledgement comprises sending a response packet for an atomic scatter operation, the response packet including one or more of:

a burst identification indicator, a source cache indicator, a destination cache indicator, and/or a status indicator to indicate completion or failure of the updating operation.

11. the method of claim 2, further comprising receiving the acknowledgement from the remote cache and updating a status indicator to reflect completion or failure of the updating operation.

12. A machine-readable medium having one or more executable instructions stored thereon, which when executed by a digital processing system, cause the digital processing system to perform a method, the method comprising:

(a) allocating a memory buffer for use in a data mirroring operation;

(b) updating a local copy of data stored in a local cache while storing, in said memory buffer, addresses of memory locations of the local cache that are modified during said updating;

(c) performing a burst update to a remote cache communicatively coupled to the local cache by (i) fetching data words identified by the addresses of the memory locations of the local cache, (ii) transmitting the data words to the remote cache, and (iii) receiving an acknowledgement from the remote cache indicating that updating of the remote cache has been completed, said updating of the remote cache involving first storing the data words in a temporary location accessible by the remote cache until an end-of-burst signal is received, following receipt of the end-of-burst signal, updating, using the data words, memory locations in the remote cache that correspond to the addresses of the memory locations of the local cache, said updating taking place atomically so as not to permit intervening access to the remote cache;

and sending the acknowledgement.

13. The machine-readable medium of claim 12, wherein allocating the memory buffer comprises allocating storage locations of a scatter burst descriptor buffer (SBDB) for atomic memory operation status indicators and said address of memory locations of the local cache.

14. The machine readable medium of claim 13, wherein allocating the memory buffer further comprises allocating storage locations of the SBDB for a count of memory locations corresponding to data words to be transmitted to the remote cache.

15. The machine readable medium of claim 12, wherein the addresses of the local cache memory locations are placed in the memory buffer along with a count of said local cache memory locations and a remote cache identification indicator.

16. The machine readable medium of claim 12, wherein the method further comprises, prior to performing the burst update, requesting the local cache to perform a burst update to the remote cache subsequent to a time when the local cache has achieved a consistent data state.

17. The machine readable medium of claim 16, wherein requesting the local cache to perform a burst update comprises issuing a request packet for an atomic scatter operation to the local cache, the request packet including a series of memory address/data value pairs.

18. The machine readable medium of claim 17, wherein the memory address/data value pairs include respective ones of the memory addresses of the local cache, each followed by a corresponding one of the data values.

19. The machine readable medium of claim 17, wherein the request packet includes a count indicator to indicate the number of memory addresses.

20. The machine readable medium of claim 19, wherein the request packet also includes one or more of:

a source cache indicator, a destination cache indicator, a burst identification, and a cyclic redundancy check field.

21. A method, comprising:

(a) allocating a memory buffer for use in a data mirroring operation;

(b) populating the memory buffer with addresses that need to be read atomically from a remote cache;

(c) initiating a burst read operation;

(d) transmitting the contents of the memory buffer;

(e) receiving, at the remote cache, the contents of the memory buffer and storing them in a temporary location until an end-of-burst signal is received;

(f) following receipt of the end-of-burst signal, performing read operations to all of the addresses in the remote cache atomically while allowing no intervening access to the remote cache;

(g) transmitting a series of response packets corresponding to the read operations;

(h) receiving the response packets and placing data values for the addresses in the memory buffer; and

(i) upon receipt of an end of response indicator for the burst read operation, updating a status indicator to indicate completion of the burst read operation.

22. The method of claim 21, wherein allocating the memory buffer comprises allocating a gather burst descriptor buffer so as to store the addresses that need to be read atomically from the remote cache as well as a remote cache identification indicator and a count indicator to indicate the number of addresses to be read from.

23. The method of claim 21, wherein the burst read operation is initiated following a write operation into a status field of the memory buffer.

24. The method of claim 21, wherein the burst read operation is initiated following a read operation performed on contents of a status field of the memory buffer.

25. The method of claim 21, wherein transmitting the contents of the memory buffer comprises transmitting a series of request packets, each of the request packets including a series of memory addresses.

26. The method of claim 25, wherein each of the request packets further includes one or more of:

a count indicator to indicate the number of said memory addresses, a burst identification indicator, a source cache indicator, a destination cache indicator, and a cyclic redundancy check field.

27. The method of claim 25, wherein each of the response packets comprises one or more of:

a number of address/value pairs, each including memory addresses as requested followed by memory values as read from the remote cache;

a count indicator to indicate the number of said address/value pairs;

a burst identification indicator;

a source cache indicator;

a destination cache indicator; and

a cyclic redundancy check field.

28. A machine-readable medium having one or more executable instructions stored thereon, which when executed by a digital processing system, cause the digital processing system to perform a method, the method comprising:

(a) allocating a memory buffer for use in a data mirroring operation;

(b) populating the memory buffer with addresses that need to be read atomically from a remote cache;

(c) initiating a burst read operation;

(d) transmitting the contents of the memory buffer;

(e) receiving a series of response packets indicating completion of the burst read operation by which the contents of the memory buffer were received and stored in a temporary location until an end-of-burst signal was received, after which read operations to all of the addresses in the remote cache were performed atomically while allowing no intervening access to the remote cache, and the response packets corresponding to the read operations transmitted upon completion thereof; and

(f) upon receipt of an end of response indicator for the burst read operation, updating a status indicator to indicate completion of the burst read operation.

29. The machine readable medium of claim 28, wherein allocating the memory buffer comprises allocating a gather burst descriptor buffer so as to store the addresses that need to be read atomically from the remote cache as well as a remote cache identification indicator and a count indicator to indicate the number of addresses to be read from.

30. The machine readable medium of claim 28, wherein the burst read operation is initiated following a write operation into a status field of the memory buffer.

31. The machine readable medium of claim 28, wherein the burst read operation is initiated following a read operation performed on contents of a status field of the memory buffer.

32. The machine readable medium of claim 28, wherein transmitting the contents of the memory buffer comprises transmitting a series of request packets, each of the request packets including a series of memory addresses.

33. The machine readable medium of claim 32, wherein each of the request packets further includes one or more of:

a count indicator to indicate the number of said memory addresses, a burst identification indicator, a source cache indicator, a destination cache indicator, and a cyclic redundancy check field.

34. The machine readable medium of claim 28, wherein each of the response packets comprises one or more of:

a number of address/value pairs, each including memory addresses as requested followed by memory values as read from the remote cache;

a count indicator to indicate the number of said address/value pairs;

a burst identification indicator;

a source cache indicator;

a destination cache indicator; and

a cyclic redundancy check field.