Methods to perform disk writes in a distributed shared disk system needing consistency across failures

• US 7,200,623 B2
• Filed: 03/04/2002
• Issued: 04/03/2007
• Est. Priority Date: 11/24/1998
• Status: Expired due to Term

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Chan, Wilson Wai Shun, Srihari, Vinay, Brower, David, Bridge, William H., Bamford, Roger J., Chandrasekaran, Sashikanth, MacNaughton, Neil
• Primary Examiner(s): Le; Debbie M
• Application Number:
  US10/091,618
• Publication Number:
  US 20020095403A1
• Time in Patent Office: 1,856 Days
• Field of Search: 707/1, 707/8, 707/10, 707/100, 707/103, 707/104.1, 707200-206, 707/103.R, 709200-203, 709212-219, 711/1, 711/3, 711/100, 711/101, 711111-122, 711/130, 711141-147, 711150-154, 711/161, 711/162, 711/200, 711/213, 711/219
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/0815 Cache consistency protocols
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99954 Version management

Abstract

Techniques are provided for managing caches in a system with multiple caches that may contain different copies of the same data item. Specifically, techniques are provided for coordinating the write-to-disk operations performed on such data items to ensure that older versions of the data item are not written over newer versions, and to reduce the amount of processing required to recover after a failure. Various approaches are provided in which a master is used to coordinate with the multiple caches to cause a data item to be written to persistent storage. Techniques are also provided for managing checkpoints associated with the caches, where the checkpoints are used to determine the position at which to begin processing recovery logs in the event of a failure.

Claims

1. A method for managing caches in a system with multiple caches that contain different copies of a data item, comprising the steps of:

modifying the data item in a first node of said multiple caches to create a modified data item;

sending the modified data item from said first node to a second node of said multiple caches without durably storing the modified data item from said first node to persistent storage;

after said modified data item has been sent from said first node to said second node, said first node sending a request to a master of said data item for writing said data item to persistent storage; and

in response to said request, said master coordinating with said multiple caches to cause said data item to be written to persistent storage.

2. The method of claim 1 wherein:

the method includes the step of maintaining, within an ordered series of bins, entries for past-image versions of data items;

each bin in said ordered series corresponds to a time range;

a particular bin corresponds to the time range that covers the time at which the data item is modified in said first node; and

the step of sending a request is performed by sending a request for writing a particular bin of said ordered series of bins to persistent storage.

3. The method of claim 2 wherein the step of said master coordinating with said multiple caches to cause said data item to be written to persistent storage includes said master causing said multiple caches to write data items to persistent storage to cover all past image versions of data items that were modified during the time range of said particular bin.

4. The method of claim 3 further comprising the step of emptying said particular bin after said multiple caches write data items to persistent storage to cover all past image versions of data items that were modified during the time range of said particular bin.

5. The method of claim 4 wherein the step of emptying said particular bin includes the steps of:

discarding entries within said particular bin that are associated with past images that have last-dirtied times within the time range of said particular bin; and

moving to one or more other bins the entries within said particular bin that are associated with past images that have last-dirtied times later than the time range of said particular bin.

6. The method of claim 1 wherein the step of sending a request to a master is performed by sending the request to a global lock manager.

7. The method of claim 1 wherein the step of sending a request to a master is performed by sending the request to a lock manager that is one of a plurality of lock managers within a distributed lock management system.

8. The method of claim 1 further comprising the step of sending from the master, to interested nodes, write-notification messages indicating that said data item has been written to persistent storage, in response to said data item being written to persistent storage.

9. The method of claim 8 wherein the step of sending write-notification messages includes the master sending to at least one interested node a single message that notifies said at least one interested node that a plurality of data items have been written to persistent storage.

10. The method of claim 1 wherein the step of said first node sending a request to a master of said data item for writing said data item to persistent storage includes the first node sending to said master a single message that requests writing a plurality of data items to persistent storage, wherein said plurality of data items includes said data item.

11. The method of claim 10 wherein the step of sending a single message includes sending a message that identifies a bin to request that all data items that belong to the bin be written to persistent storage.

12. The method of claim 11 wherein the bin is associated with a range of time and includes data items that were first dirtied by the first node during said range of time and that were subsequently transferred to other nodes without first being written to persistent storage.

13. The method of claim 8 wherein the step of sending from the master to interested nodes write-notification messages includes the steps of:

immediately sending write-notification messages to a first set of interested nodes, where said first set of interested nodes includes the interested nodes that have requested said data item to be written to persistent storage; and

delaying the sending of write-notification messages to a second set of nodes, where said second set of nodes includes interested nodes that do not belong to said first set of interested nodes.

14. The method of claim 8 wherein the step of sending from the master to interested nodes includes delaying the sending of write-notification messages to at least one interested node.

15. The method of claim 14 wherein a write-notification message is sent to the at least one interested node in response to a lock request made by said at least one interested node.

16. The method of claim 14 wherein a write-notification message is sent to the at least one interested node in response to the at least one interested node requesting that said data item be written to persistent storage.

17. The method of claim 14 wherein a write-notification message is sent to the at least one interested node within a ping request that the master sends to the at least one interested node for the at least one interested node to transfer another data item to another node.

18. The method of claim 1 wherein the step of coordinating includes the steps of:

determining whether a version of said data item, that is at least as recent as said modified version, has already been written to persistent storage; and

if a version of said data item that is at least as recent as said modified version has already been written to persistent storage, then sending a write-notification message from said master to notify said first node that a version of said data item that is at least as recent as said modified version has already been written to persistent storage.

19. The method of claim 18 wherein the step of coordinating includes, if a version of said data item that is at least as recent as said modified version has not already been written to persistent storage, then sending a write-perform message from said master to grant permission for said modified version to be written to persistent storage.

20. The method of claim 1 wherein the step of coordinating includes the steps of:

selecting a particular node of said multiple caches that has a particular version of said data item, wherein said particular version is at least as recent as the modified data item in said first node; and

causing said particular version of said data item to be written from said particular node to persistent storage.

21. The method of claim 20 wherein the step of selecting a particular node includes selecting the node, of said multiple caches, that has a most recently modified version of said data item.

22. The method of claim 20 further comprising the step of the master informing the first node that said data item has been written to persistent storage in response to the master receiving confirmation that said particular version of said data item has been written to persistent storage.

23. The method of claim 20 further comprising the step of the master informing a set of caches that said data item has been written to persistent storage in response to the master receiving confirmation that said particular version of said data item has been written to persistent storage, wherein said set of caches includes caches, other than said particular node, that contain modified versions of said data item that are not more recent than said particular version.

24. A computer-readable medium carrying instructions for managing caches in a system with multiple caches that contain different copies of a data item, the instructions comprising instructions for performing the steps of:

modifying the data item in a first node of said multiple caches to create a modified data item;

sending the modified data item from said first node to a second node of said multiple caches without durably storing the modified data item from said first node to persistent storage;

after said modified data item has been sent from said first node to said second node, said first node sending a request to a master of said data item for writing said data item to persistent storage; and

in response to said request, said master coordinating with said multiple caches to cause said data item to be written to persistent storage.

25. The computer-readable medium of claim 24 wherein:

the computer-readable medium includes instructions for performing the step of maintaining, within an ordered series of bins, entries for past-image versions of data items;

each bin in said ordered series corresponds to a time range;

a particular bin corresponds to the time range that covers the time at which the data item is modified in said first node; and

the step of sending a request is performed by sending a request for writing a particular bin of said ordered series of bins to persistent storage.

26. The computer-readable medium of claim 25 wherein the step of said master coordinating with said multiple caches to cause said data item to be written to persistent storage includes said master causing said multiple caches to write data items to persistent storage to cover all past image versions of data items that were modified during the time range of said particular bin.

27. The computer-readable medium of claim 26 further comprising instructions for performing the step of emptying said particular bin after said multiple caches write data items to persistent storage to cover all past image versions of data items that were modified during the time range of said particular bin.

28. The computer-readable medium of claim 27 wherein the step of emptying said particular bin includes the steps of:

discarding entries within said particular bin that are associated with past images that have last-dirtied times within the time range of said particular bin; and

moving to one or more other bins the entries within said particular bin that are associated with past images that have last-dirtied times later than the time range of said particular bin.

29. The computer-readable medium of claim 24 wherein the step of sending a request to a master is performed by sending the request to a global lock manager.

30. The computer-readable medium of claim 24 wherein the step of sending a request to a master is performed by sending the request to a lock manager that is one of a plurality of lock managers within a distributed lock management system.

31. The computer-readable medium of claim 24 further comprising instructions for performing the step of sending from the master, to interested nodes, write-notification messages indicating that said data item has been written to persistent storage, in response to said data item being written to persistent storage.

32. The computer-readable medium of claim 31 wherein the step of sending write-notification messages includes the master sending to at least one interested node a single message that notifies said at least one interested node that a plurality of data items have been written to persistent storage.

33. The computer-readable medium of claim 24 wherein the step of said first node sending a request to a master of said data item for writing said data item to persistent storage includes the first node sending to said master a single message that requests writing a plurality of data items to persistent storage, wherein said plurality of data items includes said data item.

34. The computer-readable medium of claim 33 wherein the step of sending a single message includes sending a message that identifies a bin to request that all data items that belong to the bin be written to persistent storage.

35. The computer-readable medium of claim 34 wherein the bin is associated with a range of time and includes data items that were first dirtied by the first node during said range of time and that were subsequently transferred to other nodes without first being written to persistent storage.

36. The computer-readable medium of claim 31 wherein the step of sending from the master to interested nodes write-notification messages includes the steps of:

immediately sending write-notification messages to a first set of interested nodes, where said first set of interested nodes includes the interested nodes that have requested said data item to be written to persistent storage; and

delaying the sending of write-notification messages to a second set of nodes, where said second set of nodes includes interested nodes that do not belong to said first set of interested nodes.

37. The computer-readable medium of claim 31 wherein the step of sending from the master to interested nodes includes delaying the sending of write-notification messages to at least one interested node.

38. The computer-readable medium of claim 37 wherein a write-notification message is sent to the at least one interested node in response to a lock request made by said at least one interested node.

39. The computer-readable medium of claim 37 wherein a write-notification message is sent to the at least one interested node in response to the at least one interested node requesting that said data item be written to persistent storage.

40. The computer-readable medium of claim 37 wherein a write-notification message is sent to the at least one interested node within a ping request that the master sends to the at least one interested node for the at least one interested node to transfer another data item to another node.

41. The computer-readable medium of claim 24 wherein the step of coordinating includes the steps of:

determining whether a version of said data item, that is at least as recent as said modified version, has already been written to persistent storage; and

if a version of said data item that is at least as recent as said modified version has already been written to persistent storage, then sending a write-notification message from said master to notify said first node that a version of said data item that is at least as recent as said modified version has already been written to persistent storage.

42. The computer-readable medium of claim 41 wherein the step of coordinating includes, if a version of said data item that is at least as recent as said modified version has not already been written to persistent storage, then sending a write-perform message from said master to grant permission for said modified version to be written to persistent storage.

43. The computer-readable medium of claim 24 wherein the step of coordinating includes the steps of:

selecting a particular node of said multiple caches that has a particular version of said data item, wherein said particular version is at least as recent as the modified data item in said first node; and

causing said particular version of said data item to be written from said particular node to persistent storage.

44. The computer-readable medium of claim 43 wherein the step of selecting a particular node includes selecting the node, of said multiple caches, that has a most recently modified version of said data item.

45. The computer-readable medium of claim 43 further comprising instructions for performing the step of the master informing the first node that said data item has been written to persistent storage in response to the master receiving confirmation that said particular version of said data item has been written to persistent storage.

46. The computer-readable medium of claim 43 further comprising instructions for performing the step of the master informing a set of caches that said data item has been written to persistent storage in response to the master receiving confirmation that said particular version of said data item has been written to persistent storage, wherein said set of caches includes caches, other than said particular node, that contain modified versions of said data item that are not more recent than said particular version.