Controlling data lag in a replicated computer system

• US 8,838,919 B2
• Filed: 08/30/2010
• Issued: 09/16/2014
• Est. Priority Date: 08/30/2010
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Shi, Jia, Hu, Wei M., Srihari, Vinay H., Li, Yunrui, Girkar, Mahesh B., Garin, Benedicto E. Jr.
• Primary Examiner(s): Farrokh, Hashem
• Application Number:
  US12/871,805
• Publication Number:
  US 20120054533A1
• Time in Patent Office: 1,478 Days
• Field of Search: 711/162, 707/999.204
• US Class Current: 711/162
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

A method and system is provided for measuring, guaranteeing, and reducing replication data lag time between a primary system and one or more standby systems. Each standby system determines the lag time between the generation of a consistent version of data on the primary system and the time that the consistent version is applied on the standby system. Applications can request and be guaranteed to receive data from a standby system that is identical to the state on the primary system at the time of the query, or lag the primary state only by a maximum tolerable amount. A standby system may also publish a service that guarantees a maximum lag time and withdraw the service offer when the actual lag time exceeds the guaranteed lag time.

Implications for implementing synchronous and asynchronous replication as well as performance optimizations are also discussed.

Claims

1. A method for determining data replication lag time comprising:

a standby storage system receiving a stream of change records that record changes to data made at a primary storage system, each change record recording a change and including a system change value and a primary change time value, wherein the system change value identifies a particular state of data;

wherein the primary change time value represents a time said change occurred, said primary change time value being generated by the primary storage system;

said standby storage system maintaining a latest received primary system time value based on primary change time values recorded in the stream of change records;

said standby storage system applying said stream of change records to data at said standby storage system;

after said standby storage system applying, from said stream of change records, a particular change record having a particular system change value and a particular primary change time value, said standby storage system determining a data replication lag time for the particular system change value based on a difference between the latest received primary system time value and the particular primary change time value.

2. The method of claim 1, wherein after a set of change records are applied,

storing on said standby storage system a system change value in association with a corresponding primary change time value for one or more applied change records of said set of change records, wherein the system change value and the corresponding primary change time value are included in a same change record of said one or more applied change records.

3. The method of claim 1, further comprising storing a standby receive time value in association with the latest received primary system time value;

wherein determining said data replication lag time for the particular system change value further includes adding the difference between a current standby system time and the standby receive time value.

4. The method of claim 1, further comprising said standby storage system maintaining a current standby system change value that identifies the state of data stored in said standby storage system;

said standby storage system determining a data replication lag time for the current standby system change value.

5. The method of claim 4, further comprising said standby storage system receiving a request to read consistent data;

wherein the request to read consistent data specifies a maximum tolerable lag time; and

wherein said data replication lag time for the current standby system change value exceeds the maximum tolerable lag time, said standby storage system responding to the request with an indication that an error occurred.

6. The method of claim 4, further comprising said standby storage system updating said data replication lag time for the current standby system change value in response to a change in the current standby system change value.

7. The method of claim 6, further comprising said standby storage system providing a service that offers a maximum guaranteed lag time;

and

in response to determining that said data replication lag time for the current standby system change value exceeds the maximum guaranteed lag time, ceasing to offer said service.

8. The method of claim 1, further comprising receiving a heartbeat message from the primary storage system that includes a time value, and

in response to receiving the heartbeat message, updating the latest received primary system time value with the time value included in the heartbeat message.

9. The method of claim 1, wherein a group of change records within the stream of change records includes a header, wherein said header includes a time value for the primary storage system;

and

determining the latest received primary system time value based on the time value included in said header.

10. The method of claim 1, wherein the primary storage system is a primary database system, and said standby storage system is a standby database system.

11. A method comprising:

a primary storage system receiving a request to make a particular set of changes;

sending change records corresponding to said particular set of changes to one or more standby storage systems;

a standby storage system receiving said change records from the primary storage system and writing said change records to persistent storage before sending an acknowledgment of receipt of said change records to the primary storage system;

the primary storage system waiting to receive the acknowledgement of receipt of said change records before responding to the request;

said standby storage system receiving a read-only request to read data changed by said particular set of changes;

detecting that not all said change records have been applied at said standby storage system;

said standby storage system deferring reading of said data until said particular set of changes are applied to said standby storage system.

12. The method of claim 11, wherein said standby storage system receives a set of change records for changes that include both said particular set of changes and an additional set of changes not related to said particular set of changes;

said standby storage system receiving a request to read data changed by said particular set of changes;

said standby storage system deferring reading of said data until all changes included in said set of change records are applied to said standby storage system.

13. The method of claim 11, wherein said standby storage system receives a set of change records for changes that include both said particular set of changes and an additional set of changes not related to said particular set of changes;

said standby storage system receiving a request to read data changed by said particular set of changes, wherein the request specifies a particular system change value that identifies a state of data that is reached after applying said particular set of changes;

said standby storage system deferring reading of said data until said particular set of changes identified by the particular system change value are applied to said standby storage system; and

performing reading of said data without waiting for said additional set of changes to be applied.

14. The method of claim 11, wherein the request to read data comprises a plurality of read operations, and further comprising:

determining a set of change records to apply to said standby storage system before performing a first read operation of the plurality of read operations;

applying said set of change records;

performing the first read operation of the plurality of read operations;

performing remaining read operations of the plurality of read operations without further determining a set of change records to apply.

15. A method comprising:

a plurality of primary database system instances applying updates included in one or more transactions to a shared physical database, wherein each primary database system instance runs on a node of a multi-node clustered system;

a first primary database system instance processing a first request to commit a transaction wherein processing the request to commit the transaction includes;

determining a system change value corresponding to the transaction;

generating and sending a first change record to one or more standby database systems; and

sending a message to one or more other primary database system instances, wherein said message contains the system change value corresponding to the transaction;

in response to receiving said message, each of the one or more other primary database system instances determining whether the system change value is greater than a last system change value sent to the one or more standby database systems;

in response to determining that the system change value is greater than the last system change value sent to the one or more standby database systems, generating a second change record that includes the system change value contained in said message;

said each of the one or more other primary database system instances sending the second change record to said one or more standby database systems.

16. The method of claim 15, further comprising:

a standby database system receiving change records from a plurality of primary database system instances;

determining a lowest system change value received from a last change record received from each primary database system instance of the plurality of primary database system instances; and

applying only change records containing a system change value less than or equal to the lowest system change value.

17. A method comprising:

a primary storage system concurrently performing;

a) persistently storing change records for a set of changes; and

b) sending said change records to one or more standby storage systems;

said one or more standby storage systems receiving and persistently storing said change records;

after completing persistently storing change records for the set of changes, the primary storage system sending a notification to the one or more standby storage systems;

the one or more standby storage systems waiting to receive said notification before applying said set of changes.

18. The method of claim 17, wherein said notification is indicated by receipt of a next set of change records received by said standby storage system.

19. The method of claim 17, wherein said notification is sent independent of change records.

20. A method comprising:

a broker receiving a query from an application, wherein the query is associated with a maximum tolerable lag time value;

selecting a first standby storage system of a plurality of standby storage systems, wherein the first standby storage system monitors data lag time, wherein the data lag time is an amount of time between when data is generated on a primary storage system and when the data is applied on the standby storage system;

in response to receiving an error indicating that the data lag time monitored by the first standby storage system is greater than the maximum tolerable lag time value, said broker re-directing said query to a second storage system.

21. The method of claim 20, wherein the second storage system is a second standby storage system of the plurality of standby storage systems.

22. The method of claim 20, wherein the second storage system is the primary storage system.

23. The method of claim 20, wherein said broker selects the first standby storage system based on a guaranteed maximum lag time associated with a service published by the first standby storage system.

24. A non-transitory computer-readable medium storing one or more sequences of instructions which, when executed by one or more processors, cause performing of:

a standby storage system receiving a stream of change records that record changes to data made at a primary storage system, each change record recording a change and including a system change value and a primary change time value, wherein the system change value identifies a particular state of data;

wherein the primary change time value represents a time said change occurred, said primary change time value being generated by the primary storage system;

said standby storage system maintaining a latest received primary system time value based on primary change time values recorded in the stream of change records;

said standby storage system applying said stream of change records to data at said standby storage system;

after said standby storage system applying, from said stream of change records, a particular change record having a particular system change value and a particular primary change time value, said standby storage system determining a data replication lag time for the particular system change value based on a difference between the latest received primary system time value and the particular primary change time value.

25. The non-transitory computer-readable medium of claim 24, wherein after a set of change records are applied, the one or more sequences of instructions, when executed by the one or more processors, further cause performing of:

storing on said standby storage system a system change value in association with a corresponding primary change time value for one or more applied change records of said set of change records, wherein the system change value and the corresponding primary change time value are included in a same change record of said one or more applied change records.

26. The non-transitory computer-readable medium of claim 24, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of:

storing a standby receive time value in association with the latest received primary system time value;

wherein determining said data replication lag time for the particular system change value further includes adding the difference between a current standby system time and the standby receive time value.

27. The non-transitory computer-readable medium of claim 24, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of:

said standby storage system maintaining a current standby system change value that identifies the state of data stored in said standby storage system;

said standby storage system determining a data replication lag time for the current standby system change value.

28. The non-transitory computer-readable medium of claim 27, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of:

said standby storage system receiving a request to read consistent data;

wherein the request to read consistent data specifies a maximum tolerable lag time; and

wherein said data replication lag time for the current standby system change value exceeds the maximum tolerable lag time, said standby storage system responding to the request with an indication that an error occurred.

29. The non-transitory computer-readable medium of claim 27, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of:

said standby storage system updating said data replication lag time for the current standby system change value in response to a change in the current standby system change value.

30. The non-transitory computer-readable medium of claim 29, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of:

said standby storage system providing a service that offers a maximum guaranteed lag time; and

in response to determining that said data replication lag time for the current standby system change value exceeds the maximum guaranteed lag time, ceasing to offer said service.

31. The non-transitory computer-readable medium of claim 24, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of:

receiving a heartbeat message from the primary storage system that includes a time value, and

in response to receiving the heartbeat message, updating the latest received primary system time value with the time value included in the heartbeat message.

32. The non-transitory computer-readable medium of claim 24, wherein a group of change records within the stream of change records includes a header, wherein said header includes a time value for the primary storage system;

and wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of;

determining the latest received primary system time value based on the time value included in said header.

33. The non-transitory computer-readable medium of claim 24, wherein the primary storage system is a primary database system, and said standby storage system is a standby database system.

34. A non-transitory computer-readable medium storing one or more sequences of instructions which, when executed by one or more processors, cause performing of:

a primary storage system receiving a request to make a particular set of changes;

sending change records corresponding to said particular set of changes to one or more standby storage systems;

a standby storage system receiving said change records from the primary storage system and writing said change records to persistent storage before sending an acknowledgment of receipt of said change records to the primary storage system;

the primary storage system waiting to receive the acknowledgement of receipt of said change records before responding to the request;

said standby storage system receiving a read-only request to read data changed by said particular set of changes;

detecting that not all said change records have been applied at said standby storage system;

said standby storage system deferring reading of said data until said particular set of changes are applied to said standby storage system.

35. The non-transitory computer-readable medium of claim 34, wherein said standby storage system receives a set of change records for changes that include both said particular set of changes and an additional set of changes not related to said particular set of changes;

and wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of;

said standby storage system receiving a request to read data changed by said particular set of changes;

said standby storage system deferring reading of said data until all changes included in said set of change records are applied to said standby storage system.

36. The non-transitory computer-readable medium of claim 34, wherein said standby storage system receives a set of change records for changes that include both said particular set of changes and an additional set of changes not related to said particular set of changes;

and wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of;

said standby storage system receiving a request to read data changed by said particular set of changes, wherein the request specifies a particular system change value that identifies a state of data that is reached after applying said particular set of changes;

said standby storage system deferring reading of said data until said particular set of changes identified by the particular system change value are applied to said standby storage system; and

performing reading of said data without waiting for said additional set of changes to be applied.

37. The non-transitory computer-readable medium of claim 34, wherein the request to read data comprises a plurality of read operations, and wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of:

determining a set of change records to apply to said standby storage system before performing a first read operation of the plurality of read operations;

applying said set of change records;

performing the first read operation of the plurality of read operations;

performing remaining read operations of the plurality of read operations without further determining a set of change records to apply.

38. A non-transitory computer-readable medium storing one or more sequences of instructions which, when executed by one or more processors, cause performing of:

a plurality of primary database system instances applying updates included in one or more transactions to a shared physical database, wherein each primary database system instance runs on a node of a multi-node clustered system;

a first primary database system instance processing a first request to commit a transaction wherein processing the request to commit the transaction includes;

determining a system change value corresponding to the transaction;

generating and sending a first change record to one or more standby database systems; and

sending a message to one or more other primary database system instances, wherein said message contains the system change value corresponding to the transaction;

in response to receiving said message, each of the one or more other primary database system instances determining whether the system change value is greater than a last system change value sent to the one or more standby database systems;

in response to determining that the system change value is greater than the last system change value sent to the one or more standby database systems, generating a second change record that includes the system change value contained in said message;

said each of the one or more other primary database system instances sending the second change record to said one or more standby database systems.

39. The method of claim 38, wherein the one or more sequences of instructions, when executed by the one or more processors, further cause performing of:

a standby database system receiving change records from a plurality of primary database system instances;

determining a lowest system change value received from a last change record received from each primary database system instance of the plurality of primary database system instances; and

applying only change records containing a system change value less than or equal to the lowest system change value.

40. A non-transitory computer-readable medium storing one or more sequences of instructions which, when executed by one or more processors, cause performing of:

a primary storage system concurrently performing;

a) persistently storing change records for a set of changes; and

b) sending said change records to one or more standby storage systems;

said one or more standby storage systems receiving and persistently storing said change records;

after completing persistently storing change records for the set of changes, the primary storage system sending a notification to the one or more standby storage systems;

the one or more standby storage systems waiting to receive said notification before applying said set of changes.

41. The non-transitory computer-readable medium of claim 40, wherein said notification is indicated by receipt of a next set of change records received by said standby storage system.

42. The non-transitory computer-readable medium of claim 40, wherein said notification is sent independent of change records.

43. A non-transitory computer-readable medium storing one or more sequences of instructions which, when executed by one or more processors, cause performing of:

a broker receiving a query from an application, wherein the query is associated with a maximum tolerable lag time value;

selecting a first standby storage system of a plurality of standby storage systems, wherein the first standby storage system monitors data lag time, wherein the data lag time is an amount of time between when data is generated on a primary storage system and when the data is applied on the standby storage system;

in response to receiving an error indicating that the data lag time monitored by the first standby storage system is greater than the maximum tolerable lag time value, said broker re-directing said query to a second storage system.

44. The non-transitory computer-readable medium of claim 43, wherein the second storage system is a second standby storage system of the plurality of standby storage systems.

45. The non-transitory computer-readable medium of claim 43, wherein the second storage system is the primary storage system.

46. The non-transitory computer-readable medium of claim 43, wherein said broker selects the first standby storage system based on a guaranteed maximum lag time associated with a service published by the first standby storage system.