Efficient replication of system transactions for read-only nodes of a distributed database

• US 9,280,591 B1
• Filed: 09/20/2013
• Issued: 03/08/2016
• Est. Priority Date: 09/20/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kharatishvili, Tengiz, Madhavarapu, Pradeep Jnana, Gupta, Anurag Windlass
• Primary Examiner(s): Ho, Binh V
• Application Number:
  US14/033,343
• Time in Patent Office: 900 Days
• Field of Search: 707/624
• US Class Current: 1/1
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A distributed database system may efficiently replicate system transactions one or more read-only nodes. An update to a distributed database may be received. One or more system transactions may be performed to apply the update. For each system transaction, one or more change notifications may be generated which indicate changes to be applied in order to perform the system transaction. A particular one of the change notifications may be identified as the last change to be applied in order to complete the system transaction. The change notifications may be sent to one or more read-only nodes. The read-only nodes may process read requests for the distributed database system. The identified change notification may indicate to the read-only nodes the last change to be applied prior to presenting a state of the database that includes the system transaction when servicing read requests.

Claims

1. A system, comprising:

a plurality of nodes implemented by one or more computing devices, wherein the plurality of nodes implement a distributed database;

wherein one or more nodes of the plurality of nodes implement one or more read-only nodes, wherein the one or more read-only nodes operate as read replicas to process read operations for the distributed database;

wherein at least one other node, of the plurality of nodes implements a read-write node, wherein the read-write node processes read operations and write operations for the distributed database, and wherein the read-write node is configured to;

receive an update to the distributed database;

perform one or more system transactions as part of applying the update to the distributed database;

for each of the one or more system transactions;

generate a plurality of change notifications indicating changes to be applied in order to perform the system transaction;

identify a particular one of the plurality of change notifications as the last change to be applied in order to complete the system transaction; and

send the plurality of change notifications, including the identified particular change notification, to the one or more read-only nodes of the distributed database, wherein the particular change notification indicates to the one or more read-only nodes the last change to be applied prior to presenting a state of the database that includes the system transaction.

2. The system of claim 1, wherein each of the one or more read-only nodes is configured to:

receive the plurality of change notifications; and

apply the changes indicated in the plurality of change notifications.

3. The system of claim 2, wherein at least one of the plurality of change notifications comprises one or more changes to system metadata maintained by the read-write node, and wherein to apply the changes indicated in the plurality of change notifications, each of the one or more read-only nodes are further configured to modify system metadata for the read-only node to match the one or more indicated changes to the system metadata maintained at the read-write node.

4. The system of claim 2, wherein at least one of the plurality of change notifications comprises one or more changes to one or more data pages of a plurality of data pages maintaining data for the database at a distributed storage system, and wherein to apply the changes indicated in the plurality of change notifications, each of the one or more read-only nodes are further configured to invalidate a version of the one or more data pages maintained at the read-only node for servicing read requests.

5. The system of claim 2, wherein each of the one or more read-only nodes is further configured to:

receive a read request at the read-only node;

in response to receiving the read request;

determine a consistent state of the database to present for servicing the read request based, at least in part, on the particular one of the plurality of change notifications identified as the last change to be applied in order to complete the system transaction; and

service the read request based, at least in part, on the determined consistent state of the database.

6. A method, comprising:

performing, by one or more computing devices implementing a read-write node of a plurality of database nodes, wherein the plurality of database nodes implement a distributed database system;

generating a plurality of change notifications indicating changes to be applied in order to perform a system transaction at the read-write node, wherein the system transaction is one system transaction of one or more system transactions performed at the read-write node in order to apply an update received at the read-write node;

identifying a particular one of the plurality of change notifications as the last change to be applied in order to complete the system transaction; and

sending the plurality of change notifications, including the identified particular change notification, to one or more read-only nodes of the distributed database system, wherein the particular change notification indicates to the one or more read-only nodes of the distributed database system the last change to be applied prior to presenting a state of the database that includes the system transaction.

7. The method of claim 6, wherein at least one of the plurality of change notifications comprises one or more changes to a data page of a plurality of data pages maintaining data for the database at a data store.

8. The method of claim 7, wherein said generating the at least one change notification comprising one or more changes to the one or more data pages comprises generating a redo log record for each of the one or more changes to the one or more data pages.

9. The method of claim 6, wherein at least one of the plurality of change notifications comprises one or more changes to system metadata maintained by the read-write node, wherein the system metadata comprises:

metadata describing a database schema maintained at a data store for the database;

or

metadata describing in-flight transactions.

10. The method of claim 6, wherein said identifying the particular one of the plurality of change notifications as the last change to be applied in order to complete the system transaction comprises including a monotonically increasing identifier in the particular change notification.

11. The method of claim 6, further comprising:

performing, by one or more other computing devices implementing a read-only node of the one or more read-only nodes of the distributed database system;

receiving the plurality of change notifications; and

applying the changes indicated in the plurality of change notifications.

12. The method of claim 11, further comprising:

wherein said performing, by one or more other computing devices implementing a read-only node of the one or more read-only nodes of the distributed database system further comprises;

receiving a read request at the read-only node;

in response to receiving the read request;

determining a consistent state of the database to present for servicing the read request based, at least in part, on the particular one of the plurality of change notifications identified as the last change to be applied in order to complete the system transaction; and

servicing the read request based, at least in part, on the determined consistent state of the database, wherein at least one or more additional change notifications are received that are not identified as another last change to be applied.

13. The method of claim 12,

wherein a plurality of other change notifications including the different ones identified as the last change to be applied in order to complete a respective system transaction are received;

wherein said determining the consistent state of the database to present for servicing the read request comprises;

identifying one or more change notifications out of the plurality of other change notifications and the plurality of change notifications that are received prior to a received change notification identified as the last change to be applied;

wherein said servicing the read request based, at least in part, on the determined consistent state of the database, comprises;

including in a view of the database used to service the query the applied changes indicated in the identified one or more change notifications.

14. A non-transitory, computer-readable storage medium, storing program instructions that when executed by a plurality of computing devices cause the plurality of computing devices to implement:

receiving an update to a database maintained by a distributed database system, wherein the update is received at a read-write node of a plurality of nodes implementing the distributed database system;

performing one or more system transactions to apply the update to the distributed database system at the read-write node;

for each of the one or more system transactions;

generating at the read-write node a plurality of change notifications indicating changes to be applied in order to perform the system transaction;

identifying at the read-write node a particular one of the plurality of change notifications as the last change to be applied in order to complete the system transaction; and

sending the plurality of change notifications, including the identified particular change notification, to one or more read-only nodes of the plurality of nodes implementing the distributed database system, wherein the particular change notification indicates to the one or more read-only nodes the last change to be applied prior to presenting a state of the database that includes the system transaction.

15. The non-transitory, computer-readable storage medium of claim 14, wherein in said sending the plurality of change notifications, including the identified particular change notification, to the one or more read-only nodes of the plurality of nodes implementing the distributed database system, the program instructions cause the plurality of computing devices to further implement:

grouping the plurality of change notifications into one or more batches; and

sending the one or more batches to the one or more read-only nodes.

16. The non-transitory, computer-readable storage medium of claim 14, wherein at least one of the plurality of change notifications comprises one or more changes to system metadata maintained by the read-write node.

17. The non-transitory, computer-readable storage medium of claim 16, wherein the program instructions cause the plurality of computing devices to further implement:

receiving the plurality of change notifications at the one or more read-only nodes;

applying the changes indicated in the plurality of change notifications at the one or more read-only nodes;

receiving a read request at a read-only node of the one or more read only nodes;

in response to receiving the read request;

determining a consistent state of the database to present for servicing the read request based, at least in part, on the particular one of the plurality of change notifications identified as the last change to be applied in order to complete the system transaction; and

servicing the read request based, at least in part, on the determined consistent state of the database, wherein at least one or more additional change notifications are received that are not identified as another last change to be applied.

18. The non-transitory, computer-readable storage medium of claim 17,

wherein in said applying the changes indicated in the plurality of change notifications, the program instructions further cause the plurality of computing devices to implement:

modifying system metadata maintained at the read-only node to match the indicated one or more changes to the system metadata maintained at the read-write node;

wherein said servicing the read request is based, at least in part, on the modified system metadata maintained at the read-only node.

19. The non-transitory, computer-readable storage medium of claim 14, wherein at least one of the plurality of change notifications comprises one or more changes to one or more data pages of a plurality of data pages maintaining data for the database at a distributed storage system, and wherein each of the one or more read-only nodes and the read-write node may access the plurality of data pages stored for the database.

20. The non-transitory, computer-readable storage medium of claim 19, wherein the distributed database system is a network-based database service, wherein the distributed storage system is a multi-tenant, network-based distributed storage service.