FACILITATING DATA MIGRATION BETWEEN DATABASE CLUSTERS WHILE THE DATABASE CONTINUES OPERATING

• US 20150058289A1
• Filed: 08/26/2013
• Published: 02/26/2015
• Est. Priority Date: 08/26/2013
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Do, Cuong
• Granted Patent: US 9,298,752 B2
• Time in Patent Office: Days
• US Class Current: 707/613
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/2386 Bulk updating operations data conversion details G06F16/258
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 9/4856 resumption being on a different machine, e.g. task migration, virtual machine migration G06F9/5088 takes precedence

Abstract

A system for migrating data items from a source cluster to a destination cluster in a database system. During operation, while the database continues to process live database traffic, the system records a current position in an operation log for the database, wherein the operation log comprises a sequential record of operations applied to the database. Next, the system retrieves data items from the source cluster, and inserts the copies of the retrieved data items into the destination cluster. The system also applies a sequence of updates, starting from the recorded position in the operation log, to the copies of the data items in the destination cluster. Finally, after the sequence of updates is applied, the system performs a cut-over operation that diverts the live database traffic from the data items on the source cluster to the copies of the data items on the destination cluster.

Claims

1. A method, comprising:

while a database continues to process live database traffic, performing the following operations;

recording a current position in an operation log for the database, wherein the operation log contains a sequential record of operations applied to the database;

retrieving data items from a source cluster in the database;

inserting the copies of the retrieved data items into a destination cluster in the database;

applying a sequence of updates, starting from the recorded position in the operation log, to the copies of the data items in the destination cluster; and

after the sequence of updates is applied, performing a cut-over operation that diverts the live database traffic from the data items on the source cluster to the copies of the data items on the destination cluster.

2. The method of claim 1, further comprising using multiple threads to perform the migration operation for respective subsets of the data items in parallel.

3. The method of claim 1, wherein applying the sequence of updates includes applying the sequence of updates asynchronously by:

keeping track of in-progress updates to copies of the data items in the destination cluster, wherein the in-progress updates are updates that have started but are not yet completed;

applying the sequence of updates asynchronously without waiting for preceding updates to complete; and

if an update-to-be-applied is directed to a same data item as an in-progress update, pausing application of the update-to-be-applied and subsequent updates until the in-progress update completes.

4. The method of claim 1, further comprising performing a verification operation to verify that the data items have been successfully migrated from the source cluster to the destination cluster, wherein the verification operation comprises comparing original data items on the source cluster with copies of the data items on the destination cluster to ensure that the copies of the data items match the original data items.

5. The method of claim 4,

wherein the verification operation takes place in parallel with applying the sequence of updates to the copies of the data items in the destination cluster;

and

wherein ensuring that the copies of the data items match the original data items comprises keeping track of failed matches, and periodically retrying comparisons for the failed matches.

6. The method of claim 1, wherein the method further comprises copying one or more indices, which are used to access fields in the data items, from the source cluster to the destination cluster.

7. The method of claim 1,

wherein the database includes one or more replica set primaries that accept both read and write operations;

wherein each replica set primary is associated with one or more replica set secondaries that replicate a primary'"'"'s data set and only accept read operations; and

wherein retrieving the data items from the source cluster comprises retrieving the data items from the replica set secondaries, so that the corresponding replica set primaries can continue to service requests, and also to facilitate performing migration operations in parallel by accessing multiple replica set secondaries simultaneously.

8. The method of claim 1, wherein an update can include one or more of:

an insertion of a data item;

an update to a data item; and

a deletion of a data item.

9. A system, comprising:

a database server within a database; and

a migration module within the database server, wherein the migration module includes a memory storing instructions for controlling a processor to perform a migration operation, the migration operation comprising;

while the database continues to process live database traffic performing the following operations;

recording a current position in an operation log for the database, wherein the operation log contains a sequential record of operations applied to the database;

retrieving data items from a source cluster in the database;

inserting the copies of the retrieved data items into a destination cluster in the database;

applying a sequence of updates, starting from the recorded position in the operation log, to the copies of the data items in the destination cluster; and

after the sequence of updates is applied, performing a cut-over operation that diverts the live database traffic from the data items on the source cluster to the copies of the data items on the destination cluster.

10. The system of claim 9, wherein the migration module is configured to use multiple threads to perform the migration operation for respective subsets of the data items in parallel.

11. The system of claim 9, wherein the migration module is configured to apply the sequence of updates asynchronously by:

keeping track of in-progress updates to copies of the data items in the destination cluster, wherein the in-progress updates are updates that have started but are not yet completed;

applying the sequence of updates asynchronously without waiting for preceding updates to complete; and

if an update-to-be-applied is directed to a same data item as an in-progress update, pausing application of the update-to-be-applied and subsequent updates until the in-progress update completes.

12. The system of claim 9, wherein the migration module is additionally configured to perform a verification operation to verify that the data items have been successfully migrated from the source cluster to the destination cluster, wherein the verification operation comprises comparing original data items on the source cluster with copies of the data items on the destination cluster to ensure that the copies of the data items match the original data items.

13. The system of claim 12,

wherein the migration module is configured to perform the verification operation in parallel with applying the sequence of updates to the copies of the data items in the destination cluster;

and

wherein while ensuring that the copies of the data items match the original data items, the migration module is configured to keep track of failed matches, and periodically retry comparisons for the failed matches.

14. The system of claim 9,

wherein the database includes one or more replica set primaries that accept both read and write operations;

wherein each replica set primary is associated with one or more replica set secondaries that replicate a primary'"'"'s data set and only accept read operations; and

wherein while retrieving the data items from the source cluster, the migration module is configured to retrieve the data items from the replica set secondaries, so that the corresponding replica set primaries can continue to service requests, and also to facilitate performing migration operations in parallel by accessing multiple replica set secondaries simultaneously.

15. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, comprising:

while a database continues to process live database traffic, performing the following operations;

recording a current position in an operation log for the database, wherein the operation log contains a sequential record of operations applied to the database;

retrieving data items from a source cluster in the database;

inserting the copies of the retrieved data items into a destination cluster in the database;

applying a sequence of updates, starting from the recorded position in the operation log, to the copies of the data items in the destination cluster; and

after the sequence of updates is applied, performing a cut-over operation that diverts the live database traffic from the data items on the source cluster to the copies of the data items on the destination cluster.

16. The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises using multiple threads to perform the migration operation for respective subsets of the data items in parallel.

17. The non-transitory computer-readable storage medium of claim 15, wherein applying the sequence of updates includes applying the sequence of updates asynchronously by:

keeping track of in-progress updates to copies of the data items in the destination cluster, wherein the in-progress updates are updates that have started but are not yet completed;

applying the sequence of updates asynchronously without waiting for preceding updates to complete; and

if an update-to-be-applied is directed to a same data item as an in-progress update, pausing application of the update-to-be-applied and subsequent updates until the in-progress update completes.

18. The non-transitory computer-readable storage medium of claim 15, further comprising performing a verification operation to verify that the data items have been successfully migrated from the source cluster to the destination cluster, wherein the verification operation comprises comparing original data items on the source cluster with copies of the data items on the destination cluster to ensure that the copies of the data items match the original data items.

19. The non-transitory computer-readable storage medium of claim 18,

wherein the verification operation takes place in parallel with applying the sequence of updates to the copies of the data items in the destination cluster;

and

wherein ensuring that the copies of the data items match the original data items comprises keeping track of failed matches, and periodically retrying comparisons for the failed matches.

20. The non-transitory computer-readable storage medium of claim 15,

wherein the database includes one or more replica set primaries that accept both read and write operations;

wherein each replica set primary is associated with one or more replica set secondaries that replicate a primary'"'"'s data set and only accept read operations; and

wherein retrieving the data items from the source cluster comprises retrieving the data items from the replica set secondaries, so that the corresponding replica set primaries can continue to service requests, and also to facilitate performing migration operations in parallel by accessing multiple replica set secondaries simultaneously.