Facilitating data migration between database clusters while the database continues operating

• US 9,298,752 B2
• Filed: 08/26/2013
• Issued: 03/29/2016
• Est. Priority Date: 08/26/2013
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Do, Cuong
• Primary Examiner(s): Coby, Frantz
• Application Number:
  US14/010,215
• Publication Number:
  US 20150058289A1
• Time in Patent Office: 946 Days
• Field of Search: 707/611, 707/658, 707/825, 707/782, 707/655, 707/661, 707/657, 711/165, 711/161, 711/162, 711/112, 711/114
• US Class Current: 1/1
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/2386 Bulk updating operations data conversion details G06F16/258
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 9/4856 resumption being on a different machine, e.g. task migration, virtual machine migration G06F9/5088 takes precedence

Abstract

A system for migrating data items from a source cluster to a destination cluster in a database system. During operation, while the database continues to process live database traffic, the system records a current position in an operation log for the database, wherein the operation log comprises a sequential record of operations applied to the database. Next, the system retrieves data items from the source cluster, and inserts the copies of the retrieved data items into the destination cluster. The system also applies a sequence of updates, starting from the recorded position in the operation log, to the copies of the data items in the destination cluster. Finally, after the sequence of updates is applied, the system performs a cut-over operation that diverts the live database traffic from the data items on the source cluster to the copies of the data items on the destination cluster.

Claims

1. A method, comprising:

while a database continues to process live database traffic, performing the following operations;

receiving a request to migrate a set of data items from a source cluster in the database to a destination cluster in the database;

recording a current position in an operation log for the database, wherein the operation log records a sequential record of operations that are applied to the source cluster subsequent to receiving the migration request;

copying the set of data items from the source cluster to the destination cluster in parallel with continuing to process live database traffic for the set of data items at the source cluster;

applying a sequence of updates, starting from the recorded position in the operation log, to the set of data items that was copied to the destination cluster; and

after the sequence of updates is applied, performing a cut-over operation that diverts the live database traffic for the set of data items to the destination cluster, wherein prior to the cut-over operation the source cluster continues to directly receive and handle requests for the set of data items, wherein subsequent to the cut-over operation live database traffic for the set of data items is diverted to the destination cluster.

2. The method of claim 1,

wherein the set of data items that are migrated to the destination cluster comprise a subset of the data items in the source cluster;

wherein subsequent to the cut-over operation the source cluster continues to process live database traffic for a distinct set of un-migrated data items that are stored in the source cluster;

wherein processing live database traffic for the database in parallel on both the source cluster and the destination cluster enables the database to process a larger volume of requests.

3. The method of claim 1, wherein applying the sequence of updates includes applying the sequence of updates asynchronously by:

tracking in-progress updates to the set of data items that have been copied to the destination cluster, wherein the in-progress updates are sequential updates from the operation log that are being applied to the copied set of data items on the destination cluster but have not yet completed;

applying the sequence of updates asynchronously on the destination cluster without waiting for preceding updates to complete; and

upon detecting that a pending update applies to a same data item as an in-progress update, pausing application of the pending and subsequent updates until the in-progress update completes.

4. The method of claim 1, further comprising performing a verification operation to verify that the data items have been successfully migrated from the source cluster to the destination cluster, wherein the verification operation comprises comparing data items that were updated on the source cluster while the set of data items were being copied with copies of the data items on the destination cluster to ensure that the changes in the operation log were applied to the copy of the set of data items on the destination cluster.

5. The method of claim 4,

wherein the verification operation takes place in parallel with applying the sequence of updates to the copied set of data items in the destination cluster;

and

wherein ensuring that the copied set of data items matches the original data items comprises keeping track of failed matches, and periodically retrying comparisons for the failed matches.

6. The method of claim 1, wherein the method further comprises copying one or more indices, which are used to access fields in the data items, from the source cluster to the destination cluster.

7. The method of claim 1,

wherein the database includes one or more replica set primaries that accept both read and write operations;

wherein each replica set primary is associated with one or more replica set secondaries that replicate a primary'"'"'s data set and only accept read operations; and

wherein copying the data items from the source cluster comprises copying the data items in parallel from a set of two or more replica set secondaries that all store the set of data items to the destination cluster, wherein using the set of replica set secondaries to copy the set of data items enables the corresponding replica set primaries to service other requests for the set of data items while simultaneously performing migration operations in parallel via the multiple replica set secondaries.

8. The method of claim 1, wherein an update can include one or more of:

an insertion of a data item;

an update to a data item; and

a deletion of a data item.

9. A system, comprising:

a database server within a database; and

a migration module within the database server, wherein the migration module includes a memory storing instructions for controlling a processor to perform a migration operation, the migration operation comprising;

while the database continues to process live database traffic performing the following operations;

receiving a request to migrate a set of data items from a source cluster in the database to a destination cluster in the database;

recording a current position in an operation log for the database, wherein the operation log records a sequential record of operations that are applied to the source cluster subsequent to receiving the migration request;

copying the set of data items from the source cluster to the destination cluster in parallel with continuing to process live database traffic for the set of data items at the source cluster;

applying a sequence of updates, starting from the recorded position in the operation log, to the set of data items that was copied to the destination cluster; and

after the sequence of updates is applied, performing a cut-over operation that diverts the live database traffic for the set of data items to the destination cluster, wherein prior to the cut-over operation the source cluster continues to directly receive and handle requests for the set of data items, wherein subsequent to the cut-over operation live database traffic for the set of data items is diverted to the destination cluster.

10. The system of claim 9,

wherein the set of data items that are migrated to the destination cluster comprise a subset of the data items in the source cluster;

wherein subsequent to the cut-over operation the source cluster continues to process live database traffic for a distinct set of un-migrated data items that are stored in the source cluster;

wherein processing live database traffic for the database in parallel on both the source cluster and the destination cluster enables the database to process a larger volume of requests.

11. The system of claim 9, wherein the migration module is configured to apply the sequence of updates asynchronously by:

tracking in-progress updates to the set of data items that have been copied to the destination cluster, wherein the in-progress updates are sequential updates from the operation log that are being applied to the copied set of data items on the destination cluster but have not yet completed;

applying the sequence of updates asynchronously on the destination cluster without waiting for preceding updates to complete; and

upon detecting that a pending update applies to a same data item as an in-progress update, pausing application of the pending and subsequent updates until the in-progress update completes.

12. The system of claim 9, wherein the migration module is additionally configured to perform a verification operation to verify that the data items have been successfully migrated from the source cluster to the destination cluster, wherein the verification operation comprises comparing data items that were updated on the source cluster while the set of data items were being copied with copies of the data items on the destination cluster to ensure that the changes in the operation log were applied to the copy of the set of data items on the destination cluster.

13. The system of claim 12,

wherein the migration module is configured to perform the verification operation in parallel with applying the sequence of updates to the copied set of data items in the destination cluster;

and

wherein while ensuring that the copied set of data items matches the original data items, the migration module is configured to keep track of failed matches, and periodically retry comparisons for the failed matches.

14. The system of claim 9,

wherein the database includes one or more replica set primaries that accept both read and write operations;

wherein each replica set primary is associated with one or more replica set secondaries that replicate a primary'"'"'s data set and only accept read operations; and

wherein while copying the data items from the source cluster, the migration module is configured to copy the data items in parallel from a set of two or more replica set secondaries that all store the set of data items to the destination cluster, wherein using the set of replica set secondaries to copy the set of data items enables the corresponding replica set primaries to service other requests for the set of data items while simultaneously performing migration operations in parallel via the multiple replica set secondaries.

15. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, comprising:

while a database continues to process live database traffic, performing the following operations;

receiving a request to migrate a set of data items from a source cluster in the database to a destination cluster in the database;

recording a current position in an operation log for the database, wherein the operation log records a sequential record of operations that are applied to the source cluster subsequent to receiving the migration request;

copying the set of data items from the source cluster to the destination cluster in parallel with continuing to process live database traffic for the set of data items at the source cluster;

applying a sequence of updates, starting from the recorded position in the operation log, to the set of data items that was copied to the destination cluster; and

after the sequence of updates is applied, performing a cut-over operation that diverts the live database traffic for the set of data items to the destination cluster, wherein prior to the cut-over operation the source cluster continues to directly receive and handle requests for the set of data items, wherein subsequent to the cut-over operation live database traffic for the set of data items is diverted to the destination cluster.

16. The non-transitory computer-readable storage medium of claim 15,

wherein the set of data items that are migrated to the destination cluster comprise a subset of the data items in the source cluster;

wherein subsequent to the cut-over operation the source cluster continues to process live database traffic for a distinct set of un-migrated data items that are stored in the source cluster;

wherein processing live database traffic for the database in parallel on both the source cluster and the destination cluster enables the database to process a larger volume of requests.

17. The non-transitory computer-readable storage medium of claim 15, wherein applying the sequence of updates includes applying the sequence of updates asynchronously by:

tracking in-progress updates to the set of data items that have been copied to the destination cluster, wherein the in-progress updates are sequential updates from the operation log that are being applied to the copied set of data items on the destination cluster but have not yet completed;

applying the sequence of updates asynchronously on the destination cluster without waiting for preceding updates to complete; and

upon detecting that a pending update applies to a same data item as an in-progress update, pausing application of the pending and subsequent updates until the in-progress update completes.

18. The non-transitory computer-readable storage medium of claim 15, further comprising performing a verification operation to verify that the data items have been successfully migrated from the source cluster to the destination cluster, wherein the verification operation comprises comparing data items that were updated on the source cluster while the set of data items were being copied with copies of the data items on the destination cluster to ensure that the changes in the operation log were applied to the copy of the set of data items on the destination cluster.

19. The non-transitory computer-readable storage medium of claim 18,

wherein the verification operation takes place in parallel with applying the sequence of updates to the copied set of data items in the destination cluster;

and

wherein ensuring that the copied set of data items matches the original data items comprises keeping track of failed matches, and periodically retrying comparisons for the failed matches.

20. The non-transitory computer-readable storage medium of claim 15,

wherein the database includes one or more replica set primaries that accept both read and write operations;

wherein each replica set primary is associated with one or more replica set secondaries that replicate a primary'"'"'s data set and only accept read operations; and

wherein copying the data items from the source cluster comprises copying the data items in parallel from a set of two or more replica set secondaries that all store the set of data items to the destination cluster, wherein using the set of replica set secondaries to copy the set of data items enables the corresponding replica set primaries to service other requests for the set of data items while simultaneously performing migration operations in parallel via the multiple replica set secondaries.