System and methods for live data migration

• US 9,805,071 B1
• Filed: 03/30/2017
• Issued: 10/31/2017
• Est. Priority Date: 11/10/2016
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Ellis, Joseph, Elliot, Mark, Fink, Robert, Van Wassenhove, Sandor
• Primary Examiner(s): Arjomandi, Noosha
• Application Number:
  US15/474,713
• Time in Patent Office: 215 Days
• CPC Class Codes: G06F 16/213 with details for schema evolution support
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/245 Query processing

Abstract

Systems and methods are provided for data migration. The system may comprise one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the system to migrate at least one first table of a first database schema to at least one second table of a second database schema, determine a query for modifying the first table during the migration, modify the second table based at least in part on the query, and update a mutation table to describe the modification, wherein the mutation table at least describes the modification.

Claims

1. A system for live data migration, the system comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to perform;

migrating at least one first table of a first database schema to at least one second table of a second database schema, the at least one first table storing live data;

determining a query for modifying the at least one first table during the migration;

modifying the at least one second table based at least in part on the query;

updating a mutation table to describe the modification, wherein the mutation table tracks one or more changes to the at least one first table;

determining at least one field to be migrated from the at least one first table to the at least one second table;

determining that no write operations have been performed to a row corresponding to the at least one field during the migration, the determining based at least in part on the mutation table; and

migrating the at least one field from the at least one first table to the at least one second table, the migrating causing data corresponding to the at least one field from the first table to be populated in one or more fields of the second table based at least in part on a respective schema of the second table.

2. The system of claim 1, wherein the query modifies the at least one field in the at least one first table, wherein the instructions further cause the system to perform:

determining a second query for accessing the at least one field during the migration;

determining that the at least one field has been modified based at least in part on the mutation table; and

providing data corresponding to the field from the second table in response to the second query.

3. The system of claim 1, wherein the query modifies the at least one field in the first table, wherein the instructions further cause the system to perform:

determining a second query for accessing the at least one field during the migration;

determining that the at least one field has not been modified based at least in part on the mutation table; and

providing the data corresponding to the field from the first table in response to the second query.

4. The system of claim 1, wherein the modifying the first table includes at least one of modifying data, inserting data, or deleting data in the first table.

5. The system of claim 1, wherein the updating the mutation table further comprises:

determining a row in the second table to which the modification was performed; and

updating the mutation table to indicate that the row has been modified.

6. The system of claim 5, wherein the mutation table is updated to indicate the row was modified by updating a Boolean value that corresponds to the row.

7. The system of claim 1, wherein the query deletes at least one row in the first table, wherein the instructions further cause the system to perform:

determining a second query for accessing the at least one row during the migration;

determining that the at least one row has been modified based at least in part on the mutation table;

determining that the row has been deleted from the second table; and

providing a null value in response to the second query.

8. A computer-implemented method, the method being implemented by a computing system including one or more processors and storage media storing machine-readable instructions, the method comprising:

migrating at least one first table of a first database schema to at least one second table of a second database schema, the at least one first table storing live data;

determining a query for modifying the at least one first table during the migration;

modifying the at least one second table based at least in part on the query;

updating a mutation table to describe the modification, wherein the mutation table tracks one or more changes to the at least one first table;

determining at least one field to be migrated from the at least one first table to the at least one second table;

determining that no write operations have been performed to a row corresponding to the at least one field during the migration, the determining based at least in part on the mutation table; and

migrating the at least one field from the at least one first table to the at least one second table, the migrating causing data corresponding to the at least one field from the first table to be populated in one or more fields of the second table based at least in part on a respective schema of the second table.

9. The computer-implemented method of claim 8, wherein the query modifies the at least one field in the at least one first table, the method further comprising:

determining a second query for accessing the at least one field during the migration;

determining that the at least one field has been modified based at least in part on the mutation table; and

providing data corresponding to the field from the second table in response to the second query.

10. The computer-implemented method of claim 8, wherein the query modifies the at least one field in the first table, the method further comprising:

determining a second query for accessing the at least one field during the migration;

determining that the at least one field has not been modified based at least in part on the mutation table; and

providing the data corresponding to the field from the first table in response to the second query.

11. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:

migrating at least one first table of a first database schema to at least one second table of a second database schema, the at least one first table storing live data;

determining a query for modifying the at least one first table during the migration;

modifying the at least one second table based at least in part on the query;

updating a mutation table to describe the modification, wherein the mutation table tracks one or more changes to the at least one first table;

determining at least one field to be migrated from the at least one first table to the at least one second table;

determining that no write operations have been performed to a row corresponding to the at least one field during the migration, the determining based at least in part on the mutation table; and

migrating the at least one field from the at least one first table to the at least one second table, the migrating causing data corresponding to the at least one field from the first table to be populated in one or more fields of the second table based at least in part on a respective schema of the second table.

12. The non-transitory computer readable medium of claim 11, wherein the query modifies the at least one field in the at least one first table, wherein the instructions further cause the processors to perform:

determining a second query for accessing the at least one field during the migration;

determining that the at least one field has been modified based at least in part on the mutation table; and

providing data corresponding to the field from the second table in response to the second query.

13. The non-transitory computer readable medium of claim 11, wherein the query modifies the at least one field in the at least one first table, wherein the instructions further cause the processors to perform:

determining a second query for accessing the at least one field during the migration;

determining that the at least one field has not been modified based at least in part on the mutation table; and

providing the data corresponding to the field from the first table in response to the second query.