System and methods for live data migration

• US 10,452,626 B2
• Filed: 09/29/2017
• Issued: 10/22/2019
• Est. Priority Date: 11/10/2016
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Ellis, Joseph, Fink, Robert, Elliot, Mark, Van Wassenhove, Sandor
• Primary Examiner(s): Arjomandi, Noosha
• Application Number:
  US15/720,766
• Publication Number:
  US 20180129654A1
• Time in Patent Office: 753 Days
• CPC Class Codes: G06F 16/213 with details for schema evolution support
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/245 Query processing

Abstract

Systems and methods are provided for data migration. The system may comprise one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the system to migrate at least one first table of a first database schema to at least one second table of a second database schema, determine a query for modifying the first table during the migration, modify the second table based at least in part on the query, and update a mutation table to describe the modification, wherein the mutation table at least describes the modification.

Claims

1. A system for live data migration, the system comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to perform;

receiving a first query requesting a modification of a first table of a first database schema during a live migration of the first table to a second table of a second database schema, the first table storing live data;

modifying, based on the first query, a first row of a diff table corresponding to a first row of the first table;

updating a mutation table to indicate the requested modification of the first table, wherein the mutation table tracks one or more requested modifications to the first table;

receiving a second query requesting a read of a particular row of the first table during the live migration of the first table to the second table;

determining, in response to receiving the second query, whether the mutation table indicates a previous query requested modification of the particular row of the first table, the previous query being previous to at least the second query;

if the mutation table indicates the previous query requested modification of the particular row of the first table;

providing data of a particular row of the diff table corresponding to the particular row of the first table in response to the second query; and

if the mutation table does not indicate the previous query requested modification of the particular row of the first table;

providing the live data of the particular row of the first table in response to the second query.

2. The system of claim 1, wherein the diff table comprises a same database schema as the second database schema.

3. The system of claim 1, wherein the first row of the diff table is modified without modifying the first row of the first table.

4. The system of claim 1, wherein the first table becomes immutable during the live migration.

5. The system of claim 1, wherein the mutation table indicates the requested modification of the first table by setting a Boolean value for a row of the mutation table corresponding to the first row the first table.

6. The system of claim 1, wherein the first query is received from a first client device over a communication network, and the second query is received from a second client device over the communication network.

7. The system of claim 1, wherein the instructions further cause the system to perform:

updating, in response to receiving the first query, a commit log to include a first timestamp associated with the first query, the commit log including one or more second timestamps associated with one or more previous queries requesting modification of the first table;

migrating data of one or more rows of the diff table corresponding to one or more rows of the first table to the second table, the data provided in a chronological order based on the first and second timestamps of the commit log.

8. The system of claim 7, wherein one or more rows of the first table are identified based on the mutation table.

9. The system of claim 7, wherein the migrating data of one or more rows of the diff table corresponding to the one or more rows of the first table to the second table is performed subsequent to the live migration of the first table to the second table.

10. The system of claim 1, wherein the particular row of the first table comprises the first row of the first table, and the particular row of the diff table comprises the first row of the diff table.

11. A computer-implemented method, the method being implemented by a computing system including one or more processors and storage media storing machine-readable instructions, the method comprising:

receiving a first query requesting a modification of a first table of a first database schema during a live migration of the first table to a second table of a second database schema, the first table storing live data;

modifying, based on the first query, a first row of a diff table corresponding to a first row of the first table;

updating a mutation table to indicate the requested modification of the first table, wherein the mutation table tracks one or more requested modifications to the first table;

receiving a second query requesting a read of a particular row of the first table during the live migration of the first table to the second table;

determining, in response to receiving the second query, whether the mutation table indicates a previous query requested modification of the particular row of the first table, the previous query being previous to at least the second query;

if the mutation table indicates the previous query requested modification of the particular row of the first table;

providing data of a particular row of the diff table corresponding to the particular row of the first table in response to the second query; and

if the mutation table does not indicate the previous query requested modification of the particular row of the first table;

providing the live data of the particular row of the first table in response to the second query.

12. The computer-implemented method of claim 11, wherein the diff table comprises a same database schema as the second database schema.

13. The computer-implemented method of claim 11, wherein the first row of the diff table is modified without modifying the first row of the first table.

14. The computer-implemented method of claim 11, wherein the first table becomes immutable during the live migration.

15. The computer-implemented method of claim 11, wherein the mutation table indicates the requested modification of the first table by setting a Boolean value for a row of the mutation table corresponding to the first row the first table.

16. The computer-implemented method of claim 11, wherein the first query is received from a first client device over a communication network, and the second query is received from a second client device over the communication network.

17. The computer-implemented method of claim 11, further comprising:

updating, in response to receiving the first query, a commit log to include a first timestamp associated with the first query, the commit log including one or more second timestamps associated with one or more previous queries requesting modification of the first table;

migrating data of one or more rows of the diff table corresponding to one or more rows of the first table to the second table, the data provided in a chronological order based on the first and second timestamps of the commit log.

18. The computer-implemented method of claim 17, wherein one or more rows of the first table are identified based on the mutation table.

19. The computer-implemented method of claim 11, wherein the first database schema is different from the second database schema.

20. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:

receiving a first query requesting a modification of a first table of a first database schema during a live migration of the first table to a second table of a second database schema, the first table storing live data;

modifying, based on the first query, a first row of a diff table corresponding to a first row of the first table;

updating a mutation table to indicate the requested modification of the first table, wherein the mutation table tracks one or more requested modifications to the first table;

receiving a second query requesting a read of a particular row of the first table during the live migration of the first table to the second table;

determining, in response to receiving the second query, whether the mutation table indicates a previous query requested modification of the particular row of the first table,. the previous query being previous to at least the second query;

if the mutation table indicates the previous query requested modification of the particular row of the first table;

providing data of a particular row of the diff table corresponding to the particular row of the first table in response to the second query; and

if the mutation table does not indicate the previous query requested modification of the particular row of the first table;

providing the live data of the particular row of the first table in response to the second query.