Transformative migration of static data

• US 9,582,524 B1
• Filed: 06/19/2012
• Issued: 02/28/2017
• Est. Priority Date: 06/19/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ahearn, Thomas J., Davison, Alan G., King, Brian P., Ranaweera, Isuru, Yakovlev, Stepan P., Murali, Srikanth, Char, Hanson, Xu, Zhengqiang
• Primary Examiner(s): Le, Miranda
• Application Number:
  US13/526,843
• Time in Patent Office: 1,715 Days
• Field of Search: 707/626, 707/999.1
• US Class Current: 1/1
• CPC Class Codes: G06F 16/214 Database migration support

Abstract

Techniques are described for migrating data from a first table to a second table while transforming at least one characteristic of the data. Characteristics transformed may include one or more of an encryption key or method, a table schema, a data structure, or a storage infrastructure. Migration may be performed in two or more phases. A first phase may migrate and transform a majority of the data, such as data that is older than a certain threshold age. A second phase may migrate and transform the remaining, newer data. A status table may indicate which tables are active and are to be written to and read from during data writing and data consuming processes. Moreover, embodiments support data replication across multiple regions.

Claims

1. A computer-implemented method comprising:

identifying, by one or more computing devices including a processing unit and memory, a first database table with a first characteristic;

creating, by the one or more computing devices, a second database table with a second characteristic that is different than the first characteristic in at least one of a table schema, an encryption key, or a data structure;

creating, by the one or more computing devices, a primary key index for the second database table;

creating or updating, by the one or more computing devices, a status table prior to a first phase of data migration to indicate that, during the first phase of data migration, the first database table is accessible to at least one data reading process and the first database table is accessible to at least one data writing process, wherein information included in the first database table and information included in the second database table are not included in the status table;

migrating, by the one or more computing devices and during the first phase of data migration, a first portion of data stored in the first database table to the second database table while transforming the first portion of data from the first characteristic to the second characteristic, the first portion of data being at least as old as a threshold age;

creating, by the one or more computing devices, one or more other indices for the second database table;

updating, by the one or more computing devices, the status table prior to a second phase of data migration to indicate that, during the second phase of data migration, the first database table is accessible to the at least one data writing process and to the at least one data reading process;

the second database table is not accessible to the at least one data reading process; and

the second database table is accessible to the at least one data writing process; and

migrating, by the one or more computing devices and during the second phase of data migration, a second portion of data stored in the first database table to the second database table while transforming the second portion of data from the first characteristic to the second characteristic, the second portion of data being more recent than the threshold age.

2. The method of claim 1 further comprising deleting all data from the first database table after completing migration of the second portion of data.

3. The method of claim 1 wherein the first database table stores sensitive data.

4. The method of claim 3 wherein the sensitive data includes at least one of credit card data or financial institution account data.

5. The method of claim 1 wherein the first database table stores data that is at least in part static data.

6. A system comprising:

at least one processor;

at least one data storage medium, storing;

a first table with a first characteristic;

a second table with a second characteristic that is different from the first characteristic; and

a status table indicating which of the first table and the second table are available for access by at least one writing process and at least one reading process, wherein information included in the first table and information included in the second table are not included in the status table; and

a data migration component executed by the at least one processor to;

update the status table to indicate that, during a first phase of data migration, the first table is accessible to the at least one reading process and the first table is accessible to the at least one writing process;

migrate, during the first phase of data migration, a first data portion stored in the first table to the second table while transforming the first data portion from the first characteristic to the second characteristic;

update the status table to indicate that, during a second phase of data migration, the first table is accessible to the at least one reading process and to the at least one writing process, the second table is not accessible to the at least one reading process, and the second table is accessible to the at least one writing process; and

migrate, during the second phase of data migration, a second data portion stored in the first table to the second table while transforming the second data portion to the second characteristic, the second data portion being more recent than a threshold age.

7. The system of claim 6 wherein the first table and the second table are associated with a table category for sensitive data.

8. The system of claim 7 wherein the sensitive data includes at least one of credit card data or financial institution account data.

9. The system of claim 6 wherein the first data portion is at least as old as the threshold age, and wherein the second data portion is more recent than the threshold age.

10. The system of claim 6 wherein the threshold age is one day.

11. The system of claim 6 wherein the status table is one of a plurality of status tables and individual status tables of the plurality of status tables are related to a respective region of data to be migrated.

12. The system of claim 6 wherein the status table is updated to indicate that the first table is not accessible to the at least one writing process a predetermined delay time after updating the status table to indicate that the second table is accessible to the at least one reading process.

13. The system of claim 6 wherein the first characteristic differs from the second characteristic in one or more of a table schema, an encryption, a data structure, or a storage infrastructure.

14. A computer-implemented method comprising:

updating, by one or more computing devices including a processing unit and memory, a status table to indicate that, during a first phase of data migration, a first database table is accessible to at least one data reading process and to at least one data writing process, wherein information included in the first database table and information included in a second database table are not included in the status table;

migrating, by the one or more computing devices and during the first phase of data migration, a first data portion stored in the first database table with a first characteristic to the second database table with a second characteristic that is different from the first characteristic, while transforming the first data portion from the first characteristic to the second characteristic;

updating, by the one or more computing devices, the status table to indicate that, during a second phase of data migration, the first database table is accessible to the at least one data reading process and to the at least one data writing process, the second database table is not accessible to the at least one data reading process, and the second database table is accessible to the at least one data writing process; and

migrating, by the one or more computing devices, a second data portion stored in the first database table to the second database table while transforming the second data portion from the first characteristic to the second characteristic, the second data portion being more recent than a threshold age.

15. The method of claim 14 wherein the first characteristic differs from the second characteristic in one or more of a table schema, an encryption, a data structure, or a storage infrastructure.

16. The method of claim 14 wherein:

the first database table and the second database table are associated with a first region;

the first database table is replicated by a third database table with the first characteristic, the third database table being associated with a second region; and

the second database table is replicated by a fourth database table with the second characteristic, the fourth database table being associated with the second region.

17. The method of claim 16 further comprising:

independently migrating a replicated first data portion stored in the third database table to the fourth database table, while transforming the replicated first data portion from the first characteristic to the second characteristic; and

prior to migrating the second data portion, initiating a replication process that replicates the second database table with the fourth database table, such that the second data portion is replicated to the fourth database table concurrently with the migrating of the second data portion to the second database table.

18. The method of claim 14 wherein the first data portion is at least as old as the threshold age, and wherein the second data portion is more recent than the threshold age.

19. The method of claim 14 further comprising disabling replication of a first change to the first database in the second database during the first phase of data migration and enabling replication of a second change to the first database in the second database during the second phase of data migration.

20. The method of claim 14 wherein at least one of the first database table and the second database table stores encrypted data.

21. The method of claim 14 wherein at least one of the first database table and the second database table stores encrypted data that is encrypted using a Hash-Based Message Authentication Code (HMAC) based technique.

22. The method of claim 14 wherein the first characteristic differs from the second characteristic in that the second characteristic includes a different encryption technique from the first characteristic and in that the second characteristic includes a table schema with an additional column than a table schema of the first characteristic, the additional column associated with the different encryption technique.

23. The method of claim 22 wherein the different encryption technique is a HMAC based technique.

24. One or more non-transitory computer-readable storage media storing instructions that, when executed, instruct at least one processor to perform actions comprising:

updating a status table to indicate that, during a first phase of data migration, a first database table is accessible to at least one data writing process and at least one data reading process, wherein information included in the first database table and information included in a second database table are not included in the status table;

migrating, during the first phase of data migration, a first data portion stored in the first database table with a first characteristic to the second database table with a second characteristic that is different from the first characteristic, while transforming the first data portion from the first characteristic to the second characteristic, the first characteristic differing from the second characteristic in at least one of a table schema, an encryption key, or a data structure;

updating the status table to indicate that, during a second phase of data migration, the first database table is accessible to the at least one data writing process and to the at least one data reading process, the second database table is accessible to the at least one data writing process, and the second database table is not accessible to the at least one data reading process; and

migrating, during the second phase of data migration, a second data portion stored in the first database table to the second database table while transforming the second data portion from the first characteristic to the second characteristic, the second data portion being more recent than a threshold age.

25. The one or more non-transitory computer-readable storage media of claim 24, wherein the first database table and the second database table store sensitive data including at least one of credit card data or financial institution account data.

26. The one or more non-transitory computer-readable storage media of claim 24, wherein the actions further comprise disabling key indexing during the first phase of data migration.

27. The one or more non-transitory computer-readable storage media of claim 24, wherein the first data portion is at least as old as the threshold age, and wherein the second data portion is more recent than the threshold age.