System and method for data redistribution in a database

• US 10,545,815 B2
• Filed: 08/03/2016
• Issued: 01/28/2020
• Est. Priority Date: 08/03/2016
• Status: Active Grant

• Current Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: Cai, Le, Zhou, QingQing, Sun, Yang
• Primary Examiner(s): Leibovich, Yair
• Application Number:
  US15/227,546
• Publication Number:
  US 20180039534A1
• Time in Patent Office: 1,273 Days
• Field of Search: 714 16
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

A method for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB) is provided. The method includes recording a snapshot of the job data, creating a first data portion in the first DN and a redistribution data portion in the first DN, collecting changes to a job data copy stored in a temporary table, and initiating transfer of the redistribution data portion to the at least one additional DN.

Claims

1. A method for data redistribution of job data in a first data node (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB), comprising:

recording, by a managing DN in the MPP DB, job data information;

splitting the job data into a data portion and a redistribution data portion in the first DN;

collecting changes to the job data;

identifying one or more of the collected changes to the data portion and identifying one or more of the collected changes to the redistribution data portion based on the job data information, wherein the job data information comprises information about the split of the job data;

initiating transfer of the redistribution data portion to the at least one additional DN;

merging the identified changes to the data portion into the data portion; and

merging the identified changes to the redistribution data portion into the redistribution data portion.

2. The method of claim 1, wherein the collected changes are stored in a temporary table.

3. The method of claim 2,

wherein the merging the identified changes to the data portion, and the merging the identified changes to the redistribution data portion comprising:

in response to success of transferring the redistribution data portion to the at least one additional DN,

merging the identified changes to the data portion from the temporary table into the data portion in the first DN, and merging the identified changes to the redistribution data portion from the temporary table into the redistribution data portion in the at least one additional DN, and

wherein the method further comprises;

indicating the first DN can continue processing; and

indicating that the at least one additional DN is ready to begin processing.

4. The method of claim 2, wherein the merging the identified changes to the data portion, and the merging the identified changes to the redistribution data portion comprising:

in response to failure of transferring the redistribution data portion to the at least one additional DN, merging the identified changes to the data portion from the temporary table into the data portion in the first DN, and merging the identified changes to the redistribution data portion from the temporary table into the redistribution data portion in the first DN.

5. The method of claim 4, further comprising:

indicating that the first DN is ready to re-initiate processing in the first DN.

6. The method of claim 4, further comprising:

re-trying the data redistribution.

7. The method of claim 1, further comprising:

in response to failure of transferring the redistribution data portion to the at least one additional DN, re-trying the data redistribution.

8. A device for data redistribution of job data in a first data node (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database, comprising:

a non-transitory memory storage comprising instructions; and

one or more processors in communication with the memory, wherein the one or more processors execute the instructions to;

record job data information;

split the job data into a data portion and a redistribution data portion in the first DN;

collect changes to the job data;

identify one or more of the collected changes to the data portion and identify one or more of the collected changes to the redistribution data portion based on the job data information, wherein the job data information comprises information about the split of the job data;

initiate transfer of the redistribution data portion to the at least one additional DN;

merge the identified changes to the data portion into the data portion; and

merge the identified changes to the redistribution data portion into the redistribution data portion.

9. The device of claim 8, wherein the collected changes are stored in a temporary table.

10. The device of claim 9, wherein the one or more processors further execute the instructions to:

in response to success of transferring the redistribution data portion to the at least one additional DN, merge the identified changes to the data portion from the temporary table into the data portion in the first DN, and merge the identified changes to the redistribution data portion from the temporary table into the redistribution data portion in the at least one additional DN indicate that the first DN continues processing.

11. The device of claim 9, the one or more processors further execute the instructions to:

in response to failure of transferring the redistribution data portion to the at least one additional DN, merge the identified changes to the data portion from the temporary table into the data portion in the first DN, and merge the identified changes to the redistribution data portion from the temporary table into the redistribution data portion in the first DN.

12. The device of claim 11, wherein the one or more processors further execute the instructions to

indicate that the first DN is ready to re-initiate processing in the first DN.

13. The device of claim 11, the one or more processors further execute the instructions to:

re-try the data redistribution.

14. The device of claim 8, wherein the one or more processors further execute the instructions to:

in response to failure of transferring the redistribution data portion to the at least one additional DN, re-try the data redistribution.