SYSTEM AND METHOD FOR DATA REDISTRIBUTION IN A DATABASE

• US 20180039534A1
• Filed: 08/03/2016
• Published: 02/08/2018
• Est. Priority Date: 08/03/2016
• Status: Active Grant

• Current Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: CAI, Le, ZHOU, QingQing, SUN, Yang
• Granted Patent: US 10,545,815 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

A method for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB) is provided. The method includes recording a snapshot of the job data, creating a first data portion in the first DN and a redistribution data portion in the first DN, collecting changes to a job data copy stored in a temporary table, and initiating transfer of the redistribution data portion to the at least one additional DN.

Claims

1. A method for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database (DB), comprising:

recording a snapshot of the job data;

creating a first data portion in the first DN and a redistribution data portion in the first DN;

collecting changes to a job data copy stored in a temporary table; and

initiating transfer of the redistribution data portion to the at least one additional DN.

2. The method of claim 1, with the collecting the changes comprising collecting the changes to a first DN data changes stored in the temporary table and to a redistribution data changes stored in the temporary table.

3. The method of claim 1, wherein the snapshot is used to determine the first DN data changes and the redistribution data changes stored in the temporary table.

4. The method of claim 1, further comprising:

if transferring the redistribution data portion to the at least one additional DN succeeds;

merging first DN data changes from the temporary table into the first data portion in the first DN; and

indicating the first DN can continue processing.

5. The method of claim 1, further comprising:

if transferring the redistribution data portion to the at least one additional DN succeeds;

merging redistribution data changes from the temporary table into the at least one additional DN; and

indicating the at least one additional DN is ready to begin processing.

6. The method of claim 1, further comprising:

if transferring the redistribution data portion to the at least one additional DN fails;

merging first DN data changes from the temporary table into the first data portion in the first DN;

merging redistribution data changes from the temporary table into the redistribution data portion in the first DN; and

indicating the first DN is ready to re-initiate processing in the first DN.

7. The method of claim 1, further comprising:

if transferring the redistribution data portion to the at least one additional DN fails;

merging first DN data changes from the temporary table into the first data portion in the first DN;

merging redistribution data changes from the temporary table into the redistribution data portion in the first DN; and

re-trying the data redistribution.

8. The method of claim 1, further comprising:

if transferring the redistribution data portion to the at least one additional DN succeeds;

identifying first DN data changes in the temporary table using the snapshot;

merging the first DN data changes into the first data portion in the first DN; and

indicating the first DN can continue processing.

9. The method of claim 1, further comprising:

if transferring the redistribution data portion to the at least one additional DN succeeds;

identifying redistribution data changes in the temporary table using the snapshot;

merging the redistribution data changes into the redistribution data portion in the at least one additional DN; and

indicating the at least one additional DN is ready to begin processing.

10. The method of claim 1, further comprising:

if transferring the redistribution data portion to the at least one additional DN fails;

identifying first DN data changes in the temporary table using the snapshot;

merging the first DN data changes into the first data portion in the first DN;

identifying redistribution data changes in the temporary table using the snapshot;

merging the redistribution data changes into the redistribution data portion in the first DN; and

indicating the first DN is ready to re-initiate processing in the first DN.

11. The method of claim 1, further comprising:

if transferring the redistribution data portion to the at least one additional DN fails, re-trying the data redistribution.

12. The method of claim 1, further comprising:

if transferring the redistribution data portion to the at least one additional DN fails;

identifying first DN data changes in the temporary table using the snapshot;

merging the first DN data changes into the first data portion in the first DN;

identifying redistribution data changes in the temporary table using the snapshot;

merging the redistribution data changes into the redistribution data portion in the first DN; and

re-trying the data redistribution.

13. The method of claim 1, wherein the job data comprises the first data portion and the redistribution data portion.

14. The method of claim 1, wherein the job data comprises the redistribution data portion.

15. A device for data redistribution of a job data in a first datanode (DN) to at least one additional DN in a Massively Parallel Processing (MPP) Database, comprising:

a non-transitory memory storage comprising instructions; and

one or more processors in communication with the memory, wherein the one or more processors execute the instructions to;

record a snapshot of the job data;

create a first data portion in the first DN and a redistribution data portion in the first DN;

collect changes to a job data copy stored in a temporary table; and

initiate transfer of the redistribution data portion to the at least one additional DN.

16. The device of claim 15, with the collecting the changes comprising collecting the changes to a first DN data changes stored in the temporary table and to a redistribution data changes stored in the temporary table.

17. The device of claim 15, wherein the snapshot is used to determine the first DN data changes and the redistribution data changes stored in the temporary table.

18. The device of claim 15, wherein the one or more processors are further configured to:

if transferring the redistribution data portion to the at least one additional DN succeeds;

merging first DN data changes from the temporary table into the first data portion in the first DN; and

indicating the first DN can continue processing.

19. The device of claim 15, wherein the one or more processors are further configured to:

if transferring the redistribution data portion to the at least one additional DN succeeds;

merging redistribution data changes from the temporary table into the at least one additional DN; and

indicating the at least one additional DN is ready to begin processing.

20. The device of claim 15, wherein the one or more processors are further configured to:

if transferring the redistribution data portion to the at least one additional DN fails;

merging first DN data changes from the temporary table into the first data portion in the first DN;

merging redistribution data changes from the temporary table into the redistribution data portion in the first DN; and

indicating the first DN is ready to re-initiate processing in the first DN.

21. The device of claim 15, further comprising:

if transferring the redistribution data portion to the at least one additional DN fails;

merging first DN data changes from the temporary table into the first data portion in the first DN;

merging redistribution data changes from the temporary table into the redistribution data portion in the first DN; and

re-trying the data redistribution.

22. The device of claim 15, wherein the one or more processors are further configured to:

if transferring the redistribution data portion to the at least one additional DN succeeds;

identifying first DN data changes in the temporary table using the snapshot;

merging the first DN data changes into the first data portion in the first DN; and

indicating the first DN can continue processing.

23. The device of claim 15, wherein the one or more processors are further configured to:

if transferring the redistribution data portion to the at least one additional DN succeeds;

identifying redistribution data changes in the temporary table using the snapshot;

merging the redistribution data changes into the redistribution data portion in the at least one additional DN; and

indicating the at least one additional DN is ready to begin processing.

24. The device of claim 15, wherein the one or more processors are further configured to:

if transferring the redistribution data portion to the at least one additional DN fails;

identifying first DN data changes in the temporary table using the snapshot;

merging the first DN data changes into the first data portion in the first DN;

identifying redistribution data changes in the temporary table using the snapshot;

merging the redistribution data changes into the redistribution data portion in the first DN; and

indicating the first DN is ready to re-initiate processing in the first DN.

25. The device of claim 15, further comprising:

if transferring the redistribution data portion to the at least one additional DN fails, re-trying the data redistribution.

26. The device of claim 15, further comprising:

if transferring the redistribution data portion to the at least one additional DN fails;

identifying first DN data changes in the temporary table using the snapshot;

merging the first DN data changes into the first data portion in the first DN;

identifying redistribution data changes in the temporary table using the snapshot;

merging the redistribution data changes into the redistribution data portion in the first DN; and

re-trying the data redistribution.

27. The device of claim 15, wherein the job data comprises the first data portion and the redistribution data portion.

28. The device of claim 15, wherein the job data comprises the redistribution data portion.