System and Method for Massively Parallel Processor Database

• US 20160210340A1
• Filed: 01/21/2015
• Published: 07/21/2016
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: Cai, Le, Zhou, Qingqing, Jagtiani, Kamini
• Granted Patent: US 9,959,332 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

In one embodiment, a method includes determining a number of initial servers in a massively parallel processing (MPP) database cluster and determining an initial bucket configuration of the MPP database cluster, where the initial bucket configuration has a number of initial buckets. The method also includes adding a number of additional servers to the MPP database cluster to produce a number of updated servers, where the updated servers include the initial servers and the additional servers and creating an updated bucket configuration in accordance with the number of initial servers, the initial bucket configuration, and the number of additional servers, where the updated bucket configuration has a number of updated buckets. Additionally, the method includes redistributing data of the MPP cluster in accordance with the updated bucket configuration.

Claims

1. A method comprising:

determining a number of initial servers in a massively parallel processing (MPP) database cluster;

determining an initial bucket configuration of the MPP database cluster, wherein the initial bucket configuration has a number of initial buckets;

adding a number of additional servers to the MPP database cluster to produce a number of updated servers, wherein the updated servers comprise the initial servers and the additional servers;

creating an updated bucket configuration in accordance with the number of initial servers, the initial bucket configuration, and the number of additional servers, wherein the updated bucket configuration has a number of updated buckets; and

redistributing data of the MPP cluster in accordance with the updated bucket configuration.

2. The method of claim 1, wherein creating the updated bucket configuration comprises moving a subset of buckets from an initial server of the initial servers to an additional server of the additional servers.

3. The method of claim 1, wherein the updated bucket configuration comprises a mapping of buckets to the updated servers, wherein each of the updated servers has either a minimum number of buckets or a maximum number of buckets, wherein the maximum number of buckets is one bucket more than the minimum number of buckets.

4. The method of claim 1, wherein creating the updated bucket configuration comprises determining whether the number of updated buckets is greater than the number of initial buckets.

5. The method of claim 4, wherein determining whether the number of updated buckets is greater than the number of initial buckets comprises:

determining a redistributed bucket configuration having the number of updated buckets and the number of initial buckets;

determining a percentage load variation of the redistributed bucket configuration;

determining whether the percentage load variation is acceptable;

setting the number of updated buckets to be greater than the number of initial buckets when the percentage load variation is not acceptable; and

setting the number of updated buckets to the number of initial buckets when the percentage load variation is acceptable.

6. The method of claim 1, wherein the number of updated buckets is a power of two of the number of initial buckets.

7. The method of claim 6, wherein the number of updated buckets is two times the number of initial buckets.

8. The method of claim 1, wherein the number of initial buckets is a power of two.

9. The method of claim 1, wherein redistributing data of the MPP cluster comprises:

determining an initial bucket-server mapping;

determining a subset of buckets to be moved from the initial servers to the additional servers;

determining data associated with the subset of buckets to produce a subset of data; and

moving the subset of data from the initial servers to the additional servers.

10. The method of claim 1, further comprising placing data on the number of initial servers in accordance with the initial bucket configuration before adding the number of additional servers to the MPP database cluster.

11. The method of claim 10, wherein placing the data on the initial servers comprises:

determining a hash value for a row of a table; and

determining a bucket associated with the row in accordance with the hash value.

12. A method comprising:

determining an updated bucket-server mapping for a massively parallel processor (MPP) database cluster in accordance with a number of initial servers and a number of added servers;

determining whether a first table is to be redistributed in accordance with the updated bucket-server mapping and an initial bucket-server mapping;

starting a first transaction when the first table is to be redistributed;

performing the first transaction comprising moving data from an initial server of the initial servers to an added server of the added servers; and

committing the first transaction after performing the first transaction.

13. The method of claim 12, wherein performing the transaction further comprises:

creating a temporary table in accordance with the updated bucket-server mapping;

redistributing the data in accordance with the updated bucket-server mapping; and

merging the temporary table and the first table after redistributing the data.

14. The method of claim 13, wherein redistributing the data comprises:

building delete statements in accordance with a difference between the updated bucket-server mapping and the initial bucket-server mapping; and

issuing the delete statements.

15. The method of claim 13, wherein redistributing the data comprises:

building insert statements in accordance with a difference between the updated bucket-server mapping and the initial bucket-server mapping; and

issuing the insert statements.

16. The method of claim 12, further comprising:

determining whether a second table is to be redistributed after committing the first transaction;

starting a second transaction when the second table is to be redistributed; and

removing the initial bucket-server mapping when the second table is not to be redistributed.

17. The method of claim 12, further comprising creating a list of tables to be redistributed, wherein determining whether the first table is to be redistributed comprises determining whether the first table is to be redistributed in accordance with the list of tables.

18. The method of claim 12, further comprising installing a number of added the servers before determining whether the first table is to be redistributed.

19. A computer comprising:

a processor; and

a non-transitory computer readable storage medium storing programming for execution by the processor, the programming including instructions to

determine a number of initial servers in a massively parallel processing (MPP) database cluster,

determine an initial bucket configuration of the MPP database cluster, wherein the initial bucket configuration has a number of initial buckets,

add a number of additional servers to the MPP database cluster to produce a number of updated servers, wherein the updated servers comprise the initial servers and the additional servers,

create an updated bucket configuration in accordance with the number of initial servers, the initial bucket configuration, and the number of additional servers, wherein the updated bucket configuration has a number of updated buckets, and

redistribute data of the MPP cluster in accordance with the updated bucket configuration.

20. A computer comprising:

a processor; and

a non-transitory computer readable storage medium storing programming for execution by the processor, the programming including instructions to

determine an updated bucket-server mapping for a massively parallel processor (MPP) database cluster in accordance with a number of initial servers and a number of added servers,

determine whether a first table is to be redistributed in accordance with the updated bucket-server mapping and an initial bucket-server mapping,

start a first transaction when the first table is to be redistributed,

perform the first transaction comprising moving data from an initial server of the initial servers to an added server of the added servers, and

commit the first transaction after performing the first transaction.