Data processing method and apparatus

• US 10,353,896 B2
• Filed: 12/08/2016
• Issued: 07/16/2019
• Est. Priority Date: 06/09/2014
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Ke, Liang, Li, Huaizhou, Wang, Yuhu
• Primary Examiner(s): Mackes, Kris E
• Assistant Examiner(s): Bui, Tiffany Thuy
• Application Number:
  US15/372,489
• Publication Number:
  US 20170091273A1
• Time in Patent Office: 950 Days
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 7/57 Arithmetic logic units [ALU], i.e. arrangements or devices for performing two or more of the operations covered by groups G06F7/483 – G06F7/556 or for performing logical operations G06F7/49, G06F7/491 take precedence

Abstract

A data processing method and apparatus are provided. The data processing method includes determining, according to a database execution plan, a partition quantity corresponding to a currently free programmable memory, partitioning the currently free programmable memory according to the partition quantity, to obtain a programmable memory partition corresponding to the partition quantity, and executing, using the programmable memory partition, a relational algebra logical operation corresponding to the database execution plan. Embodiments of the present disclosure can be used to improve data processing efficiency.

Claims

1. A data processing method comprising:

determining, according to a database execution plan, a partition quantity corresponding to a currently free programmable memory;

partitioning the currently free programmable memory according to the partition quantity to obtain a programmable memory partition corresponding to the partition quantity; and

executing, using the programmable memory partition, a relational algebra logical operation corresponding to the database execution plan,

wherein determining, according to the database execution plan, the partition quantity corresponding to the currently free programmable memory at least comprises;

collecting statistics about an optimal value op of a ratio of a data throughput processed when the free programmable memory executes the relational algebra logical operation to a size of the free programmable memory;

determining, according to the database execution plan, an estimated value rc of a data volume required for the database execution plan;

finding a maximum value rp of a ratio of a data area to a complementary area;

obtaining an amount mc of the currently free programmable memory; and

determining, according to op, rc, rp, and mc, the partition quantity corresponding to the currently free programmable memory.

2. The method according to claim 1, wherein determining, according to the database execution plan, the partition quantity corresponding to the currently free programmable memory further comprises:

generating an execution plan tree according to the database execution plan; and

traversing the execution plan tree to find the maximum value rp of the ratio of the data area to the complementary area, wherein the data area and the complementary area are required for a relational algebra logical operation of a node on the execution plan tree, wherein the data area refers to storage space for storing input/output data information of the relational algebra logical operation, and wherein the complementary area refers to storage space for storing temporary data information obtained during the relational algebra logical operation.

3. The method according to claim 2, wherein before executing the relational algebra logical operation corresponding to the database execution plan, the method further comprises flashing a relational algebra logic set into the programmable memory partition, wherein the relational algebra logic set comprises the relational algebra logical operation corresponding to the database execution plan.

4. The method according to claim 2, wherein executing the relational algebra logical operation corresponding to the database execution plan comprises:

obtaining, according to the execution plan tree, execution sequence information of relational algebra logic corresponding to the database execution plan; and

sequentially activating the relational algebra logic in the programmable memory partition according to the execution sequence information to execute an operation.

5. The method according to claim 4, wherein activating the relational algebra logic in the programmable memory partition to execute the operation comprises:

directly obtaining data in a programmable memory partition corresponding to a unary operator to execute the operation when the activated relational algebra logic is the unary operator; and

synchronizing data addressing information in the two programmable memory partitions corresponding to a binary operator and executing a currently active logical operation by a programmable memory partition in the two programmable memory partitions that has a larger data volume when the activated relational algebra logic is the binary operator and two programmable memory partitions corresponding to the binary operator have same execution sequence information.

6. The method according to claim 2, wherein executing the relational algebra logical operation corresponding to the database execution plan comprises:

obtaining, according to the execution plan tree, logic execution sequence information of a relational algebra operator corresponding to the database execution plan; and

sending a logical instruction to the programmable memory partition according to the logic execution sequence information, and wherein the logical instruction causes the programmable memory partition to execute a logical operation.

7. The method according to claim 4, wherein executing the relational algebra logical operation corresponding to the database execution plan further comprises:

determining whether to execute an aggregation operation on operation results distributed in different programmable memory partitions after all relational algebra logical operations corresponding to the logic execution sequence information are executed; and

activating a logic aggregation operation and generating a final result according to the operation results distributed in the different programmable memory partitions when the aggregation operation needs to be executed.

15. The method according to claim 1, further comprising determining whether to flash a relational algebra logic set into the programmable memory partition before executing the relational algebra logical operation corresponding to the database execution plan, wherein the relational algebra logic set comprises the relational algebra logical operation corresponding to the database execution plan.

16. The method according to claim 15, wherein determining whether to flash the relational algebra logic set into the programmable memory partition is determined according to a speed of writing logic into a logic control area in the programmable memory partition.

17. The method according to claim 1, wherein before executing the relational algebra logical operation corresponding to the database execution plan, the method further comprises flashing a relational algebra logic set into the programmable memory partition when a speed of writing a new instruction into a logic control area of the programmable memory partition is lower than a threshold.

8. A data processing apparatus comprising:

a processor;

a memory configured to store a computer execution instruction; and

a bus coupling the processor and the memory,

wherein the processor is configured to execute the computer execution instruction stored in the memory to;

determine, according to a database execution plan, a partition quantity corresponding to a currently free programmable memory;

partition the currently free programmable memory according to the partition quantity to obtain a programmable memory partition corresponding to the partition quantity;

execute, using the programmable memory partition, a relational algebra logical operation corresponding to the database execution plan;

collect statistics about an optimal value op of a ratio of a data throughput processed when the free programmable memory executes the relational algebra logical operation to a size of the free programmable memory;

determine, according to the database execution plan, an estimated value rc of a data volume required for the database execution plan;

find a maximum value rp of a ratio of a data area to a complementary area;

obtain an amount mc of the currently free programmable memory; and

determine, according to op, rc, rp, and mc, the partition quantity corresponding to the currently free programmable memory.

9. The apparatus according to claim 8, wherein the processor is further configured to:

generate an execution plan tree according to the database execution plan; and

traverse the execution plan tree to find the maximum value rp of the ratio of the data area to the complementary area, wherein the data area and the complementary area are required for a relational algebra logical operation of a node on the execution plan tree, wherein the data area refers to storage space for storing input/output data information of the relational algebra logical operation, and wherein the complementary area refers to storage space for storing temporary data information obtained during the relational algebra logical operation.

10. The apparatus according to claim 9, the processor is further configured to flash a relational algebra logic set into the programmable memory partition before executing the relational algebra logical operation corresponding to the database execution plan, wherein the relational algebra logic set comprises the relational algebra logical operation corresponding to the database execution plan.

11. The apparatus according to claim 9, wherein the processor is further configured to:

obtain, according to the execution plan tree, execution sequence information of relational algebra logic corresponding to the database execution plan; and

sequentially activate the relational algebra logic in the programmable memory partition according to the execution sequence information to execute an operation.

12. The apparatus according to claim 11, wherein the processor is further configured to:

directly obtain data in a programmable memory partition corresponding to a unary operator to execute the operation when the activated relational algebra logic is the unary operator; and

synchronize data addressing information in the two programmable memory partitions corresponding to a binary operator and execute a currently active logical operation by a programmable memory partition in the two programmable memory partitions that has a larger data volume when the activated relational algebra logic is the binary operator and two programmable memory partitions corresponding to the binary operator have same execution sequence information.

13. The apparatus according to claim 9, wherein the processor is further configured to:

obtain, according to the execution plan tree, logic execution sequence information of a relational algebra operator corresponding to the database execution plan; and

send a logical instruction to the programmable memory partition according to the logic execution sequence information, and wherein the logical instruction causes the programmable memory partition to execute a logical operation.

14. The apparatus according to claim 11, wherein the processor is further configured to:

determine whether to execute an aggregation operation on operation results distributed in different programmable memory partitions after all relational algebra logical operations corresponding to the logic execution sequence information are executed; and

activate a logic aggregation operation and generate a final result according to the operation results distributed in the different programmable memory partitions when the aggregation operation needs to be executed.