REAL TIME DATA REPLICATION FOR QUERY EXECUTION IN A MASSIVELY PARALLEL COMPUTER

• US 20090132609A1
• Filed: 11/16/2007
• Published: 05/21/2009
• Est. Priority Date: 11/16/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barsness, Eric Lawrence, Santosuosso, John Matthew, Darrington, David L., Peters, Amanda
• Granted Patent: US 7,844,620 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24532 of parallel queries

Abstract

Embodiments of the invention may be used to increase query processing parallelism of an in-memory database stored on a parallel computing system. A group of compute nodes each store a portion of data as part of the in-memory database. Further, a pool of compute nodes may be reserved to create copies of data from the compute nodes of the in-memory database as part of query processing. When a query is received for execution, the query may be evaluated to determine whether portions of in-memory should be duplicated to allow multiple elements of the query (e.g., multiple query predicates) to be evaluated in parallel.

Claims

1. A method for processing a database query, comprising:

receiving a query of a database, wherein the database is stored on a plurality of compute nodes provided by a parallel computing system;

identifying two or more portions of the query evaluated using data records stored on a first compute node of the plurality of compute nodes;

copying the data records stored on the first compute node to a second compute node;

transmitting a first portion of the query to the first compute node and a second portion of the query to the second compute node, wherein the first compute node and the second compute node execute the respective first query portion and second query portion in parallel, thereby producing respective query results; and

receiving the respective query results from the first compute node and the second compute node.

2. The method of claim 1, further comprising:

reconciling the results received from the first compute node and the second compute node; and

returning the reconciled results.

3. The method of claim 2, wherein the respective query results received from the first compute node and the second compute node are represented as bit strings, and wherein each bit of the bit strings indicates whether a given database record satisfied a query condition included in the respective portion of the database query executed by the first compute node and the second compute node.

4. The method of claim 1, wherein the second compute node is allocated from a pool of compute nodes reserved for creating copies of data from the plurality of compute nodes provided by the parallel computing system.

5. The method of claim 4, wherein the node pool includes a minimum and a maximum number of compute nodes available for copying portions of the in-memory database, and a minimum and a maximum number of compute nodes used for copying portions of the in-memory database while processing the database query.

6. The method of claim 1, wherein the data records stored on the first compute node are copied to the second compute node upon determining that query processing efficiency may be improved by at least a predetermined threshold amount by processing the two or more portions in parallel.

7. The method of claim 1, wherein the two or more portions of the query each comprise a query predicate evaluated against data records on the first compute node of the plurality of compute nodes.

8. A computer readable storage medium containing a program which, when executed, performs an operation, comprising:

receiving a query of a database, wherein the database is stored on a plurality of compute nodes provided by a parallel computing system;

identifying two or more portions of the query evaluated using data records stored on a first compute node of the plurality of compute nodes;

copying the data records stored on the first compute node to a second compute node;

transmitting a first portion of the query to the first compute node and a second portion of the query to the second compute node, wherein the first compute node and the second compute node execute the respective first query portion and second query portion in parallel, thereby producing respective query results; and

receiving the respective query results from the first compute node and the second compute node.

9. The computer readable storage medium of claim 8, wherein the operation further comprises:

reconciling the results received from the first compute node and the second compute node; and

returning the reconciled results.

10. The computer readable storage medium of claim 9, wherein the respective query results received from the first compute node and the second compute node are represented as bit strings, and wherein each bit of the bit strings indicates whether a given database record satisfied a query condition included in the respective portion of the database query executed by the first compute node and the second compute node.

11. The computer readable storage medium of claim 8, wherein the second compute node is allocated from a pool of compute nodes reserved for creating copies of data from the plurality of compute nodes provided by the parallel computing system.

12. The computer readable storage medium of claim 11, wherein the node pool includes a minimum and a maximum number of compute nodes available for copying portions of the in-memory database, and a minimum and a maximum number of compute nodes used for copying portions of the in-memory database while processing the database query.

13. The computer readable storage medium of claim 8, wherein the data records stored on the first compute node are copied to the second compute node upon determining that query processing efficiency may be improved by at least a predetermined threshold amount by processing the two or more portions in parallel.

14. The computer readable storage medium of claim 8, wherein the two or more portions of the query each comprise a query predicate evaluated against data records on the first compute node of the plurality of compute nodes.

15. A parallel computing system, comprising:

a plurality of compute nodes, each having at least a processor and a memory, wherein each of the plurality of compute nodes stores a portion of an in-memory database; and

a master node having at least a processor and a memory and a database controller program configured to;

receive a query of the in-memory database,

identify two or more portions of the query evaluated using data records stored on a first compute node of the plurality of compute nodes,

copy the data records stored on the first compute node to a second compute node,

transmit a first portion of the query to the first compute node and a second portion of the query to the second compute node, wherein the first compute node and the second compute node execute the respective first query portion and second query portion in parallel, thereby producing respective query results; and

receive the respective query results from the first compute node and the second compute node.

16. The parallel computing system of claim 15, further comprising:

reconciling the results received from the first compute node and the second compute node; and

returning the reconciled results.

17. The parallel computing system of claim 16, wherein the respective query results received from the first compute node and the second compute node are represented as bit strings, and wherein each bit of the bit strings indicates whether a given database record satisfied a query condition included in the respective portion of the database query executed by the first compute node and the second compute node.

18. The parallel computing system of claim 15, wherein the second compute node is allocated from a pool of compute nodes reserved for creating copies of data from the plurality of compute nodes used to store the in-memory database.

19. The parallel computing system of claim 18, wherein the node pool includes a minimum and a maximum number of compute nodes available for copying portions of the in-memory database, and a minimum and a maximum number of compute nodes used for copying portions of the in-memory database while processing the database query.

20. The parallel computing system of claim 15, wherein the data records stored on the first compute node are copied to the second compute node upon determining that query processing efficiency may be improved by at least a predetermined threshold amount by processing the two or more portions in parallel.

21. The parallel computing system of claim 15, wherein the two or more portions of the query each comprise a query predicate evaluated against data records on the first compute node of the plurality of compute nodes.