PARALLEL PROCESSING DATABASE SYSTEM

• US 20180011902A1
• Filed: 09/25/2017
• Published: 01/11/2018
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Welton, Caleb E., Harada, Hitoshi, Cohen, Jeffrey Ira, Chang, Lei, Reddy, Radhika, Ma, Tao, Wang, Zhanwei
• Granted Patent: US 10,572,479 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24524 Access plan code generation and invalidation; Reuse of access plans
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method and system for executing database queries in parallel using a shared metadata store. The metadata store may reside on a master node, where the master node is the root node in a tree. The master node may distribute query plans and query metadata to other nodes in the cluster. These additional nodes may request additional metadata from each other or the master nodes as necessary.

Claims

1. A parallel processing database system, comprising:

a database catalog associated with a master node comprising metadata defining database objects;

the master node, wherein the master node comprises one or more processors, wherein the master node is configured to receive a query from a client node, to obtain metadata from the database catalog using a catalog server session that is established in connection with the query, to send one or more query plans and corresponding metadata to one or more of a plurality of worker nodes, and to obtain additional metadata to be used in connection with execution of the corresponding query plan from the database catalog using the catalog server session that was established in connection with the query, and to provide the addition metadata to one or more of the plurality of worker nodes; and

the plurality of worker nodes in communication with the master node, wherein one or more of the plurality of worker nodes are configured to use the corresponding query metadata received from the master node in connection with execution of the one or more query plans.

2. The system of claim 1, wherein the master node is further configured to receive a request from a first worker node of the one or more of the plurality of worker nodes, the request being for additional metadata to be used in connection with execution of the corresponding query plan, and wherein the additional metadata is obtained in response to the master node receiving the request for additional metadata.

3. The system of claim 2, wherein the request for additional metadata is received from a first worker node in the event that the first worker node determines that received corresponding query metadata is not sufficient to execute the corresponding query plan, and in the event that at least one other worker node of the plurality of worker nodes is queried for the additional metadata and the at least one other worker node does not have the additional metadata.

4. The system of claim 1, wherein individual nodes of the plurality of worker nodes include a cache.

5. The system of claim 4, wherein the received query metadata is stored in the cache of the individual nodes.

6. The system of claim 1, wherein the master node and the plurality of worker nodes have a tree configuration.

7. The system of claim 6, wherein the master node is a root node.

8. The system of claim 1, wherein an individual node of the plurality of worker nodes requests additional metadata from the master node.

9. The system of claim 1, wherein an individual node of the plurality of worker nodes requests additional metadata from a parent node.

10. The system of claim 9, wherein the parent node is another worker node.

11. The system of claim 1, wherein the database objects comprise database table definitions, database functions, database users, database views, and database indexes.

12. The system of claim 1, wherein the master node receives the query from the client, develops the one or more query plans based at least in part on the query, and communicates each of the one or more query plans to a corresponding one of the plurality of worker nodes in connection with metadata from the database catalog that is determined to be required for execution of the query plan.

13. The system of claim 1, wherein in response to receiving the query, the catalog server session is established, and the corresponding query metadata is retrieved in the catalog server session.

14. The system of claim 13, wherein in the event that a first worker node requests additional query metadata to be used in connection with the execution of the corresponding query plan, the additional query metadata is retrieved using the catalog server session that was established in connection with the received query.

15. The system of claim 1, wherein in the event that metadata stored in the database catalog is updated at a time between when metadata to be sent in connection with the one or more query plans is obtained from the database catalog and when the request for additional metadata is received by the master node, the additional metadata is obtained from the database catalog using the catalog server session that was established in connection with the query such that the additional metadata does not include the updated metadata.

16. The system of claim 1, wherein a first worker node of the plurality of worker nodes determines that the received corresponding query metadata is not sufficient to execute the corresponding query plan, and in response to determining that the received corresponding query metadata is not sufficient to execute the corresponding query plan requests additional metadata from one or more other of the plurality of worker nodes.

17. The system of claim 16, wherein the master node is queried for the additional metadata after the one or more other of the plurality of worker nodes are requested for the additional metadata.

18. The system of claim 16, wherein the master node and the plurality of worker nodes have a tree configuration, and the one or more other of the plurality of worker nodes are queried for additional metadata based at least in part on the tree configuration.

19. A method comprising:

receiving, by a master node, a query from a client node, wherein the master node is associated with at least one processor configured to communicate with a database catalog that includes metadata defining database objects, and wherein the master node is in communication with a plurality of worker nodes;

obtaining, by the master node, metadata from the database catalog using a catalog server session that is established in connection with the query;

sending, by the master node, a corresponding query plan and corresponding query metadata to one or more of the plurality of worker nodes, wherein the one or more of the plurality of worker nodes are configured to use the corresponding query metadata received from the master node in connection with execution of the corresponding query plan;

obtaining, by the master node, additional metadata to be used in connection with execution of the corresponding query from the database catalog using the catalog server session that was established in connection with the query; and

providing the addition metadata to one or more of the plurality of worker nodes.

20. A computer program product embodied in a tangible non-transitory computer-readable storage medium and comprising computer instructions for:

receiving, by a master node, a query from a client node, wherein the master node is associated with a database catalog that includes metadata defining database objects, and wherein the master node is in communication with a plurality of worker nodes;

obtaining, by the master node, metadata from the database catalog using a catalog server session that is established in connection with the query;

sending, by the master node, a corresponding query plan and corresponding query metadata to one or more of the plurality of worker nodes, wherein the one or more of the plurality of worker nodes are configured to use the corresponding query metadata received from the master node in connection with execution of the corresponding query plan;

obtaining, by the master node, additional metadata to be used in connection with execution of the corresponding query from the database catalog using the catalog server session that was established in connection with the query; and

providing the addition metadata to one or more of the plurality of worker nodes.