PARALLEL PROCESSING DATABASE SYSTEM WITH A SHARED METADATA STORE

• US 20160342647A1
• Filed: 08/03/2016
• Published: 11/24/2016
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Welton, Caleb E., Harada, Hitoshi, Cohen, Jeffrey Ira, Chang, Lei, Reddy, Radhika, Ma, Tao, Wang, Zhanwei
• Granted Patent: US 10,013,456 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24524 Access plan code generation and invalidation; Reuse of access plans
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method and system for executing database queries in parallel using a shared metadata store. The metadata store may reside on a master node, and the master node may distribute this metadata with query plans to worker nodes. The worker nodes may request additional metadata from the master node when necessary.

Claims

1. A parallel processing database system, comprising:

a master node comprising one or more processors, wherein the master node is configured to receive a query from a client node, develop a query plan based at least in part on the query, determine metadata to be used in connection with execution of the query plan, and to communicate, to a first worker node, the query plan and the metadata determined to be used in connection with execution of the query plan, wherein the metadata is stored in a database catalog that comprises database objects, and wherein the metadata is associated with corresponding ones of the database objects;

the first worker node comprising one or more processors, wherein the first worker node is in communication with the master node and receives the query plan from the master node, executes at least a part of the query in connection with executing the query plan received from the master node, and in an event that the first worker node requires additional metadata in connection with executing the at least the part of the query, communicates a request for the additional metadata to a parent node corresponding to the first worker node; and

the parent node comprising one or more processors, wherein in the event that the parent node does not have the additional metadata, the parent node forwards the request for the additional metadata to a next parent node in a hierarchical structure of nodes.

2. The system of claim 1, wherein the parent node corresponding to the first worker node is determined in relation to the hierarchical structure of nodes.

3. The system of claim 2, wherein hierarchical structure of nodes includes the master node, the first worker node, and the parent node arranged in a hierarchy.

4. The system of claim 1, wherein in the event that the parent node has the additional metadata, the parent node communicates the additional metadata to the first worker node.

5. The system of claim 1, wherein the first worker node comprises a query execution process that is configured to execute a database query, wherein the first worker node runs the query execution process in response to receiving the query plan from the master node.

6. The system of claim 5, wherein the database query comprises the query plan and query metadata.

7. The system of claim 6, wherein the query metadata is stored in the database catalog and corresponds to the metadata that is associated with the database objects and that is determined to be used in connection with execution of the query plan.

8. The system of claim 1, wherein the master node comprises a query dispatch process.

9. The system of claim 8, wherein the query dispatch process receives the query from the client node.

10. The system of claim 9, wherein the query dispatch process creates the query plan based at least in part on the query.

11. The system of claim 10, wherein the query dispatch process transmits the query plan to the first worker node.

12. The system of claim 8, wherein the query dispatch process obtains query metadata from the database catalog, wherein the query metadata corresponds to the metadata that is associated with the database objects and that is determined to be used in connection with execution of the query plan.

13. The system of claim 12, wherein the query dispatch process transmits the query metadata to the first worker node.

14. The system of claim 8, wherein the query dispatch process transmits a query plan and query metadata concurrently to the first worker node.

15. The system of claim 1, further comprising a plurality of worker nodes, wherein the first worker node corresponds to at least one of the plurality of worker nodes.

16. The system of claim 15, wherein the parent node corresponds to at least one of the plurality of worker nodes.

17. The system of claim 15, wherein the master node communicates a plurality of query plans and corresponding query metadata to the plurality of worker nodes.

18. The system of claim 17, wherein the plurality of worker nodes execute the query plans in parallel.

19. The system of claim 1, wherein the database objects comprise one or more of database table definitions, database functions, database users, database views, and database indexes.

20. The system of claim 1, wherein the first worker node executes the query plan using at least the metadata received from the master node.

21. The system of claim 1, wherein the parent node is another worker node connected to the master node via the hierarchical structure of nodes.

22. The system of claim 1, wherein the first worker node stores its ancestry in relation to the hierarchical structure of nodes.

23. The system of claim 1, wherein in an event that a change is made to the metadata in the database catalog, the change is not propagated to the worker nodes to update a database at the worker nodes.

24. A method for processing a query, comprising:

receiving, by a worker node, a query plan and metadata from a master node, wherein the query plan is generated by the master node based at least in part on a query that the master node receives from a client node, wherein the metadata is determine metadata to be used in connection with execution of the query plan, and wherein the metadata stored in a database catalog that comprises database objects, and wherein the metadata is associated with corresponding ones of the database objects;

executing, by the worker node, at least a part of the query in connection with executing the query plan received from the master node; and

in an event that the worker node requires additional metadata in connection with executing the at least the part of the query, communicating a request for the additional metadata to a parent node corresponding to the worker node, wherein in the event that the parent node does not have the additional metadata, the parent node forwards the request for the additional metadata to a next parent node in a hierarchical structure of nodes.