Parallel processing database system with a shared metadata store

• US 9,454,573 B1
• Filed: 03/15/2013
• Issued: 09/27/2016
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Welton, Caleb, Harada, Hitoshi, Cohen, Jeffrey, Chang, Lei, Reddy, Radhika, Ma, Tao, Wang, Zhanwei
• Primary Examiner(s): Ho, Binh V
• Application Number:
  US13/838,749
• Time in Patent Office: 1,292 Days
• Field of Search: 707/718
• US Class Current: 1/1
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24524 Access plan code generation and invalidation; Reuse of access plans
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method and system for executing database queries in parallel using a shared metadata store. The metadata store may reside on a master node, and the master node may distribute this metadata with query plans to worker nodes. The worker nodes may request additional metadata from the master node when necessary.

Claims

1. A parallel processing database system, comprising:

a master node comprising a processor and a database catalog, wherein the database catalog includes metadata defining database objects, and wherein the master node is configured to receive a query from a client node, wherein the master node includes a query dispatcher that receives the query from the client node, develops the query plan based at least in part on the query, and communicates the query plan to a worker node in connection with metadata from the database catalog that the query dispatcher determines is required for execution of the query plan; and

the worker node, wherein the worker node is in communication with the master node such that the worker node receives a query plan from the master node, and the worker node comprising a processor and a query execution process, wherein the processor runs the query execution process so as to execute at least a part of the query in connection with executing the query plan received from the master node, and wherein in an event that the worker node requires additional metadata in connection with executing the at least the part of the query, the processor makes a request, for the additional metadata, to a parent node corresponding to the worker node in relation to a hierarchical structure of nodes,

wherein the hierarchical structure of nodes includes the master node, the worker node, and the parent node arranged in a hierarchy, and

wherein in the event that the parent node does not have the additional metadata, the parent node forwards the request for the additional metadata to the next parent node in the hierarchical structure of nodes.

2. The system of claim 1, wherein the master node comprises a processor and a non-transitory computer readable storage medium.

3. The system of claim 1, wherein the worker node comprises a processor and a non-transitory computer readable storage medium.

4. The system of claim 1, wherein the query execution process executes a database query received from the master node, and wherein the query plan is included in the database query received from the master node.

5. The system of claim 4, wherein the database query comprises a query plan and query metadata.

6. The system of claim 5, wherein the query metadata is stored in the database catalog.

7. The system of claim 1, wherein the master node comprises a query dispatch process.

8. The system of claim 7, wherein the query dispatch process receives the query from the client.

9. The system of claim 8, wherein the query dispatch process creates the query plan from the query.

10. The system of claim 9, wherein the query dispatch process transmits the query plan to the worker node.

11. The system of claim 7, wherein the query dispatch process retrieves query metadata from the database catalog.

12. The system of claim 11, wherein the query dispatch process transmits the query metadata to the worker node.

13. The system of claim 7, wherein the query dispatch process transmits a query plan and query metadata concurrently to the worker node.

14. The system of claim 1, further comprising a plurality of worker nodes.

15. The system of claim 14, wherein the master node transmits a plurality of query plans and query metadata to the plurality of worker nodes.

16. The system of claim 15, wherein the plurality of worker nodes execute the query plans in parallel.

17. The system of claim 1, wherein the database objects comprise database table definitions, database functions, database users, database views, and database indexes.

18. The system of claim 1, wherein the worker node comprises a non-transitory computer readable storage medium.

19. The system of claim 1, wherein the query execution process of the worker node executes the query plan using at least the metadata received from the master node.

20. The system of claim 1, wherein the parent node is another worker node connected to the master node via the hierarchical structure.

21. The system of claim 1, wherein the worker node stores its ancestry in relation to the hierarchical structure of nodes.

22. The system of claim 1, wherein in an event that a change is made to the metadata in the database catalog, the change is not propagated to the worker nodes to update a database at the worker nodes.