Processing a database query using a shared metadata store

• US 10,120,900 B1
• Filed: 03/15/2013
• Issued: 11/06/2018
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Welton, Caleb, Harada, Hitoshi, Cohen, Jeffrey, Chang, Lei, Reddy, Radhika, Ma, Tao, Wang, Zhanwei
• Primary Examiner(s): Mahmoudi, Tony
• Assistant Examiner(s): Le, Michael
• Application Number:
  US13/838,955
• Time in Patent Office: 2,062 Days
• Field of Search: 707718
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24524 Access plan code generation and invalidation; Reuse of access plans
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H05K 999/99 dummy group

Abstract

A method and system for executing a query in parallel is disclosed. A master node may receive a query from a client and develop query plans from that query. The query plans may be forwarded to worker nodes for execution, and each query plan may be accompanied by query metadata. The metadata may be stored in a catalog on the master node.

Claims

1. A method for executing queries in a parallel processing database system, comprising:

receiving a query at a master node, the master node comprising a database catalog including metadata defining database objects;

in response to receiving the query at the master node, initiating a catalog server session, taking a snapshot of the metadata, and associating the snapshot of the metadata with the catalog server session, wherein a separate catalog server session is initiated and a separate snapshot of the metadata is taken for each query;

transmitting a query plan and query metadata to a worker node, wherein the query plan is based on the query, wherein the query metadata includes metadata for executing the query plan, wherein the query metadata includes database table definitions that define database objects, and wherein the query metadata is retrieved from the snapshot of the metadata associated with the catalog server session;

receiving, by the master node, a request for additional metadata that is required for the worker node to execute the query plan, wherein in response to a determination that the worker node requires the additional metadata for the worker node to execute the query plan, the worker node queries a parent in a tree structure of a plurality of worker nodes in the parallel processing database system for the additional metadata, wherein the parent node is a node between the master node and the worker node in relation to the tree structure;

communicating, to the worker node, the additional metadata that is required for the worker node to execute the query plan, wherein the additional metadata is retrieved from a same session as the catalog server session corresponding to the query;

executing the query plan on the worker node; and

returning, to the master node, a result associated with the execution of the query plan on the worker node.

2. The method of claim 1, further comprising generating a plurality of query plans based on the query.

3. The method of claim 2, further comprising transmitting the plurality query plans to a plurality of worker nodes.

4. The method of claim 3, further comprising executing the plurality of query plans in parallel.

5. The method of claim 1, further comprising storing the query metadata in a cache on the worker node.

6. The method of claim 5, further comprising clearing the cache after executing the query plan.

7. The method of claim 5, further comprising retrieving the query metadata from the cache while executing the query plan.

8. The method of claim 1, further comprising transmitting a request for the additional metadata from the worker node to the master node.

9. The method of claim 8, further comprising transmitting the additional metadata to the worker node.

10. The method of claim 1, further comprising:

compiling the result associated with the execution of the query plan on the worker node with another result associated with the query; and

returning the compiled result and other result as a final query result to a client.

11. The method of claim 1, wherein the query metadata includes one or more of a user defined database function, a system defined database function, a database view, and a database index.

12. The method of claim 1, further comprising:

receiving another query at the master node;

in response to receiving the other query, initiating another catalog server session, taking, another snapshot of the metadata as the metadata existed when the other catalog server session is initiated, and associating the other snapshot of the metadata with the other catalog server session; and

transmitting another query plan and other query metadata to the worker node, wherein the other query metadata is retrieved from the snapshot of the metadata associated with the other catalog server session.

13. The method of claim 1, further comprising:

in response to receiving the query plan and the query metadata, determining, by the worker node, that additional metadata is required for the worker node to execute the query plan;

in response to determining that additional metadata is required, requesting, by the worker node, the additional metadata; and

receiving the additional metadata, wherein the additional metadata is retrieved from the snapshot of the metadata associated with the catalog server session.

14. The method of claim 1, wherein in the event that additional metadata is required in connection with the query, the additional metadata is obtained from the snapshot of the metadata corresponding to the query.

15. The method of claim 1, wherein the querying a parent in the tree structure of a plurality of worker nodes comprises successively querying one or more parent nodes for the additional metadata before querying the master node for the additional metadata.

16. The method of claim 15, wherein the successively querying one or more parent nodes for the additional metadata before querying the master node for the additional metadata comprises at least one of the one or more parent nodes forwarding a request for the additional metadata to another of the one or more parent nodes.

17. The method of claim 1, wherein the request for additional metadata is communicated to the master node in response to the worker node determining that the worker node requires the additional metadata to execute the query plan.

18. A computer program product for executing queries in a parallel processing database system, comprising a non-transitory computer readable medium having program instructions embodied therein for:

receiving a query at a master node, the master node comprising a database catalog including metadata defining database objects;

in response to receiving the query at the master node, initiating a catalog server session, taking a snapshot of the metadata, and associating the snapshot of the metadata with the catalog server session, wherein a separate catalog server session is initiated and a separate snapshot of the metadata is taken for each query;

transmitting a query plan and query metadata to a worker node, wherein the query plan is based on the query, wherein the query metadata includes metadata for executing the query plan, wherein the query metadata includes database table definitions that define database objects, and wherein the query metadata is retrieved from the snapshot of the metadata associated with the catalog server session;

receiving, by the master node, a request for additional metadata that is required for the worker node to execute the query plan, wherein in response to a determination that the worker node requires the additional metadata for the worker node to execute the query plan, the worker node queries a parent in a tree structure of a plurality of worker nodes in the parallel processing database system for the additional metadata, wherein the parent node is a node between the master node and the worker node in relation to the tree structure;

communicating, to the worker node, the additional metadata that is required for the worker node to execute the query plan, wherein the additional metadata is retrieved from a same session as the catalog server session corresponding to the query;

executing the query plan on the worker node; and

returning, to the master node, a result associated with the execution of the query plan on the worker node.

19. The computer program of claim 18, further comprising instructions for storing the query metadata in a cache on the worker node.

20. The computer program of claim 19, further comprising instructions for clearing the cache after executing the query plan.

21. The computer program of claim 19, further comprising instructions for retrieving the query metadata from the cache while executing the query plan.

22. The computer program of claim 18, further comprising instructions for transmitting a request for the additional metadata from the worker node to the master node.

23. The computer program of claim 22, further comprising instructions for transmitting the additional metadata to the worker node.

24. A system for executing queries in a parallel processing database, comprising a non-transitory computer readable medium and a processor configured to:

receive a query at a master node, the master node comprising a database catalog including metadata defining database objects;

in response to receiving the query at the master node, initiating a catalog server session, taking a snapshot of the metadata, and associating the snapshot of the metadata with the catalog server session, wherein a separate catalog server session is initiated and a separate snapshot of the metadata is taken for each query;

transmit a query plan and query metadata to a worker node, wherein the query plan is based on the query, wherein the query metadata includes metadata for executing the query plan, wherein the query metadata includes database table definitions that define database objects, and wherein the query metadata is retrieved from the snapshot of the metadata associated with the catalog server session;

receive, by the master node, a request for additional metadata that is required for the worker node to execute the query plan, wherein in response to a determination that the worker node requires the additional metadata for the worker node to execute the query plan, the worker node queries a parent in a tree structure of a plurality of worker nodes in the parallel processing database system for the additional metadata, wherein the parent node is a node between the master node and the worker node in relation to the tree structure;

communicate, to the worker node, the additional metadata that is required for the worker node to execute the query plan, wherein the additional metadata is retrieved from a same session as the catalog server session corresponding to the query;

execute the query plan on the worker node; and

return, to the master node, a result associated with the execution of the query plan on the worker node.

25. The system of claim 24, wherein the processor is further configured to generate a plurality of query plans based on the query.

26. The system of claim 25, wherein the processor is further configured to transmit the plurality query plans to a plurality of worker nodes.

27. The system of claim 26, wherein the processor is further configured to execute the plurality of query plans in parallel.

28. The system of claim 24, wherein the processor is further configured to store the query metadata in a cache on the worker node.