System and method for querying hybrid multi data sources

• US 9,372,891 B2
• Filed: 12/13/2013
• Issued: 06/21/2016
• Est. Priority Date: 12/13/2013
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Eli, Filip, Nguyen, Filip
• Primary Examiner(s): Nguyen, Cam-Linh
• Application Number:
  US14/106,488
• Publication Number:
  US 20150169686A1
• Time in Patent Office: 921 Days
• Field of Search: 707/718, 707/722, 707/743
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination

Abstract

A system and method of querying hybrid multi data sources includes receiving a query at a hybrid multi data source running on a computer server, creating a list of data sources, filtering the list of data sources based on data source hints included in the query, filtering the list of data sources based on data source rules, forming one or more query plans for processing the query, evaluating each of the query plans, selecting a first query plan from the query plans based on results of the evaluating, and performing the first query plan. Each query plan corresponds to a data source in the filtered list of data sources. In some examples, the first query plan has a lowest cost of each of the query plans. In some examples, the first query plan is based on probabilities associated with the cost of each of the query plans.

Claims

1. A method of processing a data source query, the method comprising:

receiving the data source query at a hybrid multi data source running on a computer server;

creating a list of data sources, each data source in the list of data sources having a data source type selected from a group consisting of a relational data base, a nosql database, a flat file, an XML file, a representational state transfer (REST) web service, and a Simple Object Access Protocol (SOAP) web service;

removing data sources from the list of data sources based on the data source type of each of the data sources in the list of data sources and data source type hints included in the data source query;

removing data sources from the list of data sources based on the data source type of each of the data sources in the list of data sources and data source rules identifying preferred data source types based on features of the data source query;

forming one or more query plans for processing the data source query, each query plan corresponding to a data source in the list of data sources after the removings;

evaluating each of the query plans;

selecting a first query plan from the query plans based on results of the evaluating; and

performing the first query plan.

2. The method of claim 1, further comprising extracting the data source hints from a source type clause in the data source query.

3. The method of claim 1, further comprising omitting the removing of data sources from the list of data sources based on the data source rules when there is no overlap between the preferred data source types and data source types of the data sources in the list of data sources.

4. The method of claim 1 wherein removing data sources from the list of data sources based on the data source rules comprises removing each of the data sources from the list of data sources that do not have a data source type in the preferred data source types.

5. The method of claim 1, further comprising omitting the removing of data sources from the list of data sources based on the data source rules when there is an overlap between first data source types included in the data source type hints and second data source types of the data sources in the list of data sources.

6. The method of claim 1, further comprising omitting the removing of data sources from the list of data sources based on the data source type hints when there is no overlap between first data source types included in the data source type hints and second data source types of the data sources in the list of data sources.

7. The method of claim 1 wherein removing data sources from the list of data sources based on the data source type hints comprises removing any data sources from the list of data sources that do not have a first data source type associated with second data source types included in the data source type hints.

8. The method of claim 1 wherein:

each of the query plans is associated with a respective query plan tree comprising one or more nodes; and

evaluating each of the query plans comprises adding a node cost associated with a type of each of the nodes in the respective query plan tree to a query plan cost for that query plan.

9. The method of claim 8 wherein when one of the nodes is a join node, the node cost for the join node is based on an aggregation of a cost of each sub-query plan associated with children of the join node.

10. The method of claim 1 wherein selecting the first query plan based on the results of the evaluating comprises selecting the first query plan from the query plans that has a lowest cost.

11. The method of claim 1 wherein selecting the first query plan based on the results of the evaluating comprises selecting the first query plan based on probabilities associated with costs of each of the query plans.

12. The method of claim 1, further comprising selecting a second query plan from the query plans when the first query plan cannot be performed.

13. An application server comprising:

memory; and

one or more processors coupled to the memory;

wherein the application server hosts a hybrid multi data source comprising;

a hint resolver;

a rule evaluator coupled to the hint resolver; and

a federated query engine coupled to the hint resolver and the rule evaluator;

wherein;

the hybrid multi data source is configured to;

receive a query; and

create a list of data sources coupled to the hybrid multi data source, each data source in the list of data sources having a data source type selected from a group consisting of a relational data base, a nosql database, a flat file, an XML file, a representational state transfer (REST) web service, and a Simple Object Access Protocol (SOAP) web service;

the hint resolver is configured to;

receive the list of data sources from the hybrid multi data source; and

remove data sources from the list of data sources based on the data source type of each of the data sources in the list of data sources and data source type hints included in the query;

the rule evaluator is configured to;

receive the list of data sources after the removings from the hybrid multi data source or the hint resolver;

receive the query from the hybrid multi data source; and

remove data sources from the list of data sources based on the data source type of each of the data sources in the list of data sources after the removings and data source rules identifying preferred data source types based on features of the data source query;

the federated query engine is configured to;

receive the filtered list of data sources from the hint resolver or the rule evaluator;

receive the query from the hybrid multi data source;

form one or more query plans for processing the query, each query plan corresponding to a data source in the filtered list of data sources;

evaluate each of the query plans to determine a respective estimated cost to perform each of the query plans;

select a query plan from the query plans based on each respective estimated cost; and

perform the selected query plan.

14. The hybrid multi data source of claim 13 wherein the selected query plan has a lowest estimated cost from among each of the query plans.

15. The hybrid multi data source of claim 13 wherein the selected query plan is selected based on probabilities associated with estimated costs of the query plans.

16. A non-transitory machine-readable medium comprising a first plurality of machine-readable instructions which when executed by one or more processors associated with an application server are adapted to cause the one or more processors to perform a method comprising:

receiving a data query at a hybrid multi data source running on the application server;

creating a list of data sources coupled to the hybrid multi data source, each data source in the list of data sources having a data source type selected from a group consisting of a relational data base, a nosql database, a flat file, an XML file, a representational state transfer (REST) web service, and a Simple Object Access Protocol (SOAP) web service;

removing data sources from the list of data sources based on the data source type of each of the data sources in the list of data sources and data source type hints included in the data query;

removing data sources from the list of data sources based on the data source type of each of the data sources in the list of data sources and data source rules identifying preferred data source types based on features of the data query;

forming one or more query plans for processing the data query, each query plan corresponding to a data source in the list of data sources after the removings;

evaluating each of the query plans to determine a respective estimated query complexity for each of the query plans;

selecting a query plan from the query plans based on each respective estimated query complexity; and

performing the selected query plan.

17. The non-transitory machine-readable medium of claim 16, further comprising a second plurality of machine-readable instructions which when executed by the one or more processors are adapted to cause the one or more processors to perform a method comprising:

selecting the query plan from the query plans that has a lowest estimated query complexity.

18. The non-transitory machine-readable medium of claim 16, further comprising a second plurality of machine-readable instructions which when executed by the one or more processors are adapted to cause the one or more processors to perform a method comprising:

selecting the query plan from the query plans based on probabilities associated with estimated query complexities of the query plans.