SYSTEM AND METHOD FOR QUERYING HYBRID MULTI DATA SOURCES

• US 20150169686A1
• Filed: 12/13/2013
• Published: 06/18/2015
• Est. Priority Date: 12/13/2013
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Elis, Filip, Nguyen, Filip
• Granted Patent: US 9,372,891 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination

Abstract

A system and method of querying hybrid multi data sources includes receiving a query at a hybrid multi data source running on a computer server, creating a list of data sources, filtering the list of data sources based on data source hints included in the query, filtering the list of data sources based on data source rules, forming one or more query plans for processing the query, evaluating each of the query plans, selecting a first query plan from the query plans based on results of the evaluating, and performing the first query plan. Each query plan corresponds to a data source in the filtered list of data sources. In some examples, the first query plan has a lowest cost of each of the query plans. In some examples, the first query plan is based on probabilities associated with the cost of each of the query plans.

Claims

1. A method of processing a data source query, the method comprising:

receiving the data source query at a hybrid multi data source running on a computer server;

creating a list of data sources;

filtering the list of data sources based on data source hints included in the data source query;

filtering the list of data sources based on data source rules;

forming one or more query plans for processing the data source query, each query plan corresponding to a data source in the filtered list of data sources;

evaluating each of the query plans;

selecting a first query plan from the query plans based on results of the evaluating; and

performing the first query plan.

2. The method of claim 1 wherein each of the list of data sources has a respective type selected from a group consisting of a relational data base, a nosql database, a flat file, an XML file, a representational state transfer (REST) web service, and a Simple Object Access Protocol (SOAP) web service.

3. The method of claim 1, further comprising extracting the data source hints from a source type clause in the data source query.

4. The method of claim 1, further comprising evaluating each of the data source rules to identify one or more preferred data source types based on features of the data source query.

5. The method of claim 4, further comprising omitting the filtering of the list of data sources based on the data source rules when there is no overlap between the preferred data source types and first data source types associated with data sources in the list of data sources.

6. The method of claim 4 wherein filtering the list of data sources based on the data source rules comprises removing any data sources from the list of data sources that do not have a first data source type associated with the preferred data source types.

7. The method of claim 1, further comprising omitting the filtering of the list of data sources based on the data source rules when there is an overlap between first data source types included in the data source hints and second data source types associated with data sources in the list of data sources.

8. The method of claim 1, further comprising omitting the filtering of the list of data sources based on the data source hints when there is no overlap between first data source types included in the data source hints and second data source types associated with data sources in the list of data sources.

9. The method of claim 1 wherein filtering the list of data sources based on the data source hints comprises removing any data sources from the list of data sources that do not have a first data source type associated with second data source types included in the data source hints.

10. The method of claim 1 wherein:

each of the query plans is associated with a respective query plan tree comprising one or more nodes; and

evaluating each of the query plans comprises adding a node cost associated with a type of each of the nodes in the respective query plan tree to a query plan cost for that query plan.

11. The method of claim 10 wherein when one of the nodes is a join node, the node cost for the join node is based on an aggregation of a cost of each sub-query plan associated with children of the join node.

12. The method of claim 1 wherein selecting the first query plan based on the results of the evaluating comprises selecting the first query plan from the query plans that has a lowest cost.

13. The method of claim 1 wherein selecting the first query plan based on the results of the evaluating comprises selecting the first query plan based on probabilities associated with costs of each of the query plans.

14. The method of claim 1, further comprising selecting a second query plan from the query plans when the first query plan cannot be performed.

15. A hybrid multi data source hosted in an application server comprising:

a hint resolver;

a rule evaluator coupled to the hint resolver; and

a federated query engine coupled to the hint resolver and the rule evaluator;

wherein;

the hybrid multi data source is configured to;

receive a query; and

create a list of data sources coupled to the hybrid multi data source;

the hint resolver is configured to;

receive the list of data sources from the hybrid multi data source; and

filter the list of data sources based on hints included in the query;

the rule evaluator is configured to;

receive the list of data sources from the hybrid multi data source or the hint resolver;

receive the query from the hybrid multi data source; and

filter the list of data sources based on data source rules;

the federated query engine is configured to;

receive the filtered list of data sources from the hint resolver or the rule evaluator;

receive the query from the hybrid multi data source;

form one or more query plans for processing the query, each query plan corresponding to a data source in the filtered list of data sources;

evaluate each of the query plans to determine a respective estimated cost to perform each of the query plans;

select a query plan from the query plans based on each respective estimated cost; and

perform the selected query plan.

16. The hybrid multi data source of claim 15 wherein the selected query plan has a lowest estimated cost from among each of the query plans.

17. The hybrid multi data source of claim 15 wherein the selected query plan is selected based on probabilities associated with estimated costs of the query plans.

18. A non-transitory machine-readable medium comprising a first plurality of machine-readable instructions which when executed by one or more processors associated with an application server are adapted to cause the one or more processors to perform a method comprising:

receiving a data query at a hybrid multi data source running on the application server;

creating a list of data sources coupled to the hybrid multi data source;

filtering the list of data sources based on source hints included in the data query;

filtering the list of data sources based on source rules;

forming one or more query plans for processing the data query, each query plan corresponding to a data source in the filtered list of data sources;

evaluating each of the query plans to determine a respective estimated query complexity for each of the query plans;

selecting a query plan from the query plans based on each respective estimated query complexity; and

performing the selected query plan.

19. The non-transitory machine-readable medium of claim 18, further comprising a second plurality of machine-readable instructions which when executed by the one or more processors are adapted to cause the one or more processors to perform a method comprising:

selecting the query plan from the query plans that has a lowest estimated query complexity.

20. The non-transitory machine-readable medium of claim 18, further comprising a second plurality of machine-readable instructions which when executed by the one or more processors are adapted to cause the one or more processors to perform a method comprising:

selecting the query plan from the query plans based on probabilities associated with estimated query complexities of the query plans.