Source query caching as fault prevention for federated queries

• US 10,114,874 B2
• Filed: 02/24/2014
• Issued: 10/30/2018
• Est. Priority Date: 02/24/2014
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Nguyen, Filip, Elia{hacek over (s)}, Filip
• Primary Examiner(s): Le, Miranda
• Application Number:
  US14/188,279
• Publication Number:
  US 20150242464A1
• Time in Patent Office: 1,709 Days
• Field of Search: 707713
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/1474 in transactions G06F16/20 takes precedence
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/2471 Distributed queries

Abstract

An example system for processing a federated query includes a query proxy that receives a federated query including a plurality of source queries and receives an indication that a failed set of one or more source queries failed to execute successfully. Each source query is specific to an autonomous data source belonging to a federation. The system also includes a data federation engine that identifies a plurality of autonomous data sources to which to send the plurality of source queries. The plurality of autonomous data sources belong to the federation. The system further includes a query fail analyzer that updates a data structure to reflect the unsuccessful execution of one or more source queries of the failed set.

Claims

1. A method of processing a federated query, comprising:

receiving an indication that a first set of source queries embedded in a first federated query failed to execute successfully, each source query specifying a set of source tables stored in a target autonomous data source of a plurality of target autonomous data sources belonging to a federation, at least two source queries of the first set of source queries being specific to different data sources, and the first federated query being sent from a client;

storing the first set of source queries and metadata associated with the first set of source queries into a data structure, the first set of source queries including a first source query;

for each source query of the first set of source queries that is determined to be stored in the data structure, updating metadata of each entry corresponding to the respective source query stored in the data structure, the metadata including a number of times the respective source query has failed and further including a timestamp of the respective failure;

selecting a second set of source queries from the data structure, the second set of source queries including the first source query and having a higher probability of failure than a third set of source queries stored in the data structure;

submitting the second set of source queries to one or more target data sources;

for each result of a source query of the second set of source queries, storing the result in a cache external to the federation;

receiving an indication that the first source query embedded in a second federated query failed to execute successfully, the second federated query including a second source query, and the first source query specifying a first set of source tables stored in a first target autonomous data source;

generating a third source query by replacing a first set of source table names included in the first source query with a second set of source table names that identifies a second set of source tables, the second set of source tables being stored in the cache and storing data cached from the first set of source tables, and the first set of source table names being different from the second set of source table names;

generating a third federated query including the second and third source queries;

submitting each source query embedded in the third federated query to one or more data sources, the third source query specifying the second set of source tables and being submitted to the cache; and

sending a combined result set responsive to the third federated query to the client, the combined result set including a first result set responsive to the second source query and further including a cached result set responsive to the third source query, and the cached result set being stored in the cache.

2. The method of claim 1, further including:

determining whether the first set of source queries is stored in the data structure,

wherein for each source query of the first set of source queries that is determined to not be stored in the data structure, the storing the first set of source queries includes inserting into the data structure an entry including the respective source query and first metadata of the respective source query, the first metadata including a federated query identifier that identifies the first federated query and a data source identifier that identifies a target data source to which the respective source query was sent.

3. The method of claim 1, wherein for each source query of the second set of source queries, the method further includes:

identifying a source table included in the respective source query;

determining a cache table name based on a name of the identified source table, wherein the storing includes storing the result in a table identified by the cache table name.

4. The method of claim 3, wherein the determining a cache table name includes identifying a source table name of the identified source table and adding a suffix to the source table name.

5. The method of claim 3, wherein the generating a third federated query includes:

identifying one or more source queries embedded in the second federated query that is specific to an unavailable data source;

rewriting each of the identified one or more source queries embedded in the second federated query; and

replacing the identified one or more source queries in the second federated query with one or more corresponding rewritten source queries.

6. The method of claim 5, wherein the rewriting includes replacing one or more source table names included in the identified one or more source queries with one or more source table names included in the cache.

7. The method of claim 6, wherein the replacing one or more source table names includes appending a suffix to the respective source table name.

8. The method of claim 1, wherein the metadata includes a fail count for each failed source query.

9. The method of claim 1, wherein the metadata includes a fail count for each failed source query.

10. The method of claim 9, wherein the second set of source queries has incurred more failures than the third set of source queries.

11. The method of claim 1, wherein the second federated query is different from the third federated query.

12. The method of claim 11, wherein the third federated query includes one or more source queries included in the second federated query.

13. A system for processing a federated query, comprising:

a non-transitory memory; and

one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising;

receiving an indication that a first set of source queries embedded in a first federated query failed to execute successfully, each source query specifying a set of source tables stored in a target autonomous data source of a plurality of target autonomous data sources belonging to a federation, at least two source queries of the first set of source queries being specific to different data sources, and the first federated query being sent from a client;

storing the first set of source queries and metadata associated with the first set of source queries into a data structure, the first set of source queries including a first source query;

for each source query of the first set of source queries that is determined to be stored in the data structure, updating metadata of each entry corresponding to the respective source query stored in the data structure, the metadata including a number of times the respective source query has failed and further including a timestamp of the respective failure;

selecting a second set of source queries from the data structure, the second set of source queries including the first source query and having a higher probability of failure than a third set of source queries stored in the data structure;

submitting the second set of source queries to one or more target data sources;

for each result of a source query of the second set of source queries, storing the result in a cache external to the federation;

receiving an indication that the first source query embedded in a second federated query failed to execute successfully, the second federated query including a second source query, and the first source query specifying a first set of source tables stored in a first target autonomous data source;

generating a third source query by replacing a first set of source table names included in the first source query with a second set of source table names that identifies a second set of source tables, the second set of source tables being stored in the cache and storing data cached from the first set of source tables, and the first set of source table names being different from the second set of source table names;

generating a third federated query including the second and third source queries;

submitting each source query embedded in the third federated query to one or more data sources, the third source query specifying the second set of source tables and being submitted to the cache; and

sending a combined result set responsive to the third federated query to the client, the combined result set including a first result set responsive to the second source query and further including a cached result set responsive to the third source query, and the cached result set being stored in the cache.

14. The system of claim 13, wherein the operations further include determining whether the first set of source queries is stored in the data structure,

wherein for each source query of the first set of source queries that is determined to not be stored in the data structure, the storing the first set of source queries includes inserting into the data structure an entry including the respective source query and first metadata of the respective source query, the first metadata including a federated query identifier that identifies the first federated query and a data source identifier that identifies a target data source to which the respective source query was sent.

15. The system of claim 13, wherein for each source query of the second set of source queries, the method further includes:

identifying a source table included in the respective source query;

determining a cache table name based on a name of the identified source table, wherein the storing includes storing the result in a table identified by the cache table name.

16. The system of claim 15, wherein the determining a cache table name includes identifying a source table name of the identified source table and adding a suffix to the source table name.

17. The system of claim 16, wherein the third federated query includes one or more rewritten names of source tables included in the second federated query, and wherein the one or more rewritten names of source tables corresponds to information stored in tables that are unavailable.

18. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions that when executed by one or more processors is adapted to cause the one or more processors to perform a method comprising:

receiving an indication that a first set of source queries embedded in a first federated query failed to execute successfully, each source query specifying a set of source tables stored in a target autonomous data source of a plurality of target autonomous data sources belonging to a federation, at least two source queries of the first set of source queries being specific to different data sources, and the first federated query being sent from a client;

storing the first set of source queries and metadata associated with the first set of source queries into a data structure, the first set of source queries including a first source query;

for each source query of the first set of source queries that is determined to be stored in the data structure, updating metadata of each entry corresponding to the respective source query stored in the data structure, the metadata including a number of times the respective source query has failed and further including a timestamp of the respective failure;

selecting a second set of source queries from the data structure, the second set of source queries including the first source query and having a higher probability of failure than a third set of source queries stored in the data structure;

submitting the second set of source queries to one or more target data sources;

for each result of a source query of the second set of source queries, storing the result in a cache external to the federation;

receiving an indication that the first source query embedded in a second federated query failed to execute successfully, the second federated query including a second source query, and the first source query specifying a first set of source tables stored in a first target autonomous data source;

generating a third source query by replacing a first set of source table names included in the first source query with a second set of source table names that identifies a second set of source tables, the second set of source tables being stored in the cache and storing data cached from the first set of source tables, and the first set of source table names being different from the second set of source table names;

generating a third federated query including the second and third source queries;

submitting each source query embedded in the third federated query to one or more data sources, the third source query specifying the second set of source tables and being submitted to the cache; and

sending a combined result set responsive to the third federated query to the client, the combined result set including a first result set responsive to the second source query and further including a cached result set responsive to the third source query, and the cached result set being stored in the cache.