Managing data queries

• US 9,576,028 B2
• Filed: 02/23/2015
• Issued: 02/21/2017
• Est. Priority Date: 05/02/2011
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Schechter, Ian, Allin, Glenn John
• Primary Examiner(s): BROWN, SHEREE N
• Application Number:
  US14/628,643
• Publication Number:
  US 20150169687A1
• Time in Patent Office: 729 Days
• Field of Search: 707/718
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2428 Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: G06F 16/24526 Internal representations for queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence

Abstract

In one aspect, in general, a method of generating a dataflow graph representing a database query includes receiving a query plan from a plan generator, the query plan representing operations for executing a database query on at least one input representing a source of data, producing a dataflow graph from the query plan, wherein the dataflow graph includes at least one node that represents at least one operation represented by the query plan, and includes at least one link that represents at least one dataflow associated with the query plan, and altering one or more components of the dataflow graph based on at least one characteristic of the at least one input representing the source of data.

Claims

1. A computer-implemented method of operating on data from at least two data sources, the data stored in respective tangible, non-transitory computer-readable media of the data sources, the operating based on a query that is expressed in accordance with a query language applicable to a relational database, the operating including executing the query on an executing system other than a system managing a relational database, the executing including:

receiving a query;

identifying a first data source based on the query;

identifying a second data source based on the query, the second data source being associated with at least some data processing capability different from at least some data processing capability of the first data source;

identifying an executing system other than a system managing a relational database;

generating a request to a query planner based on the query;

providing the request to the query planner;

receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps for performance by a system managing a relational database;

generating a computer program that includes at least one data processing operation for execution, the at least one data processing operation including at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation, wherein the at least one data processing operation is chosen based on a particular step described by the query plan, and the particular step corresponds to the at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation;

receiving data from the first data source and the second data source; and

executing, on the identified executing system, program code based on the computer program.

2. The method of claim 1 in which generating the computer program includes:

generating, based on the query plan, a data structure instantiating a dataflow graph that includes;

a first node that represents at least one operation to be executed,

the first node associated with information usable by an executing system to invoke executable program code to perform the operation,

the first node associated with information usable by an executing system to make data available to the program code, and

the operation being chosen based on a first step described by the query plan, and

at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph.

3. The method of claim 2 in which the dataflow graph includes a third node associated with information usable by an executing system to receive the data from the first data source, and a fourth node associated with information usable by an executing system to receive the data from the second data source.

4. The method of claim 3 in which the third node is associated with information usable by an executing system to invoke executable program code that causes a request to perform at least one operation chosen based on a second step described by the query plan to be transmitted to a system managing the first data source.

5. The method of claim 4 including determining that the system managing the first data source is capable of performing the at least one operation chosen based on the second step described by the query plan.

6. The method of claim 4 including determining that a system managing the second data source is not capable of performing the at least one operation chosen based on the second step described by the query plan.

7. The method of claim 2 in which executing the program code based on the dataflow graph includes instantiating a link between a node of the dataflow graph and a node of a second dataflow graph, the second dataflow graph including at least one node that represents an operation to be executed other than an operation chosen based on any step described by the query plan.

8. The method of claim 2 in which the query contains a reference to an executable function, and the dataflow graph includes a node representing the executable function.

9. The method of claim 1 in which the first data source includes a relational database table.

10. The method of claim 9 in which the second data source includes a portion of data other than a relational database table.

11. The method of claim 9 in which the second data source includes a flat file.

12. The method of claim 1 in which the query includes an SQL query.

13. The method of claim 1, including receiving, as output of the program code, results of the query.

14. The method of claim 1 in which the query plan generator includes a query optimizer.

15. The method of claim 1 in which the query plan includes data types of parameters in the query.

16. The method of claim 1 in which the first data source is associated with executable functionality for performing operations on data stored by the first data source and the second data source is not associated with executable functionality for performing operations on data stored by the second data source.

17. The method of claim 1 in which generating a request to a query planner based on the query includes:

producing a structural representation of the first data source wherein the structural representation includes a formatted data organization reflective of the first data source and is absent data of the first data source; and

including the structural representation in the request to the query planner.

18. The method of claim 1 in which the computer program is generated by the identified executing system.

19. The method of claim 1 in which the computer program is generated by a system other than the identified executing system.

35. The method of claim 1, wherein the computer program includes at least one data processing operation chosen based on some data processing capability of the second data source or some data processing capability of the first data source.

36. The method of claim 1, wherein data processing capability includes executable functionality configured to carry out at least one data processing instruction.

37. The method of claim 1, wherein the combination operation includes a join operation.

38. The method of claim 1, wherein the data summarization operation includes an aggregation operation.

20. A system, including one or more processors, for operating on data from at least two data sources, the data stored in respective tangible, non-transitory computer-readable media of the data sources, the operating based on a query that is expressed in accordance with a query language applicable to a relational database, the system configured to execute operations for:

receiving a query;

identifying a first data source based on the query;

identifying a second data source based on the query, the second data source being associated with at least some data processing capability different from at least some data processing capability of the first data source;

identifying an executing system other than a system managing a relational database;

generating a request to a query planner based on the query;

providing the request to the query planner;

receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps for performance by a system managing a relational database;

generating a computer program that includes at least one data processing operation for execution, the at least one data processing operation including at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation, wherein the at least one data processing operation is chosen based on a particular step described by the query plan, and the particular step corresponds to the at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation;

receiving data from the first data source and the second data source; and

executing, on the identified executing system, program code based on the computer program.

21. The system of claim 20 in which generating the computer program includes:

generating, based on the query plan, a data structure instantiating a dataflow graph that includes;

a first node that represents at least one operation to be executed,

the first node associated with information usable by an executing system to invoke executable program code to perform the operation,

the first node associated with information usable by an executing system to make data available to the program code, and

the operation being chosen based on a first step described by the query plan, and

at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph.

22. The system of claim 20 in which the first data source includes a relational database table.

23. The system of claim 22 in which the second data source includes a portion of data other than a relational database table.

24. The system of claim 22 in which the second data source includes a flat file.

25. The system of claim 20, the operations including receiving, as output of the program code, results of the query.

26. The system of claim 20 in which the first data source is associated with executable functionality for performing operations on data stored by the first data source and the second data source is not associated with executable functionality for performing operations on data stored by the second data source.

27. A non-transitory computer-readable medium including computer-executable instructions for operating on data from at least two data sources, the data stored in respective tangible, non-transitory computer-readable media of the data sources, the operating based on a query that is expressed in accordance with a query language applicable to a relational database, the operating including executing the query on an executing system other than a system managing a relational database, the executable instructions causing a computer to carry out steps including:

receiving a query;

identifying a first data source based on the query;

identifying a second data source based on the query, the second data source being associated with at least some data processing capability different from at least some data processing capability of the first data source;

identifying an executing system other than a system managing a relational database;

generating a request to a query planner based on the query;

providing the request to the query planner;

receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps for performance by a system managing a relational database;

generating a computer program that includes at least one data processing operation for execution, the at least one data processing operation including at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation, wherein the at least one data processing operation is chosen based on a particular step described by the query plan, and the particular step corresponds to the at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation;

receiving data from the first data source and the second data source; and

executing, on the identified executing system, program code based on the computer program.

28. The computer-readable medium of claim 27 in which generating the computer program includes:

generating, based on the query plan, a data structure instantiating a dataflow graph that includes;

a first node that represents at least one operation to be executed,

the first node associated with information usable by an executing system to invoke executable program code to perform the operation,

the first node associated with information usable by an executing system to make data available to the program code, and

the operation being chosen based on a first step described by the query plan, and

at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph.

29. The computer-readable medium of claim 27 in which the first data source includes a relational database table.

30. The computer-readable medium of claim 29 in which the second data source includes a portion of data other than a relational database table.

31. The computer-readable medium of claim 29 in which the second data source includes a flat file.

32. The computer-readable medium of claim 27, the operations including receiving, as output of the program code, results of the query.

33. The computer-readable medium of claim 27 in which the first data source is associated with executable functionality for performing operations on data stored by the first data source and the second data source is not associated with executable functionality for performing operations on data stored by the second data source.

34. A system, including one or more processors, for operating on data from at least two data sources, the data stored in respective tangible, non-transitory computer-readable media of the data sources, the operating based on a query that is expressed in accordance with a query language applicable to a relational database, the system including:

means for receiving a query;

means for identifying a first data source based on the query;

means for identifying a second data source based on the query, the second data source being associated with at least some data processing capability different from at least some data processing capability of the first data source;

means for identifying an executing system other than a system managing a relational database;

means for generating a request to a query planner based on the query;

means for providing the request to the query planner;

means for receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps for performance by a system managing a relational database;

means for generating a computer program that includes at least one data processing operation for execution, the at least one data processing operation including at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation, wherein the at least one data processing operation is chosen based on a particular step described by the query plan, and the particular step corresponds to the at least one of a data selection operation, a sort operation, a data summarization operation, or a combination operation;

means for receiving data from the first data source and the second data source; and

means for executing, on the identified executing system, program code based on the computer program.