Managing data queries

• US 9,665,620 B2
• Filed: 01/15/2010
• Issued: 05/30/2017
• Est. Priority Date: 01/15/2010
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Wholey, J. Skeffington, Schechter, Ian, Allin, Glenn John
• Primary Examiner(s): Vu, Bai D.
• Application Number:
  US12/688,316
• Publication Number:
  US 20110179014A1
• Time in Patent Office: 2,692 Days
• Field of Search: None
• CPC Class Codes: G06F 16/2425 Iterative querying; Query formulation based on the results of a preceding query
• CPC Class Codes: G06F 16/2433 Query languages
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/7343 Query language or query format

Abstract

One method includes receiving a database query, receiving information about a database table in data storage populated with data elements, producing a structural representation of the database table that includes a formatted data organization reflective of the database table and is absent the data elements of the database table, and providing the structural representation and the database query to a plan generator capable of producing a query plan representing operations for executing the database query on the database table. Another method includes receiving a query plan from a plan generator, the plan representing operations for executing a database query on a database table, and producing a dataflow graph from the query plan, wherein the dataflow graph includes at least one node that represents at least one operation represented by the query plan, and includes at least one link that represents at least one dataflow associated with the query plan.

Claims

1. A method of operating on data from a data source, the data stored in a tangible, non-transitory computer-readable medium of the data source, the operating based on a query that is expressed in accordance with a query language applicable to a relational database, the operating including:

receiving a query;

identifying a data source based on the query;

identifying a requesting system other than a system managing a relational database storing data of the data source;

generating, by the requesting system, a request to a query planner based on the query;

providing, by the requesting system, the request to the query planner;

receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps to be performed by a system managing a relational database;

generating a data structure instantiating a dataflow graph that includes;

a first node that represents at least one operation to be executed,

the first node associated with information usable by an executing system to invoke executable program code to perform the at least one operation,

the first node associated with information usable by an executing system to make data available to the program code, and

the at least one operation chosen based on a step described by the query plan, and

at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph;

receiving data from the data source;

identifying an executing system other than a system managing a relational database storing data of the data source; and

executing, on the identified executing system, program code based on the dataflow graph.

2. The method of claim 1, wherein the dataflow graph includes a third node that represents an operation to be executed other than an operation chosen based on any step described by the query plan.

3. The method of claim 1, wherein executing the program code based on the dataflow graph includes instantiating a link between a node of the dataflow graph and a node of a second dataflow graph, the second dataflow graph including at least one node that represents an operation to be executed other than an operation chosen based on any step described by the query plan.

4. The method of claim 1, wherein the data source includes a relational database table.

5. The method of claim 1, wherein the data source includes a portion of data other than a relational database table.

6. The method of claim 1, wherein the data source includes a flat file.

7. The method of claim 1, wherein the query includes an SQL query.

8. The method of claim 1, including receiving, from the program code based on the dataflow graph, results of the query.

9. The method of claim 1, wherein the query plan generator includes a query optimizer.

10. The method of claim 1, wherein the query plan includes data types of parameters in the query.

11. The method of claim 1, wherein the query contains a reference to an executable function, and the dataflow graph includes a node representing the executable function.

12. The method of claim 1, wherein generating a request to a query planner based on the query includes

producing a structural representation of the data source wherein the structural representation includes a formatted data organization reflective of the data source and is absent data of the data source;

and

including the structural representation in the request to the query planner.

13. The method of claim 1, wherein the data structure is generated by the identified executing system.

14. The method of claim 1, wherein the data structure is generated by a system other than the identified executing system.

15. The method of claim 1, wherein the executing system and the requesting system are the same system.

16. The method of claim 1, wherein the executing system and the requesting system are different systems.

17. A system for operating on data from a data source, the data stored in a tangible, non-transitory computer-readable medium of the data source, the operating based on a query that is expressed in accordance with a query language applicable to a relational database, the system including:

a computer system configured for;

receiving a query;

identifying a data source based on the query;

identifying a requesting system other than a system managing a relational database storing data of the data source;

generating, by the requesting system, a request to a query planner based on the query;

providing, by the requesting system, the request to the query planner;

receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps to be performed by a system managing a relational database;

generating a data structure instantiating a dataflow graph that includes;

a first node that represents at least one operation to be executed,

the first node associated with information usable by an executing system to invoke executable program code to perform the at least one operation,

the first node associated with information usable by an executing system to make data available to the program code, and

the at least one operation chosen based on a step described by the query plan, and

at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph;

receiving data from the data source;

identifying an executing system other than a system managing a relational database storing data of the data source; and

providing the program code based on the dataflow graph to the identified executing system;

a requesting system other than a system managing a relational database storing data of the data source, the requesting system configured to generate and provide the request to the query planner; and

an executing system other than a system managing a relational database storing data of the data source, the executing system configured to receive the program code and execute the program code.

18. The system of claim 17, wherein the data source includes a relational database table.

19. The system of claim 17, wherein the data source includes a portion of data other than a relational database table.

20. The system of claim 17, wherein the data source includes a flat file.

21. The system of claim 17, wherein the data structure is generated by the identified executing system.

22. The system of claim 17, wherein the data structure is generated by a system other than the identified executing system.

23. The system of claim 17, wherein the executing system and the requesting system are the same system.

24. The system of claim 17, wherein the executing system and the requesting system are different systems.

25. A computer-readable storage device storing a computer program including instructions for causing a computer to operate on data from a data source, the data stored in a tangible, non-transitory computer-readable medium of the data source, the operating based on a query that is expressed in accordance with a query language applicable to a relational database, the operating including:

receiving a query;

identifying a data source based on the query;

identifying a requesting system other than a system managing a relational database storing data of the data source;

generating, by the requesting system, a request to a query planner based on the query;

providing, by the requesting system, the request to the query planner;

receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps to be performed by a system managing a relational database;

generating a data structure instantiating a dataflow graph that includes;

a first node that represents at least one operation to be executed,

the first node associated with information usable by an executing system to invoke executable program code to perform the at least one operation,

the first node associated with information usable by an executing system to make data available to the program code, and

the at least one operation chosen based on a step described by the query plan, and

at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph;

receiving data from the data source;

identifying an executing system other than a system managing a relational database storing data of the data source; and

providing the program code based on the dataflow graph to the identified executing system.

26. The computer-readable storage device of claim 25, wherein the data source includes a relational database table.

27. The computer-readable storage device of claim 25, wherein the data source includes a portion of data other than a relational database table.

28. The computer-readable storage device of claim 25, wherein the data source includes a flat file.

29. The computer-readable storage device of claim 25, wherein the data structure is generated by the identified executing system.

30. The computer-readable storage device of claim 25, wherein the data structure is generated by a system other than the identified executing system.

31. The computer-readable storage device of claim 25, wherein the executing system and the requesting system are the same system.

32. The computer-readable storage device of claim 25, wherein the executing system and the requesting system are different systems.

33. A system for operating on data from a data source, the data stored in a tangible, non-transitory computer-readable medium of the data source, the operating based on a query that is expressed in accordance with a query language applicable to a relational database, the system including:

means for;

receiving a query;

identifying a data source based on the query;

identifying a requesting system other than a system managing a relational database storing data of the data source;

generating, by the requesting system, a request to a query planner based on the query;

providing, by the requesting system, the request to the query planner;

receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps to be performed by a system managing a relational database;

generating a data structure instantiating a dataflow graph that includes;

a first node that represents at least one operation to be executed,

the first node associated with information usable by an executing system to invoke executable program code to perform the at least one operation,

the first node associated with information usable by an executing system to make data available to the program code, and

the at least one operation chosen based on a step described by the query plan, and

at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph;

receiving data from the data source;

identifying an executing system other than a system managing a relational database storing data of the data source and

providing the program code based on the dataflow graph to the identified executing system; and

means for receiving the program code and executing the program code.

34. A method of operating on data from a data source, the data stored in a tangible, non-transitory computer-readable medium of the data source, the operating based on a query that is expressed in accordance with a query language applicable to a relational database, the operating including:

receiving a query;

identifying a data source based on the query;

identifying an executing system other than a system managing a relational database storing data of the data source;

generating a structural representation of data of the data source, the structural representation including a formatted data organization reflective of the data source and absent at least some of the data of the data source;

generating a request to a query planner based on the query, the request instructing the query planner to generate a query plan based on the query and providing the structural representation as data to be acted upon by the query;

providing the request to the query planner;

receiving a query plan generated by the query planner based on the request, the query plan including a description of one or more steps to be performed by a system managing a relational database;

generating a data structure instantiating a dataflow graph that includes;

a first node that represents at least one operation to be executed,

the first node associated with information usable by an executing system to invoke executable program code to perform the at least one operation,

the first node associated with information usable by an executing system to make data available to the program code, and

the at least one operation chosen based on a step described by the query plan, and

at least one link that represents output data of an operation of the first node being provided as input data to an operation of a second node of the dataflow graph;

receiving data from the data source; and

executing, on the identified executing system, program code based on the dataflow graph.

35. The method of claim 34, wherein the data source includes a relational database table.

36. The method of claim 34, wherein the data source includes a portion of data other than a relational database table.

37. The method of claim 34, wherein the data source includes a flat file.

38. The method of claim 34, wherein the data structure is generated by the identified executing system.

39. The method of claim 34, wherein the data structure is generated by a system other than the identified executing system.

40. The method of claim 34, wherein the executing system and the requesting system are the same system.

41. The method of claim 34, wherein the executing system and the requesting system are different systems.