PROCESSING DATA FROM MULTIPLE SOURCES

• US 20170220646A1
• Filed: 02/14/2017
• Published: 08/03/2017
• Est. Priority Date: 04/17/2014
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Schechter, Ian, Wakeling, Tim, Wollrath, Ann M.
• Granted Patent: US 10,642,850 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Claims

1. A method including:

at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage;

executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster;

receiving a dataflow graph by the data processing engine, the dataflow graph including a) at least one component representing the Hadoop cluster, b) at least one component representing the data source external to the Hadoop cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation;

executing at least part of the dataflow graph by the first instance of the data processing engine;

receiving, by the data processing engine, a second portion of data from the external data source;

and

performing, by the data processing engine, the data processing operation using at least the first portion of data and the second portion of data.

2. The method of claim 1 in which the Hadoop cluster includes nodes each executing an instance of the data processing engine, the instances of the data processing engine running concurrently to perform the data processing operation together in parallel on a) a first body of data that includes the first portion of data, the first body of data also including other portions of data being processed by the other nodes of the Hadoop cluster, and b) a second body of data that includes the second portion of data, the second body of data being stored in a format native to a relational database system, and the second body of data being divided into portions that each can be stored in volatile memory of the nodes of the Hadoop cluster.

4. The method of claim 1 in which at least one component of the dataflow graph is connected to a link representing a flow of data from the Hadoop cluster, and wherein the at least one component is connected to a link representing a flow of data from the source of the second portion of data.

5. The method of claim 1 in which the data processing engine does not implement the MapReduce programming model.

6. The method of claim 1, in which the second portion of data is stored in volatile memory.

7. The method of claim 1 including

receiving a database query, the database query including at least one operation to be performed on data received from at least one source of data that includes the Hadoop cluster;

and

the dataflow graph includes components representing operations corresponding to the database query, wherein the dataflow graph includes at least one component representing the at least one source of data and at least one link that represents at least one dataflow associated with the operation to be performed on data received from at least one source of data.

8. The method of claim 1 in which the second portion of data was chosen based on characteristics of the first portion of data.

9. The method of claim 1 in which the second portion of data includes a subset of rows of a relational database, and the second portion of data includes a subset of columns of the relational database.

10. The method of claim 1 in which the second portion of data is distinct from a third portion of data received at a second node of the Hadoop cluster from the external data source.

11. The method of claim 1 including communicating with an instance of at least part of the dataflow graph that is being executed by a second instance of the data processing engine that is outside of the Hadoop cluster.

12. The method of claim 1 including executing at least part of the dataflow graph by a second instance of the data processing engine outside of the Hadoop cluster.

3. (canceled)

13. A computer-readable storage device including instructions for causing a node of a Hadoop cluster storing a first portion of data in HDFS data storage to carry out operations including:

executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster;

receiving a dataflow graph by the data processing engine, the dataflow graph including a) at least one component representing the Hadoop cluster, b) at least one component representing the data source external to the Hadoop cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation;

executing at least part of the dataflow graph by the first instance of the data processing engine;

receiving, by the data processing engine, a second portion of data from the external data source;

and

performing, by the data processing engine, the data processing operation using at least the first portion of data and the second portion of data.

14. A node of a Hadoop cluster storing a first portion of data in HDFS storage and including a computer processing device configured to carry out operations including:

executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster;

receiving a dataflow graph by the data processing engine, the dataflow graph including a) at least one component representing the Hadoop cluster, b) at least one component representing the data source external to the Hadoop cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation;

executing at least part of the dataflow graph by the first instance of the data processing engine;

receiving, by the data processing engine, a second portion of data from the external data source;

and

performing, by the data processing engine, the data processing operation using at least the first portion of data and the second portion of data.

15. A node of a Hadoop cluster storing a first portion of data in HDFS storage and including:

means for executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster;

means for receiving a dataflow graph by the data processing engine, the dataflow graph including a) at least one component representing the Hadoop cluster, b) at least one component representing the data source external to the Hadoop cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation;

means for executing at least part of the dataflow graph by the first instance of the data processing engine;

means for receiving, by the data processing engine, a second portion of data from the external data source;

and

means for performing, by the data processing engine, the data processing operation using at least the first portion of data and the second portion of data.

16. A method including:

at a node of a cluster of nodes, the node storing a first portion of data and configured to carry out one or more data processing operations in conjunction with the cluster of nodes, the cluster storing a collection of data across the nodes, the nodes being configured to operate on the collection of data in parallel, the collection of data being split into portions, each portion being operated on by a respective node of the cluster;

executing a first instance of a data processing engine capable of receiving data from a data source external to the cluster;

receiving a dataflow graph by the data processing engine, the dataflow graph including a) at least one component representing the Hadoop cluster, b) at least one component representing the data source external to the Hadoop cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation;

executing at least part of the dataflow graph by the first instance of the data processing engine;

based on characteristics of the first portion of data, requesting, by the data processing engine, a second portion of data;

receiving, by the data processing engine, the second portion of data from the external data source;

storing the second portion of data in volatile memory of the node; and

performing, by the data processing engine, the data processing operation using at least the first portion of data and the second portion of data.

17. A method including:

at a data processing engine of a node of a Hadoop cluster, performing a data processing operation identified by a dataflow graph being executed by the data processing engine, the dataflow graph including a) at least one component representing the Hadoop cluster, b) at least one component representing the data source external to the Hadoop cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation, the data processing operation being performed using at least a first portion of data stored in HDFS data storage at the node and at least a second portion of data received from a data source external to the Hadoop cluster.

18. A method including:

receiving a SQL query specifying sources of data including a Hadoop cluster and a relational database;

generating a dataflow graph that corresponds to the SQL query, the dataflow graph including a) at least one component representing the Hadoop cluster, b) at least one component representing the data source external to the Hadoop cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation;

executing the dataflow graph at a data processing engine of a node of the Hadoop cluster; and

performing, by the data processing engine, the data processing operation using at least data of the Hadoop cluster and data of the relational database.