BACKGROUND FORMAT OPTIMIZATION FOR ENHANCED SQL-LIKE QUERIES IN HADOOP
First Claim
1. A method of data processing for query execution, the method being performed by a query engine instance running on each data node of a plurality of data nodes which together form a Hadoop™
- distributed computing cluster, wherein a query is processed by whichever data node that receives the query, the method comprising;
storing initial data in an original format at a data node in the plurality of data nodes forming a peer-to-peer network for the query, each data node functioning as a peer in the peer-to-peer network and being capable of interacting with components of the Hadoop™
cluster, each peer having an instance of a query engine running in memory;
converting, at the data node, the initial data to be in a target format that is optimized for relational database processing according to a predetermined schedule; and
storing the converted data together with the initial data.
5 Assignments
0 Petitions
Accused Products
Abstract
A format conversion engine for Apache Hadoop that converts data from its original format to a database-like format at certain time points for use by a low latency (LL) query engine. The format conversion engine comprises a daemon that is installed on each data node in a Hadoop cluster. The daemon comprises a scheduler and a converter. The scheduler determines when to perform the format conversion and notifies the converter when the time comes. The converter converts data on the data node from its original format to a database-like format for use by the low latency (LL) query engine.
-
Citations
20 Claims
-
1. A method of data processing for query execution, the method being performed by a query engine instance running on each data node of a plurality of data nodes which together form a Hadoop™
- distributed computing cluster, wherein a query is processed by whichever data node that receives the query, the method comprising;
storing initial data in an original format at a data node in the plurality of data nodes forming a peer-to-peer network for the query, each data node functioning as a peer in the peer-to-peer network and being capable of interacting with components of the Hadoop™
cluster, each peer having an instance of a query engine running in memory;converting, at the data node, the initial data to be in a target format that is optimized for relational database processing according to a predetermined schedule; and storing the converted data together with the initial data. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
- distributed computing cluster, wherein a query is processed by whichever data node that receives the query, the method comprising;
-
9. A system for data processing for query execution, the system having a plurality of data nodes of a plurality of data nodes which together form a Hadoop™
- distributed computing cluster, each data node having an instance of a query engine, wherein a query is processed by whichever data node that receives the query, and wherein the plurality of data nodes include;
a first storing unit which stores initial data in an original format at a data node in the plurality of data nodes forming a peer-to-peer network for the query, each data node functioning as a peer in the peer-to-peer network and being capable of interacting with components of the Hadoop™
cluster, each peer having the instance of the query engine running in memory;a converting unit which converts the initial data to be in a target format that is optimized for relational database processing according to a predetermined schedule; and a second storing unit which stores the converted data. - View Dependent Claims (10, 11, 12)
- distributed computing cluster, each data node having an instance of a query engine, wherein a query is processed by whichever data node that receives the query, and wherein the plurality of data nodes include;
-
13. A non-transitory machine-readable storage medium having stored thereon instructions which when executed by one or more processors perform a method, the method being performed by a query engine instance running on each data node of a plurality of data nodes which together form a Hadoop™
- distributed computing cluster, wherein a query is processed by whichever data node that receives the query, the method comprising;
storing initial data in an original format at a data node in the plurality of data nodes forming a peer-to-peer network for the query, each data node functioning as a peer in the peer-to-peer network and being capable of interacting with components of the Hadoop™
cluster, each peer having the query engine instance running in memory;converting, at the data node, the initial data to be in a target format that is optimized for relational database processing according to a predetermined schedule; and storing the converted data. - View Dependent Claims (14, 15)
- distributed computing cluster, wherein a query is processed by whichever data node that receives the query, the method comprising;
-
16. A system for performing queries on stored data in a Hadoop™
- distributed computing cluster, the system comprising;
a plurality of data nodes forming a peer-to-peer network for the queries received from a client, a respective data node of the plurality of data nodes functioning as a peer in the peer-to-peer network and being capable of interacting with components of the Hadoop™
cluster, the respective data node operating an instance of a query engine that is configured to;parse a query from a client; selectively creates query fragments based on an availability of converted data at the respective data node, the converted data corresponding to data associated with the query, wherein the converted data is the data associated with the query converted from an original format into a target format that is specified by a schema, and wherein the query is processed by whichever data node that receives the query; distribute the query fragments among the plurality of data nodes; execute the query fragments on whichever local data that corresponds to a format for which the query fragments are created, based on the schema; obtain intermediate results from other data nodes that receive the query fragments; and aggregate the intermediate results for the client. - View Dependent Claims (17, 18, 19, 20)
- distributed computing cluster, the system comprising;
Specification