Data engine integration and data refinement
First Claim
1. A computer-implemented method comprising:
- receiving, by a data refinement engine, a request for processed data;
determining, by the data refinement engine, a data source for unprocessed data and processing required to transform the unprocessed data into the processed data, the data source including a plurality of data nodes that are separate from the data refinement engine;
determining, by the data refinement engine, that both the plurality of data nodes and the data refinement engine are configured to perform processing required to transform the unprocessed data into the processed data;
determining, by the data refinement engine, (i) a portion of the processing for the plurality of data nodes to perform to transform a portion of the unprocessed data into a portion of the processed data and (ii) a remaining portion of the processing for the data refinement engine to perform to transform a remaining portion of the unprocessed data into a remaining portion of the processed data, wherein the respective portions of processing for the data refinement engine and the plurality of data nodes are determined to balance a processing load of the data refinement engine with a processing load of the plurality of data nodes;
based on determining (i) the portion of the processing for the plurality of data nodes to perform to transform the portion of the unprocessed data into the portion of the processed data and (ii) the remaining portion of the processing for the data refinement engine to perform to transform the remaining portion of the unprocessed data into the remaining portion of the processed data, transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data;
receiving, by the data refinement engine and from the plurality of data nodes, the portion of the processed data and the remaining portion of the unprocessed data;
transforming, by the data refinement engine, the remaining portion of the unprocessed data into the remaining portion of the processed data by performing the remaining portion of the processing on the remaining portion of the unprocessed data; and
transmitting, by the data refinement engine, the portion of the processed data received from the plurality of data nodes and the remaining portion of the processed data processed by the data refinement engine.
2 Assignments
0 Petitions
Accused Products
Abstract
Data engine integration and data refinement are described. The actions include receiving, by a data refinement engine, a request for data. The actions include determining a first amount of processing to be performed by the data refinement engine and a second amount of processing to be performed by one or more processors of a data source that include a plurality of data nodes. The actions include transmitting, by the data refinement engine, code to the plurality of data nodes of instructions associated with the second amount of processing. The actions include receiving, by the data refinement engine and from the plurality of data nodes, unprocessed first data and processed second data. The actions include processing, by the data refinement engine, the unprocessed first data. The actions include, in response to the request for data, transmitting, by the data refinement engine, the processed first data and the processed second data.
-
Citations
18 Claims
-
1. A computer-implemented method comprising:
-
receiving, by a data refinement engine, a request for processed data; determining, by the data refinement engine, a data source for unprocessed data and processing required to transform the unprocessed data into the processed data, the data source including a plurality of data nodes that are separate from the data refinement engine; determining, by the data refinement engine, that both the plurality of data nodes and the data refinement engine are configured to perform processing required to transform the unprocessed data into the processed data; determining, by the data refinement engine, (i) a portion of the processing for the plurality of data nodes to perform to transform a portion of the unprocessed data into a portion of the processed data and (ii) a remaining portion of the processing for the data refinement engine to perform to transform a remaining portion of the unprocessed data into a remaining portion of the processed data, wherein the respective portions of processing for the data refinement engine and the plurality of data nodes are determined to balance a processing load of the data refinement engine with a processing load of the plurality of data nodes; based on determining (i) the portion of the processing for the plurality of data nodes to perform to transform the portion of the unprocessed data into the portion of the processed data and (ii) the remaining portion of the processing for the data refinement engine to perform to transform the remaining portion of the unprocessed data into the remaining portion of the processed data, transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data; receiving, by the data refinement engine and from the plurality of data nodes, the portion of the processed data and the remaining portion of the unprocessed data; transforming, by the data refinement engine, the remaining portion of the unprocessed data into the remaining portion of the processed data by performing the remaining portion of the processing on the remaining portion of the unprocessed data; and transmitting, by the data refinement engine, the portion of the processed data received from the plurality of data nodes and the remaining portion of the processed data processed by the data refinement engine. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A system comprising:
-
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising; receiving, by a data refinement engine, a request for processed data; determining, by the data refinement engine, a data source for unprocessed data and processing required to transform the unprocessed data into the processed data, the data source including a plurality of data nodes that are separate from the data refinement engine; determining, by the data refinement engine, that both the plurality of data nodes and the data refinement engine are configured to perform processing required to transform the unprocessed data into the processed data; determining, by the data refinement engine, (i) a portion of the processing for the plurality of data nodes to perform to transform a portion of the unprocessed data into a portion of the processed data and (ii) a remaining portion of the processing for the data refinement engine to perform to transform a remaining portion of the unprocessed data into a remaining portion of the processed data, wherein the respective portions of processing for the data refinement engine and the plurality of data nodes are determined to balance a processing load of the data refinement engine with a processing load of the plurality of data nodes; based on determining (i) the portion of the processing for the plurality of data nodes to perform to transform the portion of the unprocessed data into the portion of the processed data and (ii) the remaining portion of the processing for the data refinement engine to perform to transform the remaining portion of the unprocessed data into the remaining portion of the processed data, transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data; receiving, by the data refinement engine and from the plurality of data nodes, the portion of the processed data and the remaining portion of the unprocessed data; transforming, by the data refinement engine, the remaining portion of the unprocessed data into the remaining portion of the processed data by performing the remaining portion of the processing on the remaining portion of the unprocessed data; and transmitting, by the data refinement engine, the portion of the processed data received from the plurality of data nodes and the remaining portion of the processed data processed by the data refinement engine. - View Dependent Claims (9, 10, 11, 12, 13)
-
-
14. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
-
receiving, by a data refinement engine, a request for processed data; determining, by the data refinement engine, a data source for unprocessed data and processing required to transform the unprocessed data into the processed data, the data source including a plurality of data nodes that are separate from the data refinement engine; determining, by the data refinement engine, that both the plurality of data nodes and the data refinement engine are configured to perform processing required to transform the unprocessed data into the processed data; determining, by the data refinement engine, (i) a portion of the processing for the plurality of data nodes to perform to transform a portion of the unprocessed data into a portion of the processed data and (ii) a remaining portion of the processing for the data refinement engine to perform to transform a remaining portion of the unprocessed data into a remaining portion of the processed data, wherein the respective portions of processing for the data refinement engine and the plurality of data nodes are determined to balance a processing load of the data refinement engine with a processing load of the plurality of data nodes; based on determining (i) the portion of the processing for the plurality of data nodes to perform to transform the portion of the unprocessed data into the portion of the processed data and (ii) the remaining portion of the processing for the data refinement engine to perform to transform the remaining portion of the unprocessed data into the remaining portion of the processed data, transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data; receiving, by the data refinement engine and from the plurality of data nodes, the portion of the processed data and the remaining portion of the unprocessed data; transforming, by the data refinement engine, the remaining portion of the unprocessed data into the remaining portion of the processed data by performing the remaining portion of the processing on the remaining portion of the unprocessed data; and transmitting, by the data refinement engine, the portion of the processed data received from the plurality of data nodes and the remaining portion of the processed data processed by the data refinement engine. - View Dependent Claims (15, 16, 17, 18)
-
Specification