Parallel streaming of external data
First Claim
1. A computer-implemented method comprising:
- receiving, by a first distributed system that comprises multiple segment nodes, a query that requests rows of an external table representing data stored in multiple data fragments on multiple respective data nodes in a second distributed system, wherein each of the data nodes of the second distributed system are distinct from the segment nodes of the first distributed system, wherein the first distributed system operates under control of a first master node and the second distributed system operates under control of a second master node that is distinct from the first master node, and wherein the query includes a predicate that specifies a condition on an attribute of the requested rows of the external table;
providing the predicate to a plurality of segment nodes of the first distributed system;
initiating communication by a plurality of extension services between nodes of the first distributed system and nodes of the second distributed system, including a first extension service communicating with the master nodes of the first and second distributed system and a plurality of second extension services communicating between data nodes of the second distributed system and segment nodes of the first distributed system;
receiving, by each of the plurality of segment nodes of the first distributed system, filtered data corresponding to the rows of the external table, wherein each segment node of the multiple segment nodes of the first distributed system provides the predicate to a second extension service that is between the segment node of the first distributed system and a data node of the second distributed system, wherein the second extension service;
obtains data fragments from one or more data nodes of the second distributed system,determines whether each data fragment has an attribute that satisfies the predicate, andprovides, to the segment node of the first distributed system, filtered data comprising the data fragments from the data node of the second distributed system having the attribute satisfied by the predicate; and
computing a result for the received query using the filtered data corresponding to the rows of the external table.
1 Assignment
0 Petitions
Accused Products
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for streaming external data in parallel from a second distributed system to a first distributed system. One of the methods includes receiving a query that requests a join of first rows of a first table in a first distributed system with second rows of an external table, the external table representing data in a second distributed system. Each of the segment nodes communicates with a respective extension service that obtains fragments from one or more data nodes of the second distributed system according to location information for the respective fragments, and provides to the segment node a stream of data corresponding to second rows of the external table. Each of the segment nodes computes joined rows between the first rows of the first table and the stream of data corresponding to second rows of the external table.
25 Citations
18 Claims
-
1. A computer-implemented method comprising:
-
receiving, by a first distributed system that comprises multiple segment nodes, a query that requests rows of an external table representing data stored in multiple data fragments on multiple respective data nodes in a second distributed system, wherein each of the data nodes of the second distributed system are distinct from the segment nodes of the first distributed system, wherein the first distributed system operates under control of a first master node and the second distributed system operates under control of a second master node that is distinct from the first master node, and wherein the query includes a predicate that specifies a condition on an attribute of the requested rows of the external table; providing the predicate to a plurality of segment nodes of the first distributed system; initiating communication by a plurality of extension services between nodes of the first distributed system and nodes of the second distributed system, including a first extension service communicating with the master nodes of the first and second distributed system and a plurality of second extension services communicating between data nodes of the second distributed system and segment nodes of the first distributed system; receiving, by each of the plurality of segment nodes of the first distributed system, filtered data corresponding to the rows of the external table, wherein each segment node of the multiple segment nodes of the first distributed system provides the predicate to a second extension service that is between the segment node of the first distributed system and a data node of the second distributed system, wherein the second extension service; obtains data fragments from one or more data nodes of the second distributed system, determines whether each data fragment has an attribute that satisfies the predicate, and provides, to the segment node of the first distributed system, filtered data comprising the data fragments from the data node of the second distributed system having the attribute satisfied by the predicate; and computing a result for the received query using the filtered data corresponding to the rows of the external table. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising; receiving, by a first distributed system that comprises multiple segment nodes, a query that requests rows of an external table representing data stored in multiple data fragments on multiple respective data nodes in a second distributed system, wherein each of the data nodes of the second distributed system are distinct from the segment nodes of the first distributed system, wherein the first distributed system operates under control of a first master node and the second distributed system operates under control of a second master node that is distinct from the first master node, and wherein the query includes a predicate that specifies a condition on an attribute of the requested rows of the external table; providing the predicate to a plurality of segment nodes of the first distributed system; initiating communication by a plurality of extension services between nodes of the first distributed system and nodes of the second distributed system, including a first extension service communicating with the master nodes of the first and second distributed system and a plurality of second extension services communicating between data nodes of the second distributed system and segment nodes of the first distributed system; receiving, by each of the plurality of segment nodes of the first distributed system, filtered data corresponding to the rows of the external table, wherein each segment node of the multiple segment nodes of the first distributed system provides the predicate to a second extension service that is between the segment node of the first distributed system and a data node of the second distributed system, wherein the second extension service; obtains data fragments from one or more data nodes of the second distributed system, determines whether each data fragment has an attribute that satisfies the predicate, and provides, to the segment node of the first distributed system, filtered data comprising the data fragments from the data node of the second distributed system having the attribute satisfied by the predicate; and computing a result for the received query using the filtered data corresponding to the rows of the external table. - View Dependent Claims (8, 9, 10, 11, 12)
-
13. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
-
receiving, by a first distributed system that comprises multiple segment nodes, a query that requests rows of an external table representing data stored in multiple data fragments on multiple respective data nodes in a second distributed system, wherein each of the data nodes of the second distributed system are distinct from the segment nodes of the first distributed system, wherein the first distributed system operates under control of a first master node and the second distributed system operates under control of a second master node that is distinct from the first master node, and wherein the query includes a predicate that specifies a condition on an attribute of the requested rows of the external table; providing the predicate to a plurality of segment nodes of the first distributed system; initiating communication by a plurality of extension services between nodes of the first distributed system and nodes of the second distributed system, including a first extension service communicating with the master nodes of the first and second distributed system and a plurality of second extension services communicating between data nodes of the second distributed system and segment nodes of the first distributed system; receiving, by each of the plurality of segment nodes of the first distributed system, filtered data corresponding to the rows of the external table, wherein each segment node of the multiple segment nodes of the first distributed system provides the predicate to a second extension service that is between the segment node of the first distributed system and a data node of the second distributed system, wherein the second extension service; obtains data fragments from one or more data nodes of the second distributed system, determines whether each data fragment has an attribute that satisfies the predicate, and provides, to the segment node of the first distributed system, filtered data comprising the data fragments from the data node of the second distributed system having the attribute satisfied by the predicate; and computing a result for the received query using the filtered data corresponding to the rows of the external table. - View Dependent Claims (14, 15, 16, 17, 18)
-
Specification