Scalable distributed data streaming computations across multiple data processing clusters

• US 10,404,787 B1
• Filed: 08/22/2017
• Issued: 09/03/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Masad, Ofri
• Primary Examiner(s): Tokuta, Shean
• Application Number:
  US15/683,169
• Time in Patent Office: 742 Days
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/541 via adapters, e.g. between incompatible applications
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs

Abstract

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to initiate distributed data streaming computations across data processing clusters associated with respective data zones, and in each of the data processing clusters, to separate a data stream provided by a data source of the corresponding data zone into a plurality of data batches and process the data batches to generate respective result batches. Multiple ones of the data batches across the data processing clusters are associated with a global data batch data structure, and multiple ones of the result batches across the data processing clusters are associated with a global result batch data structure based at least in part on the global data batch data structure. The result batches are processed in accordance with the global result batch data structure to generate one or more global result streams.

Claims

1. A method comprising:

initiating distributed data streaming computations across a plurality of data processing clusters associated with respective data zones;

in each of the data processing clusters, separating a data stream provided by a data source of the corresponding data zone into a plurality of data batches and processing the data batches to generate respective result batches;

associating multiple ones of the data batches across the data processing clusters with a global data batch data structure;

associating multiple ones of the result batches across the data processing clusters with a global result batch data structure based at least in part on the global data batch data structure; and

processing the result batches in accordance with the global result batch data structure to generate one or more global result streams providing global results of the distributed data streaming computations;

wherein the plurality of data processing clusters associated with the respective data zones are organized in accordance with a global computation graph for performance of the distributed data streaming computations and wherein the global computation graph comprises a plurality of nodes corresponding to respective ones of the data processing clusters and further wherein the plurality of nodes are arranged in multiple levels each including at least one of the nodes;

wherein the global data batch data structure is organized in levels with different levels of the global data batch data structure corresponding to respective ones of the levels of the global computation graph and wherein a given one of the levels of the global data batch data structure comprises data batches generated by nodes of the corresponding level in the global computation graph, the data batches at the given level of the global data batch data structure being approximately synchronized with one another as belonging to a common iteration of a global data stream data structure based at least in part on at least one of a time interval during which the data batch was generated, a sequence number associated with generation of the data batch and a time-stamp associated with generation of the data batch; and

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein the distributed data streaming computations comprise Spark streaming computations and each of the data batches comprises one or more Spark resilient distributed datasets (RDDs).

3. The method of claim 1 wherein a particular one of the data processing clusters corresponding to a root node of the global computation graph initiates the distributed data streaming computations in accordance with a control flow that propagates from the root node toward leaf nodes of the global computation graph via one or more intermediate nodes of the global computation graph and wherein local result streams from respective ones of the data processing clusters corresponding to respective ones of the nodes propagate back from those nodes toward the root node.

4. The method of claim 3 wherein a given one of the local result streams from one of the data processing clusters corresponding to one of the nodes at a given one of the levels provides at least a portion of an input data stream of another one of the data processing clusters corresponding to another one of the nodes at another one of the levels and wherein result batches of the given local result stream of the node at the given level are mapped into respective data batches of the node at the other level.

5. The method of claim 3 wherein the control flow includes one or more messages from at least one node at a given level of the global computation graph directing one or more nodes at another one of the levels to perform designated portions of the distributed data streaming computations.

6. The method of claim 1 wherein at least one of the distributed data processing clusters corresponds to multiple nodes of the global computation graph.

8. The method of claim 1 wherein each of the data processing clusters generates its corresponding data batches and result batches independently of and at least partially in parallel with the other data processing clusters.

9. The method of claim 1 wherein each of the data processing clusters generates its data batches and result batches asynchronously with respect to the other data processing clusters but the data batches and results batches of the data processing clusters are eventually synchronized across the data processing clusters in conjunction with generation of the one or more global result streams through utilization of the global data batch data structure and the global result batch data structure.

10. The method of claim 1 wherein the global data batch data structure comprises a plurality of local data batch data structures of respective ones of the data processing clusters and wherein at least a subset of the local data batch data structures have respective different formats so as to support local data batch heterogeneity across the data processing clusters.

12. The method of claim 1 wherein the global result batch data structure comprises a plurality of local result batch data structures of respective ones of the data processing clusters and wherein at least a subset of the local result batch data structures have respective different formats so as to support local result batch heterogeneity across the data processing clusters.

14. The method of claim 1 wherein the distributed data streaming computations comprise Spark streaming computations of a Spark iterative application, the plurality of data processing clusters associated with the respective data zones being organized in accordance with a global computation graph for performance of the distributed data streaming computations, wherein the global computation graph comprises a plurality of nodes corresponding to respective ones of the data processing clusters and further wherein the plurality of nodes are arranged in multiple levels each including at least one of the nodes and further wherein each of at least a subset of the plurality of nodes comprises both a local Spark streaming application instance and a local Spark iterative application instance.

15. The method of claim 14 wherein the local Spark streaming application instance of a given one of the nodes other than a leaf node is configured to receive an input data stream from another one of the nodes at a higher level of the global computation graph, to provide an output data stream as an input data stream to another one of the nodes at a lower level of the global computation graph, and to generate one or more triggers for controlling execution of the local Spark iterative application instance of the given node.

16. The method of claim 14 wherein the local Spark iterative application instances are triggered utilizing control flow that propagates from a root node of the global computation graph toward leaf nodes of the global computation graph via one or more intermediate nodes of the global computation graph wherein the control flow comprises data streams generated by respective ones of the nodes.

17. The method of claim 1 wherein the distributed data streaming computations comprise Spark streaming computations of a Spark interactive application.

18. The method of claim 17 wherein the Spark interactive application comprises an extension of a Spark iterative application.

19. The method of claim 17 wherein the Spark interactive application is configured to support user-controlled triggering of local Spark interactive application instances at respective ones of the nodes via a command-line interface.

7. A method comprising:

initiating distributed data streaming computations across a plurality of data processing clusters associated with respective data zones;

in each of the data processing clusters, separating a data stream provided by a data source of the corresponding data zone into a plurality of data batches and processing the data batches to generate respective result batches;

associating multiple ones of the data batches across the data processing clusters with a global data batch data structure;

associating multiple ones of the result batches across the data processing clusters with a global result batch data structure based at least in part on the global data batch data structure; and

processing the result batches in accordance with the global result batch data structure to generate one or more global result streams providing global results of the distributed data streaming computations;

wherein the plurality of data processing clusters associated with the respective data zones are organized in accordance with a global computation graph for performance of the distributed data streaming computations and wherein the global computation graph comprises a plurality of nodes corresponding to respective ones of the data processing clusters and further wherein the plurality of nodes are arranged in multiple levels each including at least one of the nodes;

wherein the global result batch data structure is organized in levels with different levels of the global result batch data structure corresponding to respective ones of the levels of the global computation graph and wherein a given one of the levels of the global result batch data structure comprises result batches generated by nodes of the corresponding level in the global computation graph, the result batches at the given level of the global result batch data structure being approximately synchronized with one another as belonging to a common iteration of a global data stream data structure based at least in part on at least one of a time interval during which the result batch was generated, a sequence number associated with generation of the result batch and a time-stamp associated with generation of the result batch; and

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

11. A method comprising:

initiating distributed data streaming computations across a plurality of data processing clusters associated with respective data zones;

in each of the data processing clusters, separating a data stream provided by a data source of the corresponding data zone into a plurality of data batches and processing the data batches to generate respective result batches;

associating multiple ones of the data batches across the data processing clusters with a global data batch data structure;

associating multiple ones of the result batches across the data processing clusters with a global result batch data structure based at least in part on the global data batch data structure; and

processing the result batches in accordance with the global result batch data structure to generate one or more global result streams providing global results of the distributed data streaming computations;

wherein the global data batch data structure comprises a plurality of local data batch data structures of respective ones of the data processing clusters and wherein at least a subset of the local data batch data structures have respective different formats so as to support local data batch heterogeneity across the data processing clusters;

wherein at least one of the local data batch data structures itself comprises a global data batch data structure having a plurality of additional local data batch data structures of respective additional data processing clusters associated therewith; and

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

24. The method of claim 11 wherein the distributed data streaming computations comprise Spark streaming computations and each of the data batches comprises one or more Spark resilient distributed datasets (RDDs).

13. A method comprising:

initiating distributed data streaming computations across a plurality of data processing clusters associated with respective data zones;

in each of the data processing clusters, separating a data stream provided by a data source of the corresponding data zone into a plurality of data batches and processing the data batches to generate respective result batches;

associating multiple ones of the data batches across the data processing clusters with a global data batch data structure;

associating multiple ones of the result batches across the data processing clusters with a global result batch data structure based at least in part on the global data batch data structure; and

processing the result batches in accordance with the global result batch data structure to generate one or more global result streams providing global results of the distributed data streaming computations;

wherein the global result batch data structure comprises a plurality of local result batch data structures of respective ones of the data processing clusters and wherein at least a subset of the local result batch data structures have respective different formats so as to support local result batch heterogeneity across the data processing clusters;

wherein at least one of the local result batch data structures itself comprises a global result batch data structure having a plurality of additional local result batch data structures of respective additional data processing clusters associated therewith; and

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

25. The method of claim 13 wherein the distributed data streaming computations comprise Spark streaming computations and each of the data batches comprises one or more Spark resilient distributed datasets (RDDs).

20. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to initiate distributed data streaming computations across a plurality of data processing clusters associated with respective data zones;

in each of the data processing clusters, to separate a data stream provided by a data source of the corresponding data zone into a plurality of data batches and process the data batches to generate respective result batches;

to associate multiple ones of the data batches across the data processing clusters with a global data batch data structure;

to associate multiple ones of the result batches across the data processing clusters with a global result batch data structure based at least in part on the global data batch data structure; and

to process the result batches in accordance with the global result batch data structure to generate one or more global result streams providing global results of the distributed data streaming computations;

wherein the plurality of data processing clusters associated with the respective data zones are organized in accordance with a global computation graph for performance of the distributed data streaming computations and wherein the global computation graph comprises a plurality of nodes corresponding to respective ones of the data processing clusters and further wherein the plurality of nodes are arranged in multiple levels each including at least one of the nodes; and

wherein the global data batch data structure is organized in levels with different levels of the global data batch data structure corresponding to respective ones of the levels of the global computation graph and wherein a given one of the levels of the global data batch data structure comprises data batches generated by nodes of the corresponding level in the global computation graph, the data batches at the given level of the global data batch data structure being approximately synchronized with one another as belonging to a common iteration of a global data stream data structure based at least in part on at least one of a time interval during which the data batch was generated, a sequence number associated with generation of the data batch and a time-stamp associated with generation of the data batch.

21. The computer program product of claim 20 wherein the plurality of data processing clusters associated with the respective data zones are organized in accordance with a global computation graph for performance of the distributed data streaming computations and wherein the global computation graph comprises a plurality of nodes corresponding to respective ones of the data processing clusters and further wherein the plurality of nodes are arranged in multiple levels each including at least one of the nodes.

22. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processing device is configured;

to initiate distributed data streaming computations across a plurality of data processing clusters associated with respective data zones;

in each of the data processing clusters, to separate a data stream provided by a data source of the corresponding data zone into a plurality of data batches and process the data batches to generate respective result batches;

to associate multiple ones of the data batches across the data processing clusters with a global data batch data structure;

to associate multiple ones of the result batches across the data processing clusters with a global result batch data structure based at least in part on the global data batch data structure; and

to process the result batches in accordance with the global result batch data structure to generate one or more global result streams providing global results of the distributed data streaming computations;

wherein the plurality of data processing clusters associated with the respective data zones are organized in accordance with a global computation graph for performance of the distributed data streaming computations and wherein the global computation graph comprises a plurality of nodes corresponding to respective ones of the data processing clusters and further wherein the plurality of nodes are arranged in multiple levels each including at least one of the nodes; and

wherein the global data batch data structure is organized in levels with different levels of the global data batch data structure corresponding to respective ones of the levels of the global computation graph and wherein a given one of the levels of the global data batch data structure comprises data batches generated by nodes of the corresponding level in the global computation graph, the data batches at the given level of the global data batch data structure being approximately synchronized with one another as belonging to a common iteration of a global data stream data structure based at least in part on at least one of a time interval during which the data batch was generated, a sequence number associated with generation of the data batch and a time-stamp associated with generation of the data batch.

23. The apparatus of claim 22 wherein the plurality of data processing clusters associated with the respective data zones are organized in accordance with a global computation graph for performance of the distributed data streaming computations and wherein the global computation graph comprises a plurality of nodes corresponding to respective ones of the data processing clusters and further wherein the plurality of nodes are arranged in multiple levels each including at least one of the nodes.