Scalable distributed in-memory computation utilizing batch mode extensions

• US 10,331,380 B1
• Filed: 04/30/2017
• Issued: 06/25/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Masad, Ofri, Singer, Ido
• Primary Examiner(s): Dennison, Jerry B
• Application Number:
  US15/582,743
• Time in Patent Office: 786 Days
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to distribute in-memory computations across a plurality of data processing clusters associated with respective data zones, and to combine local processing results of the distributed in-memory computations from the data processing clusters. The distributed in-memory computations utilize local data structures of respective ones of the data processing clusters. A given one of the local data structures in one of the data processing clusters receives local data of the corresponding data zone and is utilized to generate the local processing results of that data processing cluster that are combined with local processing results of other ones of the data processing clusters. The local data structures are configured to support batch mode extensions such as Spark SQL, Spark MLlib or Spark GraphX for performance of the distributed in-memory computations.

Claims

1. A method comprising:

distributing in-memory computations of a batch computation framework across a plurality of data processing clusters associated with respective data zones; and

combining local processing results of the distributed in-memory computations from respective ones of the data processing clusters;

wherein the distributed in-memory computations utilize local data structures of respective ones of the data processing clusters;

wherein a given one of the local data structures in one of the data processing clusters receives local data of the corresponding data zone and is utilized to generate the local processing results of that data processing cluster that are combined with local processing results of other ones of the data processing clusters;

wherein the local data structures are configured to support one or more batch mode extensions of the batch computation framework for performance of the distributed in-memory computations;

wherein the local data structures comprise respective portions of a global data structure characterizing the distributed in-memory computations of the batch computation framework;

wherein the global data structure comprises at least one of a global table, a global dataset and a global property graph associated with respective local data structures comprising local tables, local datasets and local property graphs; and

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein the in-memory computations comprise Spark Core batch computations and the one or more batch mode extensions comprise at least one of a Spark SQL extension, a Spark MLlib extension and a Spark GraphX extension.

3. The method of claim 1 wherein the distributed in-memory computations are initially defined by a user for performance on the global data structure and are automatically orchestrated for distributed performance utilizing the local data structures.

4. The method of claim 1 wherein at least one of the local data structures itself comprises a global data structure having a plurality of additional local data structures of respective additional data processing clusters associated therewith.

5. The method of claim 1 wherein at least a subset of the local data structures have respective different formats so as to support local data heterogeneity within respective ones of the data processing clusters.

6. The method of claim 1 wherein each of the local data structures is configured for processing independently from and in parallel with other ones of the local data structures using data from different data sources so as to support local data diversity within respective ones of the data processing clusters.

7. The method of claim 1 wherein the local data structures are determined at least in part by reference to a distributed catalog service.

8. The method of claim 1 wherein the local data structures comprise respective local tables of a plurality of tables from which a global table for the distributed in-memory computations is derived.

9. The method of claim 1 wherein the local data structures comprise respective local datasets of a plurality of datasets from which a global dataset for the distributed in-memory computations is derived.

10. The method of claim 1 wherein the local data structures comprise respective local property graphs of a plurality of property graphs from which a global property graph for the distributed in-memory computations is derived.

11. The method of claim 10 wherein the global property graph comprises a set of annotated vertexes and a set of annotated edges and wherein the set of annotated vertexes bears a containment relationship to one or more sets of annotated vertexes of respective ones of the local property graphs.

12. The method of claim 1 wherein the distributing and combining are performed at least in part in a worldwide data node coupled to one or more of the data processing clusters.

13. The method of claim 1 wherein the distributing and combining are performed at least in part in a worldwide data node that comprises a processing node of a given one of the data processing clusters.

14. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to distribute in-memory computations of a batch computation framework across a plurality of data processing clusters associated with respective data zones; and

to combine local processing results of the distributed in-memory computations from respective ones of the data processing clusters;

wherein the distributed in-memory computations utilize local data structures of respective ones of the data processing clusters;

wherein a given one of the local data structures in one of the data processing clusters receives local data of the corresponding data zone and is utilized to generate the local processing results of that data processing cluster that are combined with local processing results of other ones of the data processing clusters;

wherein the local data structures are configured to support one or more batch mode extensions of the batch computation framework for performance of the distributed in-memory computations;

wherein the local data structures comprise respective portions of a global data structure characterizing the distributed in-memory computations of the batch computation framework; and

wherein the global data structure comprises at least one of a global table, a global dataset and a global property graph associated with respective local data structures comprising local tables, local datasets and local property graphs.

15. The computer program product of claim 14 wherein the in-memory computations comprise Spark Core batch computations and the one or more batch mode extensions comprise at least one of a Spark SQL extension, a Spark MLlib extension and a Spark GraphX extension.

16. The computer program product of claim 14 wherein at least a subset of the local data structures have respective different formats so as to support local data heterogeneity within respective ones of the data processing clusters.

17. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processing device is configured;

to distribute in-memory computations of a batch computation framework across a plurality of data processing clusters associated with respective data zones; and

to combine local processing results of the distributed in-memory computations from respective ones of the data processing clusters;

wherein the distributed in-memory computations utilize local data structures of respective ones of the data processing clusters;

wherein a given one of the local data structures in one of the data processing clusters receives local data of the corresponding data zone and is utilized to generate the local processing results of that data processing cluster that are combined with local processing results of other ones of the data processing clusters;

wherein the local data structures are configured to support one or more batch mode extensions of the batch computation framework for performance of the distributed in-memory computations;

wherein the local data structures comprise respective portions of a global data structure characterizing the distributed in-memory computations of the batch computation framework; and

wherein the global data structure comprises at least one of a global table, a global dataset and a global property graph associated with respective local data structures comprising local tables, local datasets and local property graphs.

18. The apparatus of claim 17 wherein the in-memory computations comprise Spark Core batch computations and the one or more batch mode extensions comprise at least one of a Spark SQL extension, a Spark MLlib extension and a Spark GraphX extension.

19. The apparatus of claim 17 wherein at least a subset of the local data structures have respective different formats so as to support local data heterogeneity within respective ones of the data processing clusters.

20. The apparatus of claim 17 wherein at least one of the local data structures itself comprises a global data structure having a plurality of additional local data structures of respective additional data processing clusters associated therewith.