Analytics platform for scalable distributed computations

• US 10,496,926 B2
• Filed: 02/11/2019
• Issued: 12/03/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Masad, Ofri
• Primary Examiner(s): Tokuta, Shean
• Application Number:
  US16/272,555
• Publication Number:
  US 20190266496A1
• Time in Patent Office: 295 Days
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06Q 20/065 using e-cash
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

An analytics platform is configured to provide at least one interface permitting registration of datasets of respective data zones for use in performing distributed computations across a plurality of data processing clusters associated with the respective data zones. The analytics platform is further configured to maintain one or more analytic service libraries each comprising a plurality of analytic services for use in performing the distributed computations, and to initiate execution of a particular subset of the distributed computations, responsive to a corresponding request, using selected ones of the registered datasets and selected ones of the analytic services. The analytics platform allocates to an entity associated with one of more of the selected registered datasets one or more specified credits responsive to successful completion of the particular subset of the distributed computations. The analytic services are illustratively executed on the datasets without the datasets leaving their respective data zones.

Claims

1. A method comprising:

providing at least one interface permitting registration of datasets of respective data zones for use in performing distributed computations across a plurality of data processing clusters associated with the respective data zones;

maintaining one or more analytic service libraries each comprising a plurality of analytic services for use in performing the distributed computations;

initiating execution of a particular subset of the distributed computations, responsive to a corresponding request, using selected ones of the registered datasets and selected ones of the analytic services; and

allocating to an entity associated with one of more of the selected registered datasets one or more specified credits responsive to successful completion of the particular subset of the distributed computations;

wherein the method is performed by an analytics platform comprising at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein each of at least a subset of the data processing clusters comprises a plurality of processing nodes interconnected by a network.

3. The method of claim 1 wherein the selected analytic services comprise analytic services that were previously registered by respective owning entities with one or more analytic service libraries of the analytics platform.

4. The method of claim 1 wherein the selected datasets comprise datasets that were previously registered by respective owning entities with one or more dataset libraries of the analytics platform.

5. The method of claim 1 wherein the data zones comprise data zones that were previously registered by respective owning entities with one or more data zone libraries of the analytics platform.

6. The method of claim 1 wherein a given one of the data zones comprises a data lake that includes one or more of the datasets.

7. The method of claim 1 wherein the analytics platform comprises an insight exchange platform accessible to client devices via a web-based interface.

8. The method of claim 7 wherein the given request is received in the insight exchange platform from a particular one of the client devices via the web-based interface.

9. The method of claim 7 wherein a given owning entity registering at least one of an analytic service, a dataset and a data zone with the insight exchange platform is provided with one or more specified credits responsive to successful completion of the particular subset of the distributed computations.

10. The method of claim 1 wherein the selected analytic services are executed by one or more of the data processing clusters using the selected registered datasets without the datasets leaving their respective data zones.

11. The method of claim 1 wherein a first one of data processing clusters is implemented in a first cloud of a first type provided by a first cloud service provider, and at least a second one of the data processing clusters is implemented in a second cloud of a second type different than the first type, provided by a second cloud service provider different than the first cloud service provider.

12. The method of claim 11 wherein the first one of data processing clusters implemented in the first cloud utilizes a first local data structure configured to support a first computational framework and wherein at least the second one of the data processing clusters implemented in the second cloud utilizes a second local data structure different than the first local data structure and configured to support a second computational framework different than the first computational framework.

13. The method of claim 12 wherein the first computational framework comprises a batch framework and the second computational framework comprises a streaming framework.

14. The method of claim 1 wherein performing the distributed computations across the data processing clusters associated with respective data zones comprises utilizing local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results, each of at least a subset of the data processing clusters being configured to process data from a dataset of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results, at least one of the data processing clusters being configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results.

15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to provide at least one interface permitting registration of datasets of respective data zones for use in performing distributed computations across a plurality of data processing clusters associated with the respective data zones;

to maintain one or more analytic service libraries each comprising a plurality of analytic services for use in performing the distributed computations;

to initiate execution of a particular subset of the distributed computations, responsive to a corresponding request, using selected ones of the registered datasets and selected ones of the analytic services; and

to allocate to an entity associated with one of more of the selected registered datasets one or more specified credits responsive to successful completion of the particular subset of the distributed computations.

16. The computer program product of claim 15 wherein said at least one processing device implements an analytics platform providing analytic insight exchange functionality accessible to client devices via a web-based interface.

17. The computer program product of claim 15 wherein the selected analytic services are executed by one or more of the data processing clusters using the selected registered datasets without the datasets leaving their respective data zones.

18. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processing device is configured;

to provide at least one interface permitting registration of datasets of respective data zones for use in performing distributed computations across a plurality of data processing clusters associated with the respective data zones;

to maintain one or more analytic service libraries each comprising a plurality of analytic services for use in performing the distributed computations;

to initiate execution of a particular subset of the distributed computations, responsive to a corresponding request, using selected ones of the registered datasets and selected ones of the analytic services; and

to allocate to an entity associated with one of more of the selected registered datasets one or more specified credits responsive to successful completion of the particular subset of the distributed computations.

19. The apparatus of claim 18 wherein said at least one processing device implements an analytics platform providing analytic insight exchange functionality accessible to client devices via a web-based interface.

20. The apparatus of claim 18 wherein the selected analytic services are executed by one or more of the data processing clusters using the selected registered datasets without the datasets leaving their respective data zones.