Analytics platform for scalable distributed computations

• US 10,515,097 B2
• Filed: 02/11/2019
• Issued: 12/24/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Masad, Ofri
• Primary Examiner(s): Tokuta, Shean
• Application Number:
  US16/272,517
• Publication Number:
  US 20190317949A1
• Time in Patent Office: 316 Days
• CPC Class Codes: G06F 16/2246 Trees, e.g. B+trees
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

An analytics platform is configured to receive a request to execute a distributed analytics workflow specification comprising multiple distributed computations to be performed across a plurality of data processing clusters associated with respective data zones. For each of a plurality of analytic services to be executed in accordance with the distributed analytics workflow specification, the platform determines if execution of the analytic service is authorized for the request, and for each of a plurality of datasets to be operated on by one or more of the analytic services in accordance with the distributed analytics workflow specification, the platform determines if execution of the one or more analytic services on the dataset is authorized by its corresponding data zone for the request. Responsive to each of the analytic services and each of the datasets being authorized for the request, the platform initiates execution of the distributed computations of the distributed analytics workflow specification across the data processing clusters.

Claims

1. A method comprising:

receiving a request to execute a distributed analytics workflow specification comprising multiple distributed computations to be performed across a plurality of data processing clusters associated with respective data zones;

for each of a plurality of analytic services to be executed in accordance with the distributed analytics workflow specification, determining if execution of the analytic service is authorized for the request;

for each of a plurality of datasets to be operated on by one or more of the analytic services in accordance with the distributed analytics workflow specification, determining if execution of the one or more analytic services on the dataset is authorized by its corresponding data zone for the request; and

responsive to each of the analytic services and each of the datasets being authorized for the request, initiating execution of the distributed computations of the distributed analytics workflow specification across the data processing clusters;

wherein the method is performed by an analytics platform comprising at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein each of at least a subset of the data processing clusters comprises a plurality of processing nodes interconnected by a network.

3. The method of claim 1 wherein the analytic services are identified in the distributed analytics workflow specification and comprise analytic services that were previously registered by respective owning entities with one or more analytic service libraries of the analytics platform.

4. The method of claim 1 wherein the datasets are identified in the distributed analytics workflow specification and comprise datasets that were previously registered by respective owning entities with one or more dataset libraries of the analytics platform.

5. The method of claim 1 wherein the data zones are identified in the distributed analytics workflow specification and comprise data zones that were previously registered by respective owning entities with one or more data zone libraries of the analytics platform.

6. The method of claim 1 wherein a given one of the data zones comprises a data lake that includes one or more of the datasets.

7. The method of claim 1 wherein the analytics platform comprises an insight exchange platform accessible to client devices via a web-based interface.

8. The method of claim 7 wherein receiving the request comprises receiving at least a portion of the request in the insight exchange platform from a particular one of the client devices via the web-based interface.

9. The method of claim 7 wherein a given owning entity registering at least one of an analytic service, a dataset and a data zone with the insight exchange platform is provided with one or more specified credits responsive to successful completion of the distributed computations in accordance with the distributed analytics workflow specification.

10. The method of claim 1 wherein the analytic services are executed on the datasets without the datasets leaving their respective data zones.

11. The method of claim 1 wherein a first one of data processing clusters is implemented in a first cloud of a first type provided by a first cloud service provider, and at least a second one of the data processing clusters is implemented in a second cloud of a second type different than the first type, provided by a second cloud service provider different than the first cloud service provider.

12. The method of claim 11 wherein the first one of data processing clusters implemented in the first cloud utilizes a first local data structure configured to support a first computational framework and wherein at least the second one of the data processing clusters implemented in the second cloud utilizes a second local data structure different than the first local data structure and configured to support a second computational framework different than the first computational framework.

13. The method of claim 12 wherein the first computational framework comprises a batch framework and the second computational framework comprises a streaming framework.

14. The method of claim 1 wherein performing the distributed computations across the data processing clusters associated with respective data zones comprises utilizing local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results, each of at least a subset of the data processing clusters being configured to process data from a dataset of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results, at least one of the data processing clusters being configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results.

15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to receive a request to execute a distributed analytics workflow specification comprising multiple distributed computations to be performed across a plurality of data processing clusters associated with respective data zones;

for each of a plurality of analytic services to be executed in accordance with the distributed analytics workflow specification, to determine if execution of the analytic service is authorized for the request;

for each of a plurality of datasets to be operated on by one or more of the analytic services in accordance with the distributed analytics workflow specification, to determine if execution of the one or more analytic services on the dataset is authorized by its corresponding data zone for the request; and

responsive to each of the analytic services and each of the datasets being authorized for the request, to initiate execution of the distributed computations of the distributed analytics workflow specification across the data processing clusters.

16. The computer program product of claim 15 wherein said at least one processing device implements an analytics platform and wherein the analytic services are identified in the distributed analytics workflow specification and comprise analytic services that were previously registered by respective owning entities with one or more analytic service libraries of the analytics platform.

17. The computer program product of claim 15 wherein said at least one processing device implements an analytics platform and wherein the datasets are identified in the distributed analytics workflow specification and comprise datasets that were previously registered by respective owning entities with one or more dataset libraries of the analytics platform.

18. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processing device is configured;

to receive a request to execute a distributed analytics workflow specification comprising multiple distributed computations to be performed across a plurality of data processing clusters associated with respective data zones;

for each of a plurality of analytic services to be executed in accordance with the distributed analytics workflow specification, to determine if execution of the analytic service is authorized for the request;

for each of a plurality of datasets to be operated on by one or more of the analytic services in accordance with the distributed analytics workflow specification, to determine if execution of the one or more analytic services on the dataset is authorized by its corresponding data zone for the request; and

responsive to each of the analytic services and each of the datasets being authorized for the request, to initiate execution of the distributed computations of the distributed analytics workflow specification across the data processing clusters.

19. The apparatus of claim 18 wherein said at least one processing device implements an analytics platform and wherein the analytic services are identified in the distributed analytics workflow specification and comprise analytic services that were previously registered by respective owning entities with one or more analytic service libraries of the analytics platform.

20. The apparatus of claim 18 wherein said at least one processing device implements an analytics platform and wherein the datasets are identified in the distributed analytics workflow specification and comprise datasets that were previously registered by respective owning entities with one or more dataset libraries of the analytics platform.