Multi-framework distributed computation

• US 10,505,863 B1
• Filed: 10/31/2017
• Issued: 12/10/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Masad, Ofri
• Primary Examiner(s): Tokuta, Shean
• Application Number:
  US15/799,389
• Time in Patent Office: 770 Days
• CPC Class Codes: G06Q 10/06313 Resource planning in a project environment
• CPC Class Codes: G06Q 30/0283 Price estimation or determination
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70

Abstract

A computing device of a data zone includes a persistent storage and a processor. The persistent storage includes a locked data batch of the data zone. The processor obtains an upstream computation request; instantiates a computation framework to process the locked data batch based on a global data batch specified in the upstream computation request; instantiates a downstream computation manager to manage a downstream computation; and instantiates, by the downstream computation manager, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone.

Claims

1. A computing device of a first data zone, comprising:

a persistent storage comprising a first locked data batch of the first data zone; and

a processor programmed to;

obtain an upstream computation request;

instantiate a first computation framework to process the first locked data batch based on a global data batch specified in the upstream computation request;

instantiate a downstream computation manager to manage a downstream computation;

instantiate, by the downstream computation manager, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone;

obtain a first computation result from the second computation framework; and

generate a second computation result using;

the first computation framework,

the first locked data batch, and

the first computation result,

wherein the first locked data batch and the first computation result match a same grouping criteria.

2. The computing device of claim 1, wherein the second locked data batch cannot be transmitted to the computing device.

3. The computing device of claim 1, wherein the upstream computation request is a global computation request generated by a client.

4. The computing device of claim 1, wherein the upstream computation request is a downstream computation request that was instantiated by a third computation framework.

5. The computing device of claim 1, wherein the first computation framework comprises:

a local computation that generates a first computation result stored on the persistent storage,

wherein the local computation is based on the global data batch of the upstream computation request.

6. The computing device of claim 5, wherein the global data batch specifies:

the first locked data batch, and

the second locked data batch.

7. The computing device of claim 5, wherein the global data batch specifies a data type of the first locked data batch, wherein a data type of the second locked data batch is the same as the data type of the first locked data batch.

8. The computing device of claim 1, wherein the processor is further programmed to:

before instantiating the second computation framework;

determine, by the downstream computation manager, a first computing resource cost of processing the second locked data batch using the second computing device;

determine, by the downstream computation manager, a second computing resource cost of processing the second locked data batch using a third computing device of a third data zone; and

select, by the downstream computation manager, to instantiate the second computing framework in the second computing device based on the first computing resource cost and the second computing resource cost.

9. The computing device of claim 1, wherein instantiating, by the downstream computation manager, the second computation framework in the second computing device of the second data zone to process the second locked data batch of the second data zone comprises:

generating a downstream computation request; and

sending the generated downstream computation request to the second computing device.

10. The computing device of claim 9, wherein the downstream computation request comprises a description of a second local computation to be performed by the second computing device.

11. The computing device of claim 9, wherein the downstream computation request specifies a type of computation to be performed as a second local computation to be performed by the second computing device.

12. The computing device of claim 9, wherein the downstream computation request specifies a plurality of types of computations, wherein the downstream computation request specifies that one of the plurality of types of computations is to be performed as a local computation of the second computing device.

13. The computing device of claim 1, wherein instantiating the computation framework to process the first locked data batch based on the global data batch specified in the upstream computation request comprises:

determining a format of the first locked data batch;

determining a computational cost of each of a plurality of computation types for the format of the first locked data batch;

determining a computation type of the plurality of computation types that has a relatively low computation cost of the computational costs of each of the plurality of computation types; and

instantiating a local computation of the determined computation type using the locked data batch.

14. The computing device of claim 13, wherein the local computation is one selected from a group consisting of a map/reduce computation, a split-apply-combine computation, and a partially parallel computation.

15. The computing device of claim 1, wherein the first data zone is a first cloud computing environment comprising the computing device.

16. The computing device of claim 15, wherein the second data zone is a second cloud computing environment comprising the second computing device.

17. The computing device of claim 1, wherein generating the second computation result comprises:

generating a third computation result based on the first locked data batch;

generating the second computation result using the third computation result and the first computation result; and

marking the second computation result as being contemporaneous to the first computation result.

18. A method of operating a computing device of a first data zone, comprising:

obtaining, by the computing device, an upstream computation request;

instantiating, by the computing device, a first computation framework to process a first locked data batch based on a global data batch specified in the upstream computation request, wherein the first locked data batch is stored on a persistent storage of the computing device;

instantiating, by the computing device, a downstream computation manager on the computing device to manage a downstream computation;

instantiating, by the downstream computation manager of the computing device, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone;

obtaining a first computation result from the second computation framework; and

generating a second computation result using;

the first computation framework,

the first locked data batch, and

the first computation result,

wherein the first locked data batch and the first computation result match a same grouping criteria.

19. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for operating a computing device of a first data zone, the method comprising:

obtaining, by the computing device, an upstream computation request;

instantiating, by the computing device, a first computation framework to process a first locked data batch based on a global data batch specified in the upstream computation request, wherein the first locked data batch is stored on a persistent storage of the computing device;

instantiating, by the computing device, a downstream computation manager on the computing device to manage a downstream computation;

instantiating, by the downstream computation manager of the computing device, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone;

obtaining a first computation result from the second computation framework; and

generating a second computation result using;

the first computation framework,

the first locked data batch, and

the first computation result,

wherein the first locked data batch and the first computation result match a same grouping criteria.

20. The non-transitory computer readable medium of claim 19, wherein the second locked data batch cannot be transmitted to the computing device.