Blockchain integration for scalable distributed computations

• US 10,509,684 B2
• Filed: 02/07/2019
• Issued: 12/17/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Masad, Ofri
• Primary Examiner(s): Tokuta, Shean
• Application Number:
  US16/269,830
• Publication Number:
  US 20190188046A1
• Time in Patent Office: 313 Days
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: H04L 2463/121 Timestamp
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 63/0209 Architectural arrangements, e.g. perimeter networks or demilitarized zones
• CPC Class Codes: H04L 63/0407 wherein the identity of one or more communicating identities is hidden cryptographic mechanisms or cryptographic arrangements for anonymous credentials or for identity based cryptographic systems H04L9/00
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 9/0637 Modes of operation, e.g. cipher block chaining [CBC], electronic codebook [ECB] or Galois/counter mode [GCM]
• CPC Class Codes: H04L 9/12 Transmitting and receiving encryption devices synchronised or initially set up in a particular manner
• CPC Class Codes: H04L 9/3239 involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD
• CPC Class Codes: H04L 9/3297 involving time stamps, e.g. generation of time stamps
• CPC Class Codes: H04L 9/50 using hash chains, e.g. blockchains or hash trees

Abstract

An apparatus is configured to initiate distributed computations across a plurality of data processing clusters associated with respective data zones, to utilize local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results, and to update at least one distributed ledger maintained by one or more of the plurality of data processing clusters to incorporate one or more blocks each characterizing at least a portion of the distributed computations. Each of at least a subset of the data processing clusters is configured to process data from a data source of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results. At least one of the data processing clusters is configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results.

Claims

1. A method comprising:

initiating distributed computations across a plurality of data processing clusters associated with respective data zones;

utilizing local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results; and

updating at least one distributed ledger maintained by one or more of the plurality of data processing clusters to incorporate one or more blocks each characterizing at least a portion of the distributed computations;

each of at least a subset of the data processing clusters being configured to process data from a data source of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results;

at least one of the data processing clusters being configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results;

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein each of at least a subset of the data processing clusters comprises a plurality of processing nodes interconnected by a network with at least one of the plurality of processing nodes comprising a ledger maintenance node.

3. The method of claim 1 wherein initiating the distributed computations comprises initiating the distributed computations from a given one of the data processing clusters and further wherein utilizing the local processing results comprises combining the local processing results in the given one of the data processing clusters.

4. The method of claim 1 wherein the distributed ledger is created responsive to the initiating of the distributed computations.

5. The method of claim 1 wherein the distributed ledger is collectively maintained by the plurality of data processing clusters and comprises a plurality of blocks that are sequentially added to the distributed ledger by respective ones of the data processing clusters of respective ones of the data zones.

6. The method of claim 1 wherein a given block added to the distributed ledger comprises information characterizing one or more local computations performed by a given one of the data processing clusters.

7. The method of claim 1 wherein a given block added to the distributed ledger comprises information characterizing one or more global computations performed by a given one of the data processing clusters.

8. The method of claim 1 wherein a given block added to the distributed ledger comprises one or more of:

information identifying a corresponding one of the data processing clusters; and

information identifying a corresponding one of the data zones.

9. The method of claim 1 wherein a given block added to the distributed ledger comprises one or more of:

information identifying particular local code utilized by a given one of the data processing clusters in performing one or more local computations; and

information identifying particular global code utilized by a given one of the data processing clusters in performing one or more global computations.

10. The method of claim 1 wherein a given block added to the distributed ledger comprises one or more of:

at least one of the local processing results; and

at least one of the global processing results.

11. The method of claim 1 wherein a given block added to the distributed ledger comprises one or more of:

an identifier of an entity initiating the distributed computations;

a timestamp associated with initiation of the distributed computations;

identifiers of respective data zones participating in the distributed computations;

identifiers of respective data sources providing data for one or more local computations; and

identifiers of respective meta-resources to be utilized for one or more local computations.

12. The method of claim 1 wherein a given block added to the distributed ledger is also added to one or more other distributed ledgers.

13. The method of claim 1 wherein a plurality of distributed ledgers are maintained for respective ones of the data zones.

14. The method of claim 1 wherein a plurality of distributed ledgers are maintained for respective datasets utilized in respective ones of the data zones in performing the local computations.

15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to initiate distributed computations across a plurality of data processing clusters associated with respective data zones;

to utilize local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results; and

to update at least one distributed ledger maintained by one or more of the plurality of data processing clusters to incorporate one or more blocks each characterizing at least a portion of the distributed computations;

each of at least a subset of the data processing clusters being configured to process data from a data source of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results;

at least one of the data processing clusters being configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results.

16. The computer program product of claim 15 wherein the distributed ledger is created responsive to the initiating of the distributed computations.

17. The computer program product of claim 15 wherein the distributed ledger is collectively maintained by the plurality of data processing clusters and comprises a plurality of blocks that are sequentially added to the distributed ledger by respective ones of the data processing clusters of respective ones of the data zones.

18. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processing device is configured;

to initiate distributed computations across a plurality of data processing clusters associated with respective data zones;

to utilize local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results; and

to update at least one distributed ledger maintained by one or more of the plurality of data processing clusters to incorporate one or more blocks each characterizing at least a portion of the distributed computations;

each of at least a subset of the data processing clusters being configured to process data from a data source of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results;

at least one of the data processing clusters being configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results.

19. The apparatus of claim 18 wherein the distributed ledger is created responsive to the initiating of the distributed computations.

20. The apparatus of claim 18 wherein the distributed ledger is collectively maintained by the plurality of data processing clusters and comprises a plurality of blocks that are sequentially added to the distributed ledger by respective ones of the data processing clusters of respective ones of the data zones.