Beacon-based distributed data processing platform

• US 10,277,668 B1
• Filed: 12/29/2015
• Issued: 04/30/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares
• Primary Examiner(s): Nguyen, Thuong
• Application Number:
  US14/982,355
• Time in Patent Office: 1,218 Days
• Field of Search: 709201, 709206, 709230, 709221, 709208, 370360, 370389
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G16B 20/00 ICT specially adapted for functional genomics or proteomics, e.g. genotype-phenotype associations
• CPC Class Codes: G16B 30/00 ICT specially adapted for sequence analysis involving nucleotides or amino acids
• CPC Class Codes: G16B 30/10 Sequence alignment; Homology search
• CPC Class Codes: G16B 30/20 Sequence assembly
• CPC Class Codes: G16B 40/00 ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding
• CPC Class Codes: G16B 45/00 ICT specially adapted for bioinformatics-related data visualisation, e.g. displaying of maps or networks
• CPC Class Codes: G16B 5/00 ICT specially adapted for modelling or simulations in systems biology, e.g. gene-regulatory networks, protein interaction networks or metabolic networks
• CPC Class Codes: H04L 43/065 related to network devices
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 47/762 triggered by the network
• CPC Class Codes: H04L 47/783 Distributed allocation of resources, e.g. bandwidth brokers
• CPC Class Codes: H04L 47/827 Aggregation of resource allocation or reservation requests
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A method comprises initiating a first application in a first one of a plurality of distributed processing nodes, and responsive to initiation of the first application, identifying a plurality of beacon entities to be contacted in conjunction with execution of at least a portion of the first application. The method also comprises, for each of at least a subset of the identified beacon entities, initiating an additional application in an additional one of the plurality of distributed processing nodes. The method further comprises aggregating processing results from the first and one or more additional processing nodes, and providing the aggregated processing results to a client. The plurality of distributed processing nodes may comprise a plurality of YARN clusters associated with respective data zones, with each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone.

Claims

1. A method comprising:

initiating a first application in a first one of a plurality of distributed processing nodes, wherein the plurality of distributed processing nodes comprise a plurality of clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone, wherein at least a subset of the plurality of clusters comprise respective Yet Another Resource Negotiator (YARN) clusters;

determining a plurality of data resources to be utilized by at least a portion of the first application, the plurality of data resources comprising at least one data resource not locally accessible within a first data zone associated with the first distributed processing node;

responsive to initiation of the first application, identifying a plurality of beacon entities to be contacted to determine additional ones of the plurality of distributed processing nodes having associated data zones which include the at least one data resource not locally accessible within the first data zone associated with the first distributed processing node, wherein the data zones are associated with respective ones of the beacon entities and further wherein the beacon entities comprise at least one of;

(i) respective participants in a beacon network, and (ii) respective geographically-distributed regional data centers each configured to perform analytics processing utilizing locally accessible data resources of a corresponding data zone;

for each of at least a subset of the identified beacon entities, initiating an additional application in an additional one of the plurality of distributed processing nodes, wherein each of the beacon entities is configured to perform processing operations associated with at least one of the first and one or more additional applications utilizing data resources locally accessible within a corresponding data zone;

aggregating processing results from the first and one or more additional processing nodes; and

providing the aggregated processing results to a client;

wherein the processing results from the first and one or more additional processing nodes preserve at least one policy of those nodes in the respective local data resources wherein the policy relates to at least one of privacy, security, governance, risk and compliance;

wherein the policy specifies;

that processing results generated by a given one of the additional applications initiated in a given one of the additional processing nodes utilizing data resources locally accessible within a given data zone may be transmitted to the first processing node; and

that the data resources locally accessible within the given data zone associated with the given additional processing node utilized to obtain the transmitted processing results are not permitted to be transmitted to the first processing node; and

wherein the method is implemented by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein the first application is initiated via a YARN application programming interface of the client.

3. The method of claim 1 wherein each of the first and one or more additional processing nodes comprises a YARN component having an associated resource manager and wherein the first and one or more additional applications are initiated at least in part utilizing the resource managers of the respective YARN components of the respective first and one or more additional processing nodes.

4. The method of claim 1 wherein the beacon entities have respective activatable beacons.

5. The method of claim 1 further comprising accessing a distributed catalog service to identify at least a subset of the plurality of beacon entities to be contacted in conjunction with execution of at least a portion of the first application.

6. The method of claim 5 wherein the distributed catalog service is distributed over the plurality of processing nodes with each such processing node having visibility of a corresponding distinct portion of the distributed catalog based on its locally accessible data resources.

7. The method of claim 5 wherein the distributed catalog service is configured to store metadata characterizing the beacon entities so as to permit the beacon entities to be organized into a plurality of virtual beacon networks for use in processing respective different types of beacon queries.

8. The method of claim 1 wherein a given one of the applications initiated in a corresponding one of the processing nodes comprises an application master component.

9. The method of claim 8 wherein the application master component is configured to access a resolving interface of a distributed catalog service.

10. The method of claim 1 wherein the additional one of the plurality of distributed processing nodes is selected based at least in part on proximity of the additional processing node to a corresponding one of the beacon entities.

11. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to initiate a first application in a first one of a plurality of distributed processing nodes, wherein the plurality of distributed processing nodes comprise a plurality of clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone, wherein at least a subset of the plurality of clusters comprise respective Yet Another Resource Negotiator (YARN) clusters;

to determine a plurality of data resources to be utilized by at least a portion of the first application, the plurality of data resources comprising at least one data resource not locally accessible within a first data zone associated with the first distributed processing node;

responsive to initiation of the first application, to identify a plurality of beacon entities to be contacted to determine additional ones of the plurality of distributed processing nodes having associated data zones which include the at least one data resource not locally accessible within the first data zone associated with the first distributed processing node, wherein the data zones are associated with respective ones of the beacon entities and further wherein the beacon entities comprise at least one of;

(i) respective participants in a beacon network, and (ii) respective geographically-distributed regional data centers each configured to perform analytics processing utilizing locally accessible data resources of a corresponding data zone;

for each of at least a subset of the identified beacon entities, to initiate an additional application in an additional one of the plurality of distributed processing nodes, wherein each of the beacon entities is configured to perform processing operations associated with at least one of the first and one or more additional applications utilizing data resources locally accessible within a corresponding data zone;

to aggregate processing results from the first and one or more additional processing nodes; and

to provide the aggregated processing results to a client;

wherein the processing results from the first and one or more additional processing nodes preserve at least one policy of those nodes in the respective local data resources wherein the policy relates to at least one of privacy, security, governance, risk and compliance; and

wherein the policy specifies;

that processing results generated by a given one of the additional applications initiated in a given one of the additional processing nodes utilizing data resources locally accessible within a given data zone may be transmitted to the first processing node; and

that the data resources locally accessible within the given data zone associated with the given additional processing node utilized to obtain the transmitted processing results are not permitted to be transmitted to the first processing node.

12. The computer program product of claim 11 wherein the additional one of the plurality of distributed processing nodes is selected based at least in part on proximity of the additional processing node to a corresponding one of the beacon entities.

13. The computer program product of claim 11 wherein the beacon entities have respective activatable beacons.

14. The computer program product of claim 11 wherein a given one of the applications initiated in a corresponding one of the processing nodes comprises an application master component.

15. The computer program product of claim 11 wherein the program code when executed by at least one processing device further causes said at least one processing device to access a distributed catalog service to identify at least a subset of the plurality of beacon entities to be contacted in conjunction with execution of at least a portion of the first application.

16. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processing device is configured;

to initiate a first application in a first one of a plurality of distributed processing nodes, wherein the plurality of distributed processing nodes comprise a plurality of clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone, wherein at least a subset of the plurality of clusters comprise respective Yet Another Resource Negotiator (YARN) clusters;

to determine a plurality of data resources to be utilized by at least a portion of the first application, the plurality of data resources comprising at least one data resource not locally accessible within a first data zone associated with the first distributed processing node;

responsive to initiation of the first application, to identify a plurality of beacon entities to be contacted to determine additional ones of the plurality of distributed processing nodes having associated data zones which include the at least one data resource not locally accessible within the first data zone associated with the first distributed processing node, wherein the data zones are associated with respective ones of the beacon entities and further wherein the beacon entities comprise at least one of;

(i) respective participants in a beacon network, and (ii) respective geographically-distributed regional data centers each configured to perform analytics processing utilizing locally accessible data resources of a corresponding data zone;

for each of at least a subset of the identified beacon entities, to initiate an additional application in an additional one of the plurality of distributed processing nodes, wherein each of the beacon entities is configured to perform processing operations associated with at least one of the first and one or more additional applications utilizing data resources locally accessible within a corresponding data zone;

to aggregate processing results from the first and one or more additional processing nodes; and

to provide the aggregated processing results to a client;

wherein the processing results from the first and one or more additional processing nodes preserve at least one policy of those nodes in the respective local data resources wherein the policy relates to at least one of privacy, security, governance, risk and compliance; and

wherein the policy specifies;

that processing results generated by a given one of the additional applications initiated in a given one of the additional processing nodes utilizing data resources locally accessible within a given data zone may be transmitted to the first processing node; and

that the data resources locally accessible within the given data zone associated with the given additional processing node utilized to obtain the transmitted processing results are not permitted to be transmitted to the first processing node.

17. An information processing system comprising the plurality of distributed processing nodes of claim 16.

18. The apparatus of claim 16 wherein the beacon entities have respective activatable beacons.

19. The apparatus of claim 16 wherein a given one of the applications initiated in a corresponding one of the processing nodes comprises an application master component.

20. The apparatus of claim 16 wherein said at least one processing device is further configured to access a distributed catalog service to identify at least a subset of the plurality of beacon entities to be contacted in conjunction with execution of at least a portion of the first application.