Distributed catalog service for multi-cluster data processing platform

• US 10,270,707 B1
• Filed: 12/29/2015
• Issued: 04/23/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Lutati, Benny, Gudes, Ehud, Gonen, Yaron, Singer, Ido, Meisels, Amnon, Vijendra, Sudhir
• Primary Examiner(s): Nguyen, Thuong
• Application Number:
  US14/982,351
• Time in Patent Office: 1,211 Days
• Field of Search: 709224, 709205, 709221, 709248, 709246, 709228, 370392
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G16B 20/00 ICT specially adapted for functional genomics or proteomics, e.g. genotype-phenotype associations
• CPC Class Codes: G16B 30/00 ICT specially adapted for sequence analysis involving nucleotides or amino acids
• CPC Class Codes: G16B 30/10 Sequence alignment; Homology search
• CPC Class Codes: G16B 30/20 Sequence assembly
• CPC Class Codes: G16B 40/00 ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding
• CPC Class Codes: G16B 45/00 ICT specially adapted for bioinformatics-related data visualisation, e.g. displaying of maps or networks
• CPC Class Codes: G16B 5/00 ICT specially adapted for modelling or simulations in systems biology, e.g. gene-regulatory networks, protein interaction networks or metabolic networks
• CPC Class Codes: H04L 43/065 related to network devices
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 47/762 triggered by the network
• CPC Class Codes: H04L 47/783 Distributed allocation of resources, e.g. bandwidth brokers
• CPC Class Codes: H04L 47/827 Aggregation of resource allocation or reservation requests
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A first portion of a distributed catalog service is implemented for a given one of a plurality of distributed processing node clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone. The first portion of the distributed catalog service receives a request to identify for each of a plurality of data resources to be utilized by an application initiated in the given cluster whether the data resource is a local or remote data resource relative to the given cluster, and provides a response to the request. The first portion of the distributed catalog service in combination with additional portions implemented for respective additional ones of the distributed processing node clusters collectively provide the distributed catalog service with capability to resolve local or remote status of data resources in each of the data zones.

Claims

1. A method comprising:

implementing a first portion of a distributed catalog service for a given one of a plurality of distributed processing node clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone;

receiving in the first portion of the distributed catalog service a request to identify for each of a plurality of data resources to be utilized by an application initiated in the given cluster whether the data resource is a local data resource or a remote data resource relative to the given cluster; and

providing from the first portion of the distributed catalog service a response to the request;

wherein the first portion of the distributed catalog service in combination with additional portions implemented for respective additional ones of the plurality of distributed processing node clusters collectively provide the distributed catalog service with capability to resolve local or remote status of data resources in the data zones of each of the clusters responsive to requests from any other one of the clusters;

wherein a given one of the portions of the distributed catalog service in conjunction with its initiation as a Yet Another Resource Negotiator (YARN) application is registered as a service with a service registry of a resource manager of the corresponding cluster; and

wherein the method is implemented by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein the plurality of distributed processing node clusters comprises respective YARN clusters.

3. The method of claim 1 wherein the clusters comprise respective geographically-distributed regional data centers each configured to perform analytics processing utilizing the locally accessible data resources of its corresponding data zone.

4. The method of claim 1 wherein implementing a first portion of the distributed catalog service comprises:

providing a browsing interface accessible to one or more clients; and

providing a resolving interface accessible to an application master component of the application;

wherein the application master component is configured to access the resolving interface of the distributed catalog service in order to determine for each of the plurality of data resources to be utilized by the application whether the data resource is a local data resource or a remote data resource relative to the given cluster.

5. The method of claim 1 wherein the distributed catalog service is distributed over the clusters with each of the clusters having visibility of only its corresponding one of the portions of the distributed catalog service.

6. The method of claim 1 wherein the portions of the distributed catalog service are implemented as respective YARN applications running on respective ones of the clusters.

7. The method of claim 1 wherein at least one of the portions of the distributed catalog service comprises a plurality of entries with each such entry comprising a meta-resource comprising information characterizing location and accessibility of a corresponding one of the data resources.

8. The method of claim 7 wherein the meta-resource for a given local data resource comprises a file path to a storage location of that local data resource in the corresponding cluster.

9. The method of claim 7 wherein the meta-resource for a given remote data resource comprises information identifying another cluster for which that data resource is a local data resource.

10. The method of claim 7 wherein the meta-resource further comprises information regarding transformation of the data resource into one or more designated formats.

11. The method of claim 7 wherein a resolving interface of the portion of the distributed catalog service returns a given meta-resource responsive to a request that includes a corresponding meta-resource identifier.

12. The method of claim 1 wherein the service registry of the resource manager of the cluster is utilized to identify at least one of a browsing interface and a resolving interface of the given portion of the distributed catalog service implemented within that cluster.

13. The method of claim 1 wherein at least one of the portions of the distributed catalog service is configured in accordance with a configuration object that is stored in a predetermined storage location of the corresponding cluster.

14. The method of claim 1 wherein the method is performed at least in part in a worldwide data node coupled to one or more of the clusters.

15. The method of claim 1 wherein the method is performed at least in part in a worldwide data node that comprises a processing node of a given one of the clusters.

16. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to implement a first portion of a distributed catalog service for a given one of a plurality of distributed processing node clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone;

to receive in the first portion of the distributed catalog service a request to identify for each of a plurality of data resources to be utilized by an application initiated in the given cluster whether the data resource is a local data resource or a remote data resource relative to the given cluster; and

to provide from the first portion of the distributed catalog service a response to the request;

wherein the first portion of the distributed catalog service in combination with additional portions implemented for respective additional ones of the plurality of distributed processing node clusters collectively provide the distributed catalog service with capability to resolve local or remote status of data resources in the data zones of each of the clusters responsive to requests from any other one of the clusters; and

wherein a given one of the portions of the distributed catalog service in conjunction with its initiation as a Yet Another Resource Negotiator (YARN) application is registered as a service with a service registry of a resource manager of the corresponding cluster.

17. The computer program product of claim 16 wherein at least one of the portions of the distributed catalog service comprises a plurality of entries with each such entry comprising a meta-resource comprising information characterizing location and accessibility of a corresponding one of the data resources.

18. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processor is configured;

to implement a first portion of a distributed catalog service for a given one of a plurality of distributed processing node clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone;

to receive in the first portion of the distributed catalog service a request to identify for each of a plurality of data resources to be utilized by an application initiated in the given cluster whether the data resource is a local data resource or a remote data resource relative to the given cluster; and

to provide from the first portion of the distributed catalog service a response to the request;

wherein the first portion of the distributed catalog service in combination with additional portions implemented for respective additional ones of the plurality of distributed processing node clusters collectively provide the distributed catalog service with capability to resolve local or remote status of data resources in the data zones of each of the clusters responsive to requests from any other one of the clusters; and

wherein a given one of the portions of the distributed catalog service in conjunction with its initiation as a Yet Another Resource Negotiator (YARN) application is registered as a service with a service registry of a resource manager of the corresponding cluster.

19. The apparatus of claim 18 wherein the service registry of the resource manager of the cluster is utilized to identify at least one of a browsing interface and a resolving interface of the given portion of the distributed catalog service implemented within that cluster.

20. An information processing system comprising the plurality of distributed processing node clusters of claim 18.