Distributed catalog service for data processing platform

• US 10,425,350 B1
• Filed: 11/30/2017
• Issued: 09/24/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares
• Primary Examiner(s): Nguyen, Thuong
• Application Number:
  US15/827,903
• Time in Patent Office: 663 Days
• Field of Search: 709224, 709226, 709248, 709246, 709217, 709203, 715735, 715745
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: H04L 43/065 related to network devices
• CPC Class Codes: H04L 47/762 triggered by the network
• CPC Class Codes: H04L 47/783 Distributed allocation of resources, e.g. bandwidth brokers
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The one or more processing devices are operative to configure a plurality of distributed processing nodes to communicate over a network, to abstract content locally accessible in respective data zones of respective ones of the distributed processing nodes into respective catalogs of a distributed catalog service in accordance with a layered extensible data model, and to provide in the distributed processing nodes a plurality of microservices for performing processing operations on at least one of the layered extensible data model and the catalogs. The layered extensible data model comprises a plurality of layers including a core data model layer and at least one extensions layer. The microservices may comprise at least one microservice to alter the layered extensible data model and at least one microservice to query one or more of the catalogs.

Claims

1. A method comprising:

configuring a plurality of distributed processing nodes, each comprising a processor coupled to a memory, to communicate over a network;

abstracting content locally accessible in respective data zones of respective ones of the distributed processing nodes into respective catalogs of a distributed catalog service in accordance with a layered extensible data model;

providing in the distributed processing nodes a plurality of microservices for performing processing operations on at least one of the layered extensible data model and the catalogs of the distributed catalog service; and

executing an application distributed across at least two of the plurality of distributed processing nodes utilizing the catalogs of the distributed catalog service to determine, for each of the at least two distributed processing nodes, a subset of a plurality of data resources utilized by the application that are located within its corresponding one of the data zones;

wherein each of the catalogs of the distributed catalog service is configured to track data resources within its corresponding one of the data zones through addressing the data resources based on semantic content of the data resources expressed through metadata;

wherein the layered extensible model comprises;

a data layer configured to persist the catalogs of the distributed catalog service;

a core data model layer configured to provide a set of core classes for classifying the data resources in the respective data zones; and

at least one extensions layer configured to extend respective ones of the core classes to at least one of;

one or more additional classes; and

instances of one or more the core classes and the additional classes;

wherein the microservices comprise at least one microservice configured to establish relationships between data resources and metadata using one or more of the core classes, the additional classes, and the instances of the core classes and additional classes; and

wherein the microservices further comprise at least one microservice configured to automate a process of metadata collection and ingestion for one or more discovered data hubs and data sources to populate the catalogs of the distributed catalog service.

2. The method of claim 1 wherein each of at least a subset of the catalogs becomes part of a common catalog domain responsive to completion of a registration process with at least one other one of the catalogs in the subset and wherein the common catalog domain is characterized by a tracking shape determined based at least in part on an order in which the registration processes are performed.

3. The method of claim 1 wherein a first catalog of the distributed catalog service configured to track a first set of resources utilizes at least one of the microservices to identify at least a second catalog of the distributed catalog service configured to track a second set of resources.

4. The method of claim 3 wherein a registration process is carried out between the first and second catalogs in order to permit each such catalog to obtain information regarding the set of resources tracked by the other catalog.

5. The method of claim 4 wherein the first and second catalogs are associated with respective first and second data zones and wherein responsive to completion of the registration process the first and second data zones become part of a common catalog domain that encompasses the first and second data zones.

6. The method of claim 1 wherein a first catalog of the distributed catalog service configured to track a first set of resources within a first one of the data zones can directly access the first set of data resources within the first data zone and wherein the first catalog is further configured to indirectly access via a second catalog of the distributed catalog service a second set of resources tracked by the second catalog within a second one of the data zones.

7. The method of claim 1 wherein the at least one extensions layers comprises at least one of an industry extensions layer and a customer extensions layer.

8. The method of claim 1 wherein the plurality of microservices comprises one or more of the following:

at least one microservice to alter the layered extensible data model;

at least one microservice to query one or more of the catalogs;

at least one microservice to populate one or more of the catalogs with abstracted content;

at least one microservice to integrate inbound content from an inbound data source into a given one of the catalogs;

at least one microservice to provide outbound content from a given one of the catalogs to an output data target;

at least one microservice to publish from a given one of the catalogs to one or more other ones of the catalogs information characterizing resources tracked by the given catalog;

at least one microservice to initiate a probe from a given one of the catalogs to one or more other ones of the catalogs for information characterizing resources tracked by the one or more other catalogs; and

at least one microservice to register one of the catalogs with another one of the catalogs.

9. The method of claim 1 wherein a given one of the catalogs is represented in the form of a graph and wherein at least one of the microservices implements one or more functions for traversing the graph to obtain information characterizing resources tracked by the given catalog.

10. The method of claim 1 further comprising:

extending the layered extensible data model;

configuring one or more additional microservices based at least in part on extensions of the extended layered extensible data model; and

making the additional microservices accessible to the distributed processing nodes via the distributed catalog service.

11. The method of claim 1 further comprising sourcing additional content into one or more of the catalogs of the distributed catalog service from one or more external systems including a metadata management system.

12. The method of claim 1 wherein at least a subset of the distributed processing nodes are implemented in respective distributed data processing clusters corresponding to respective ones of the data zones and wherein the distributed catalog service is implemented at least in part as a plurality of interacting catalog applications running on respective ones of the distributed processing node clusters.

13. The method of claim 12 wherein the clusters comprise respective cloud-based data centers each configured to store locally accessible content of its corresponding data zone in a manner that satisfies one or more specified policies relating to at least one of privacy, security, governance, risk and compliance.

14. The method of claim 1 wherein the content of one or more catalogs of the distributed catalog service comprises a plurality of meta-resources each comprising information characterizing at least one of location and accessibility of a corresponding one of a plurality of data resources.

15. The method of claim 1 wherein the microservices comprise at least one microservice for implementing catalog integration and population services, the catalog integration and population services comprising one or more consulting microservices and one or more service delivery practices microservices.

16. The method of claim 15 wherein the one or more consulting microservices are configured to create processes and procedures for the discovery of the one or more data hubs and data sources across organizations associated with the plurality of distributed processing nodes.

17. The method of claim 16 wherein the one or more service delivery practices microservices are configured to create connectors to automate processing for integrating the one or more discovered data hubs and data sources into the distributed catalog service.

18. The method of claim 16 wherein the one or more service delivery practices microservices are configured to facilitate discovery of data sources within the organizations associated with the plurality of distributed processing nodes and to identify one or more additional data sources to be installed on the plurality of distributed processing nodes to provide connectors and ingestion mechanisms to populate metadata for the one or more discovered data hubs and data sources into the catalogs of the distributed catalog service.

19. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to configure a plurality of distributed processing nodes to communicate over a network;

to abstract content locally accessible in respective data zones of respective ones of the distributed processing nodes into respective catalogs of a distributed catalog service in accordance with a layered extensible data model;

to provide in the distributed processing nodes a plurality of microservices for performing processing operations on at least one of the layered extensible data model and the catalogs of the distributed catalog service; and

to execute an application distributed across at least two of the plurality of distributed processing nodes utilizing the catalogs of the distributed catalog service to determine, for each of the at least two distributed processing nodes, a subset of a plurality of data resources utilized by the application that are located within its corresponding one of the data zones;

wherein each of the catalogs of the distributed catalog service is configured to track a set of data resources within its corresponding one of the data zones through addressing the data resources based on semantic content of the data resources expressed through metadata;

wherein the layered extensible model comprises;

a data layer configured to persist the catalogs of the distributed catalog service;

a core data model layer configured to provide a set of core classes for classifying the data resources in the respective data zones; and

at least one extensions layer configured to extend respective ones of the core classes to at least one of;

one or more additional classes; and

instances of one or more the core classes and the additional classes; and

wherein the microservices comprise at least one microservice configured to establish relationships between data resources and metadata using one or more of the core classes, the additional classes, and the instances of the core classes and additional classes; and

wherein the microservices further comprise at least one microservice configured to automate a process of metadata collection and ingestion for one or more discovered data hubs and data sources to populate the catalogs of the distributed catalog service.

20. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processor is operative;

to configure a plurality of distributed processing nodes to communicate over a network;

to abstract content locally accessible in respective data zones of respective ones of the distributed processing nodes into respective catalogs of a distributed catalog service in accordance with a layered extensible data model;

to provide in the distributed processing nodes a plurality of microservices for performing processing operations on at least one of the layered extensible data model and the catalogs of the distributed catalog service; and

to execute an application distributed across at least two of the plurality of distributed processing nodes utilizing the catalogs of the distributed catalog service to determine, for each of the at least two distributed processing nodes, a subset of a plurality of data resources utilized by the application that are located within its corresponding one of the data zones;

wherein each of the catalogs of the distributed catalog service is configured to track a set of data resources within its corresponding one of the data zones through addressing the data resources based on semantic content of the data resources expressed through metadata;

wherein the layered extensible model comprises;

a data layer configured to persist the catalogs of the distributed catalog service;

a core data model layer configured to provide a set of core classes for classifying the data resources in the respective data zones; and

at least one extensions layer configured to extend respective ones of the core classes to at least one of;

one or more additional classes; and

instances of one or more the core classes and the additional classes;

wherein the microservices comprise at least one microservice configured to establish relationships between data resources and metadata using one or more of the core classes, the additional classes, and the instances of the core classes and additional classes; and

wherein the microservices further comprise at least one microservice configured to automate a process of metadata collection and ingestion for one or more discovered data hubs and data sources to populate the catalogs of the distributed catalog service.