Integration of distributed data processing platform with one or more distinct supporting platforms

• US 10,541,938 B1
• Filed: 02/01/2018
• Issued: 01/21/2020
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Timmerman, Hans, Florissi, Patricia Gomes Soares
• Primary Examiner(s): Nguyen, Thuong
• Application Number:
  US15/886,069
• Time in Patent Office: 719 Days
• Field of Search: 709224, 709217, 709219, 707760, 707692, 711133, 711103, 711136
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: H04L 41/0806 for initial configuration or provisioning, e.g. plug-and-play
• CPC Class Codes: H04L 41/0895 Configuration of virtualised networks or elements, e.g. virtualised network function or OpenFlow elements
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 43/065 related to network devices
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 43/20 the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
• CPC Class Codes: H04L 47/762 triggered by the network
• CPC Class Codes: H04L 47/783 Distributed allocation of resources, e.g. bandwidth brokers
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The one or more processing devices are operative to configure a plurality of distributed processing nodes to communicate over a network, to obtain metadata characterizing data locally accessible in respective data zones of respective ones of the distributed processing nodes, and to populate catalog instances of a distributed catalog service for respective ones of the data zones utilizing the obtained metadata. Distributed data analytics are performed in the distributed processing nodes utilizing the populated catalog instances of the distributed catalog service and the locally accessible data of the respective data zones. The metadata characterizing the locally accessible data is illustratively obtained in a metadata repository from at least one of a master data management platform and a governance, risk and compliance platform.

Claims

1. A method comprising:

configuring a plurality of distributed processing nodes, each comprising a processor coupled to a memory, to communicate over a network;

obtaining metadata characterizing data locally accessible in respective data zones of respective ones of the distributed processing nodes;

populating catalog instances of a distributed catalog service for respective ones of the data zones utilizing the obtained metadata; and

performing distributed data analytics for a given analytics job in the distributed processing nodes utilizing the populated catalog instances of the distributed catalog service and the locally accessible data of the respective data zones;

wherein the given analytics job utilizes one or more first data items locally accessible within a first one of the data zones of a first one of the distributed processing nodes and one or more second data items locally accessible within at least a second one of the data zones of at least a second one of the distributed processing nodes;

wherein the obtained metadata comprises one or more metadata tags identifying at least one of the one or more first data items and the one or more second data items, a first entrance key permitting access to a first internal network of the first data zone, and a second entrance key permitting access to a second internal network of the second data zone;

wherein populating the catalog instances of the distributed catalog service further comprises provisioning a first populated catalog instance associated with the first data zone of the first distributed processing node with the first entrance key and provisioning a second populated catalog instance associated with the second data zone of the second distributed processing node with the second entrance key, and

wherein performing distributed data analytics for the given analytics job comprises at least one of;

the first populated catalog instance associated with the first data zone of the first distributed processing node utilizing at least one of the one or more metadata tags to map to one or more first physical storage locations of one or more of the first data items in one or more first data storage devices of the first distributed processing node and utilizing the first entrance key to access one or more of the first data items at the one or more first physical storage locations in the one or more first data storage devices of the first distributed processing node; and

the second populated catalog instance associated with the second data zone of the second distributed processing node utilizing at least one of the one or more metadata tags to map to one or more second physical storage locations of one or more of the second data items in one or more second data storage devices of the second distributed processing node and utilizing the second entrance key to access one or more of the second data items at the one or more second physical storage locations in the one or more second data storage devices of the second distributed processing node.

2. The method of claim 1 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a master data management platform.

3. The method of claim 2 wherein metadata obtained from the master data management platform comprises at least one of location, identification and access control for each of a plurality of data items of the locally accessible data.

4. The method of claim 2 wherein the master data management platform is implemented at least in part on a processing platform that implements at least one of the distributed processing nodes.

5. The method of claim 2 wherein the distributed processing nodes comprise respective WWH nodes of a WWH platform that is at least partially integrated with the master data management platform.

6. The method of claim 2 further comprising providing at least one result of the distributed data analytics back to the master data management platform.

7. The method of claim 1 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a governance, risk and compliance platform.

8. The method of claim 7 wherein metadata obtained from the governance, risk and compliance platform comprises at least one of a policy, a rule and a restriction for each of a plurality of data items of the locally accessible data.

9. The method of claim 7 further comprising one or more of:

providing at least one result of the distributed data analytics back to the governance, risk and compliance platform; and

providing at least one of audit information, log information and notification information from at least one of the distributed processing nodes back to the governance, risk and compliance platform.

10. The method of claim 7 wherein the metadata comprises at least a subset of:

an external address of one of the distributed processing nodes corresponding to a given one of the data zones;

governance, risk and compliance information for use in conjunction with a catalog instance associated with the given data zone; and

an internal address of a data processing location within the internal network of the given data zone;

wherein one or more analytics results generated using the catalog instance in the data processing location of the given data zone are returned to the governance, risk and compliance platform from the given data zone.

11. The method of claim 1 wherein obtaining metadata comprises obtaining metadata in a metadata repository from at least one of a master data management platform and a governance, risk and compliance platform.

12. The method of claim 11 wherein the metadata repository is implemented at least in part within the master data management platform.

13. The method of claim 1 wherein a first catalog of the distributed catalog service configured to track a first set of resources within a first one of the data zones can directly access the first set of resources within the first data zone and wherein the first catalog is further configured to indirectly access via a second catalog of the distributed catalog service a second set of resources tracked by the second catalog within a second one of the data zones.

14. The method of claim 1 wherein at least a subset of the distributed processing nodes are implemented in respective distributed data processing clusters corresponding to respective ones of the data zones and wherein the distributed catalog service is implemented at least in part as a plurality of interacting catalog applications running on respective ones of the distributed processing node clusters.

15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to configure a plurality of distributed processing nodes to communicate over a network;

to obtain metadata characterizing data locally accessible in respective data zones of respective ones of the distributed processing nodes;

to populate catalog instances of a distributed catalog service for respective ones of the data zones utilizing the obtained metadata; and

to perform distributed data analytics for a given analytics job in the distributed processing nodes utilizing the populated catalog instances of the distributed catalog service and the locally accessible data of the respective data zones;

wherein the given analytics job utilizes one or more first data items locally accessible within a first one of the data zones of a first one of the distributed processing nodes and one or more second data items locally accessible within at least a second one of the data zones of at least a second one of the distributed processing nodes;

wherein the obtained metadata comprises one or more metadata tags identifying at least one of the one or more first data items and the one or more second data items, a first entrance key permitting access to a first internal network of the first data zone, and a second entrance key permitting access to a second internal network of the second data zone;

wherein populating the catalog instances of the distributed catalog service further comprises provisioning a first populated catalog instance associated with the first data zone of the first distributed processing node with the first entrance key and provisioning a second populated catalog instance associated with the second data zone of the second distributed processing node with the second entrance key; and

wherein performing distributed data analytics for the given analytics job comprises at least one of;

the first populated catalog instance associated with the first data zone of the first distributed processing node utilizing at least one of the one or more metadata tags to map to one or more first physical storage locations of one or more of the first data items in one or more first data storage devices of the first distributed processing node and utilizing the first entrance key to access one or more of the first data items at the one or more first physical storage locations in the one or more first data storage devices of the first distributed processing node; and

the second populated catalog instance associated with the second data zone of the second distributed processing node utilizing at least one of the one or more metadata tags to map to one or more second physical storage locations of one or more of the second data items in one or more second data storage devices of the second distributed processing node and utilizing the second entrance key to access one or more of the second data items at the one or more second physical storage locations in the one or more second data storage devices of the second distributed processing node.

16. The computer program product of claim 15 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a master data management platform.

17. The computer program product of claim 15 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a governance, risk and compliance platform.

18. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processor is operative;

to configure a plurality of distributed processing nodes to communicate over a network;

to obtain metadata characterizing data locally accessible in respective data zones of respective ones of the distributed processing nodes;

to populate catalog instances of a distributed catalog service for respective ones of the data zones utilizing the obtained metadata; and

to perform distributed data analytics for a given analytics job in the distributed processing nodes utilizing the populated catalog instances of the distributed catalog service and the locally accessible data of the respective data zones;

wherein the given analytics job utilizes one or more first data items locally accessible within a first one of the data zones of a first one of the distributed processing nodes and one or more second data items locally accessible within at least a second one of the data zones of at least a second one of the distributed processing nodes;

wherein the obtained metadata comprises one or more metadata tags identifying at least one of the one or more first data items and the one or more second data items, a first entrance key permitting access to a first internal network of the first data zone, and a second entrance key permitting access to a second internal network of the second data zone;

wherein populating the catalog instances of the distributed catalog service further comprises provisioning a first populated catalog instance associated with the first data zone of the first distributed processing node with the first entrance key and provisioning a second populated catalog instance associated with the second data zone of the second distributed processing node with the second entrance key, and

wherein performing distributed data analytics for the given analytics job comprises at least one of;

the first populated catalog instance associated with the first data zone of the first distributed processing node utilizing at least one of the one or more metadata tags to map to one or more first physical storage locations of one or more of the first data items in one or more first data storage devices of the first distributed processing node and utilizing the first entrance key to access one or more of the first data items at the one or more first physical storage locations in the one or more first data storage devices of the first distributed processing node; and

the second populated catalog instance associated with the second data zone of the second distributed processing node utilizing at least one of the one or more metadata tags to map to one or more second physical storage locations of one or more of the second data items in one or more second data storage devices of the second distributed processing node and utilizing the second entrance key to access one or more of the second data items at the one or more second physical storage locations in the one or more second data storage devices of the second distributed processing node.

19. The apparatus of claim 18 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a master data management platform.

20. The apparatus of claim 18 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a governance, risk and compliance platform.