Distributed analytics platform

• US 10,122,806 B1
• Filed: 12/31/2015
• Issued: 11/06/2018
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Berry, Orna
• Primary Examiner(s): Meky, Moustafa M
• Assistant Examiner(s): Mendaye, Kidest
• Application Number:
  US14/986,276
• Time in Patent Office: 1,041 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

An apparatus comprises an analytics platform having a plurality of distributed processing nodes. Data from a data source is received at a first one of the processing nodes. One or more analytics operations are performed on the data at the first processing node, and the data is updated at the first processing node based on results of the one or more analytics operations. The updated data is transmitted to another one of the processing nodes along a data path between the data source and a data destination. The performing, updating and transmitting are repeated for the other processing node and for one or more additional distributed processing nodes of the analytics platform along the data path. The analytics platform thereby performs distributed analytics processing on the data over multiple distributed processing nodes as the data moves through the data path from the data source to the data destination.

Claims

1. A method comprising:

receiving data from a first data source at a first one of a plurality of distributed processing nodes of an analytics platform;

performing one or more analytics operations on the data at the first processing node;

updating the data at the first processing node based on results of the one or more analytics operations;

transmitting the updated data to another one of the processing nodes along a first data path between the first data source and a first data destination; and

repeating the performing, updating and transmitting for the other processing node and for one or more additional distributed processing nodes of the analytics platform along the first data path;

the analytics platform thereby performing distributed analytics processing on the data over multiple ones of the distributed processing nodes as the data moves through the first data path from the first data source to the first data destination;

wherein the method is implemented by at least one processing device comprising a processor coupled to a memory;

wherein each of at least a subset of the distributed processing nodes is a part of multiple distinct data paths and performs distinct analytics operations for different ones of the data paths such that a role of a given one of the processing nodes in the analytics platform varies based at least in part on the particular data path over which data is received for processing;

wherein the first processing node is part of the subset of the distributed processing nodes and is a part of at least two distinct data paths including the first data path between the first data source and the first data destination and at least a second data path between a second data source and a second data destination;

wherein the first processing node performs a first type of analytics operation for a first role when receiving data on the first data path;

wherein the first processing node performs a second type of analytics operation for a second role when receiving data on the second data path;

wherein the first and second roles comprise respective distinct sets of one or more analytics-related processing tasks;

wherein the first role and the second role of the first processing node are defined by a distributed service of a software-defined function of the analytics platform; and

wherein updating the data based on results of the one or more analytics operations comprises two or more of;

adding metadata to the data;

modifying existing metadata of the data; and

transforming the data into a different format.

2. The method of claim 1 wherein the analytics platform is configured to determine the first data path as at least a portion of an inverse spanning tree.

3. The method of claim 1 wherein the distributed processing nodes of the analytics platform comprise a plurality of distributed processing node clusters associated with respective data zones, each of the clusters being configured to perform analytics operations utilizing local data resources locally accessible within its corresponding data zone.

4. The method of claim 3 wherein the plurality of distributed processing node clusters comprise respective Yet Another Resource Negotiator (YARN) clusters.

5. The method of claim 1 wherein the plurality of distributed processing nodes are arranged in multiple distinct abstraction levels with each such level implementing a different type of analytics functionality.

6. The method of claim 1 wherein the data comprises IoT data from a data source comprising one or more IoT devices.

7. The method of claim 1 wherein the analytics platform is configured to optimize configuration of the first data path for one or more specified parameters relating to transmission of the data from the data source to the data destination including one or more of speed, bandwidth, cost and compliance.

8. The method of claim 1 wherein the one or more analytics operations performed by a given one of the processing nodes include at least one of:

a user-defined function; and

a function selected from a catalog of available functions.

9. The method of claim 1 wherein updating the data based on results of the one or more analytics operations comprises at least one of:

combining the data with data from one or more other data paths;

replacing at least a portion of the data with data from one or more other data paths; and

modifying at least a portion of the data based at least in part on the data from one or more other data paths.

10. The method of claim 1 further comprising providing intermediate analytics results based at least in part on intermediate analytics operations performed at respective intermediate ones of the processing nodes that are part of the first data path between the data source and the data destination.

11. The method of claim 1 wherein a given analytics operation performed at a particular one of the processing nodes utilizes local information available to that processing node within a corresponding local data zone.

12. The method of claim 1 wherein the method is performed at least in part in a worldwide data node.

13. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to receive data from a first data source at a first one of a plurality of distributed processing nodes of an analytics platform;

to perform one or more analytics operations on the data at the first processing node;

to update the data at the first processing node based on results of the one or more analytics operations;

to transmit the updated data to another one of the processing nodes along a first data path between the first data source and a first data destination; and

to repeat the performing, updating and transmitting for the other processing node and for one or more additional distributed processing nodes of the analytics platform along the first data path;

the analytics platform thereby performing distributed analytics processing on the data over multiple ones of the distributed processing nodes as the data moves through the first data path from the first data source to the first data destination;

wherein each of at least a subset of the distributed processing nodes is a part of multiple distinct data paths and performs distinct analytics operations for different ones of the data paths such that a role of a given one of the processing nodes in the analytics platform varies based at least in part on the particular data path over which data is received for processing;

wherein the first processing node is part of the subset of the distributed processing nodes and is a part of at least two distinct data paths including the first data path between the first data source and the first data destination and at least a second data path between a second data source and a second data destination;

wherein the first processing node performs a first type of analytics operation for a first role when receiving data on the first data path;

wherein the first processing node performs a second type of analytics operation for a second role when receiving data on the second data path;

wherein the first and second roles comprise respective distinct sets of one or more analytics-related processing tasks;

wherein the first role and the second role of the first processing node are defined by a distributed service of a software-defined function of the analytics platform; and

wherein updating the data based on results of the one or more analytics operations comprises two or more of;

adding metadata to the data;

modifying existing metadata of the data; and

transforming the data into a different format.

14. The computer program product of claim 13 wherein the plurality of distributed processing nodes are arranged in multiple distinct abstraction levels with each such level implementing a different type of analytics functionality.

15. The computer program product of claim 13 wherein a given analytics operation performed at a particular one of the processing nodes utilizes local information available to that processing node within a corresponding local data zone.

16. An apparatus comprising:

a first processing node of an analytics platform;

the first processing node being one of a plurality of distributed processing nodes of the analytics platform and being configured for communication with other ones of the distributed processing nodes over one or more networks;

the first processing node being further configured;

to receive data from a first data source;

to perform one or more analytics operations on the data;

to update the data based on results of the one or more analytics operations; and

to transmit the updated data to another one of the processing nodes along a first data path between the first data source and a first data destination;

wherein the performing, updating and transmitting are repeated for the other processing node and for one or more additional distributed processing nodes of the analytics platform along the first data path;

the analytics platform thereby performing distributed analytics processing on the data over multiple ones of the distributed processing nodes as the data moves through the first data path from the first data source to the first data destination;

wherein each of at least a subset of the distributed processing nodes is a part of multiple distinct data paths and performs distinct analytics operations for different ones of the data paths such that a role of a given one of the processing nodes in the analytics platform varies based at least in part on the particular data path over which data is received for processing;

wherein the first processing node is part of the subset of the distributed processing nodes and is a part of at least two distinct data paths including the first data path between the first data source and the first data destination and at least a second data path between a second data source and a second data destination;

wherein the first processing node performs a first type of analytics operation for a first role when receiving data on the first data path;

wherein the first processing node performs a second type of analytics operation for a second role when receiving data on the second data path;

wherein the first and second roles comprise respective distinct sets of one or more analytics-related processing tasks;

wherein the first role and the second role of the first processing node are defined by a distributed service of a software-defined function of the analytics platform; and

wherein updating the data based on results of the one or more analytics operations comprises two or more of;

adding metadata to the data;

modifying existing metadata of the data; and

transforming the data into a different format.

17. An information processing system comprising the apparatus of claim 16.

18. The apparatus of claim 16 wherein a given analytics operation performed at a particular one of the processing nodes utilizes local information available to that processing node within a corresponding local data zone.

19. The apparatus of claim 16 wherein the plurality of distributed processing nodes are arranged in multiple distinct abstraction levels with each such level implementing a different type of analytics functionality.