Data analytics platform over parallel databases and distributed file systems

• US 9,858,315 B2
• Filed: 12/22/2016
• Issued: 01/02/2018
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Welton, Caleb E., Yang, Shengwen
• Primary Examiner(s): Bui, Tiffany
• Application Number:
  US15/389,321
• Publication Number:
  US 20170169074A1
• Time in Patent Office: 376 Days
• Field of Search: 707827
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24524 Access plan code generation and invalidation; Reuse of access plans
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Performing data analytics processing in the context of a large scale distributed system that includes a massively parallel processing (MPP) database and a distributed storage layer is disclosed. In various embodiments, a data analytics request is received. A plan is created to generate a response to the request. A corresponding portion of the plan is assigned to each of a plurality of distributed processing segments, including by invoking as indicated in the assignment one or more data analytical functions embedded in the processing segment.

Claims

1. A method, comprising:

embedding in each of a plurality of distributed processing segments a library or other shared object comprising one or more data analytical functions, wherein the library or other shared object is included in the processing segments as deployed;

receiving, by a master node, a data analysis request;

creating, by the master node, a plan to generate a response to the request;

selecting one or more of the plurality of distributed processing segments to process a corresponding portion of the plan, wherein selecting the one or more of the plurality of distributed processing segments to process the corresponding portion of the plan includes;

obtaining, by the master node, metadata associated with one or more portions of the plan to be performed by the one or more corresponding distributed processing segments of the plurality of distributed processing segments; and

embedding the metadata in an assignment communication to be sent to the one or more of the plurality of distributed processing segments, wherein the metadata indicates a location, within a distributed data storage layer, of data to be processed by the corresponding one or more distributed processing segments;

sending, by the master node, to each of the plurality of distributed processing segments for which a portion of the plan is to be processed, the corresponding portion of the plan to be processed by that corresponding segment and the metadata, wherein the metadata is used to locate or access a subset of data on which the segment is to perform an indicated processing, and wherein the metadata sent to each of the plurality of distributed processing segments for which the corresponding portion of the plan is assigned includes an identification of at least one data analytics function to be used to process the portion of the plan of the one or more data analytics functions that is embedded in each of the plurality of distributed processing segments;

receiving, from each of the plurality of distributed processing segments for which a portion of the plan is assigned, a corresponding result of processing the portion of the plan; and

generating, a master response to the data analysis request based at least in part on the corresponding result of processing the portion of the plan received from each of the plurality of distributed processing segments for which a portion of the plan is assigned.

2. The method of claim 1, wherein the plurality of distributed processing segments correspondingly invoke one or more data analytical functions respectively embedded in the corresponding plurality of distributed processing segments.

3. The method of claim 2, wherein the one or more data analytical functions that are invoked are included in an assignment of the plan to the plurality of distributed processing segments by the master node.

4. The method of claim 1, wherein the metadata identifies a location data corresponding to the one or more portions of the plan and at least a part of one or more data analytic processing to be performed in connection with processing the corresponding one or more portions of the plan.

5. The method of claim 1, wherein the data analysis request comprises one or more SQL statements.

6. The method of claim 1, wherein the data analysis request comprises one or more SQL statements to compute one or more of the following:

Logistic Regression, Multinomial Logistic Regression, K-means clustering, Association Rules based market basket analysis, and Latent Dirichlet based topic modeling.

7. The method of claim 1, wherein the data analysis request is received at a master node of a large scale distributed system.

8. The method of claim 1, wherein creating a plan to generate a response to the request includes creating a query plan, slicing the query plan into a plurality of slices, and identifying for each slice a group of processing segments to perform tasks comprising that slice of the query plan.

9. The method of claim 1, wherein each of said distributed processing segments is configured to use the metadata to access said data to be processed by that segment.

10. The method of claim 1, wherein the distributed data storage layer comprises data stored in an instance of a Hadoop Distributed File System (HDFS) and the metadata indicates a location within the HDFS of data to be processed by that segment.

11. The method of claim 1, wherein the library or other shared object embodies the one or more data analytical functions in the form of one or more of the following:

compiled C++ code, compiled Java, compiled Fortran, or other compiled code.

12. The method of claim 1, wherein the plurality of distributed processing segments comprise a subset of parallel processing segments comprising a massively parallel processing (MPP) database system.

13. The method of claim 1, wherein the embedded data analytics function includes a User-Defined function, a step function, or a final function of a User-Defined Aggregator.

14. The method of claim 1, wherein the metadata sent to each of the plurality of distributed processing segments is sent in conjunction with the corresponding portion of the plan to be performed by that segment.

15. The method of claim 14, wherein the metadata sent to each of the plurality of distributed processing segments is sent as part of the corresponding portion of the plan to be performed by that segment.

16. The method of claim 1, wherein the metadata sent to each of the plurality of distributed processing segments for which a portion of the plan is to be processed is obtained from a central metadata store.

17. A system, comprising:

a communication interface; and

one or more processors coupled to the communication interface and configured to;

embed in each of a plurality of distributed processing segments a library or other shared object comprising one or more data analytical functions, wherein the library or other shared object is included in the processing segments as deployed;

receive a data analysis request;

create a plan to generate a response to the request;

select one or more of the plurality of distributed processing segments to process a corresponding portion of the plan, wherein to select the one or more of the plurality of distributed processing segments to process the corresponding portion of the plan includes;

obtain metadata associated with one or more portions of the plan to be performed by the one or more corresponding distributed processing segments of the plurality of distributed processing segments; and

embed the metadata in an assignment communication to be sent to the one or more of the plurality of distributed processing segments, wherein the metadata indicates a location, within a distributed data storage layer, of data to be processed by the corresponding one or more distributed processing segments;

send, to each of the plurality of distributed processing segments for which a portion of the plan is to be processed, the corresponding portion of the plan to be processed by that corresponding segment and the metadata, wherein the metadata is used to locate or access a subset of data on which the segment is to perform an indicated processing, and wherein the metadata sent to each of the plurality of distributed processing segments for which the corresponding portion of the plan is assigned includes an identification of at least one data analytics function to be used to process the portion of the plan of the one or more data analytics functions that is embedded in each of the plurality of distributed processing segments;

receive, from each of the plurality of distributed processing segments for which a portion of the plan is assigned, a corresponding result of processing the portion of the plan; and

generate, a master response to the data analysis request based at least in part on the corresponding result of processing the portion of the plan received from each of the plurality of distributed processing segments for which a portion of the plan is assigned.

18. A computer program product embodied in a tangible, non-transitory computer readable storage medium, comprising computer instructions for:

embedding in each of a plurality of distributed processing segments a library or other shared object comprising one or more data analytical functions, wherein the library or other shared object is included in the processing segments as deployed;

receiving a data analysis request;

creating a plan to generate a response to the request;

selecting one or more of a plurality of distributed processing segments to process a corresponding portion of the plan, wherein selecting the one or more of the plurality of distributed processing segments to process the corresponding portion of the plan includes

obtaining metadata associated with one or more portions of the plan to be performed by the one or more corresponding distributed processing segments; and

embedding the metadata in an assignment communication to be sent to the one or more of the plurality of distributed processing segments, wherein the metadata indicates a location, within a distributed data storage layer, of data to be processed by the corresponding one or more distributed processing segments;

sending, to each of the plurality of distributed processing segments for which a portion of the plan is to be processed, the corresponding portion of the plan to be processed by that corresponding segment and metadata used to locate or access a subset of data on which the segment is to perform an indicated processing, and wherein the metadata sent to each of the plurality of distributed processing segments for which the corresponding portion of the plan is assigned includes an identification of at least one data analytics function to be used to process the portion of the plan of the one or more data analytics functions that is embedded in each of the plurality of distributed processing segments;

receiving, from each of the plurality of distributed processing segments for which a portion of the plan is assigned, a corresponding result of processing the portion of the plan; and

generating, a master response to the data analysis request based at least in part on the corresponding result of processing the portion of the plan received from each of the plurality of distributed processing segments for which a portion of the plan is assigned.