DATA ANALYTICS PLATFORM OVER PARALLEL DATABASES AND DISTRIBUTED FILE SYSTEMS

• US 20140244701A1
• Filed: 03/15/2013
• Published: 08/28/2014
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Welton, Caleb E., Yang, Shengwen
• Granted Patent: US 9,563,648 B2
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24524 Access plan code generation and invalidation; Reuse of access plans
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H05K 999/99 dummy group

Abstract

Performing data analytics processing in the context of a large scale distributed system that includes a massively parallel processing (MPP) database and a distributed storage layer is disclosed. In various embodiments, a data analytics request is received. A plan is created to generate a response to the request. A corresponding portion of the plan is assigned to each of a plurality of distributed processing segments, including by invoking as indicated in the assignment one or more data analytical functions embedded in the processing segment.

Claims

1. A method, comprising:

receiving a data analysis request;

creating a plan to generate a response to the request; and

assigning to each of a plurality of distributed processing segments a corresponding portion of the plan to be performed by that segment, including by invoking as indicated in the assignment one or more data analytical functions embedded in the processing segment.

2. The method of claim 1, wherein the data analysis request comprises one or more SQL statements.

3. The method of claim 1, wherein the data analysis request comprises one or more SQL statements to compute one or more of the following:

Logistic Regression, Multinomial Logistic Regression, K-means clustering, Association Rules based market basket analysis, and Latent Dirichlet based topic modeling.

4. The method of claim 1, wherein the data analysis request is received at a master node of a large scale distributed system.

5. The method of claim 1, wherein creating a plan to generate a response to the request includes creating a query plan, slicing the query plan into a plurality of slices, and identifying for each slice a group of processing segments to perform tasks comprising that slice of the query plan.

6. The method of claim 1, wherein assigning to each of a plurality of distributed processing segments a corresponding portion of the plan to be performed by that segment includes embedding in an assignment communication to be sent to one or more of said plurality of distributed processing segments a metadata indicating a location, within a distributed data storage layer, of data to be processed by that segment.

7. The method of claim 6, wherein each of said distributed processing segments is configured to use the metadata to access said data to be processed by that segment.

8. The method of claim 6, wherein the distributed data storage layer comprises data stored in an instance of the Hadoop Distributed File System (HDFS) and the metadata indicates a location within the HDFS of data to be processed by that segment.

9. The method of claim 1, further comprising embedding in each of said plurality of distributed processing segments a library or other shared object comprising said one or more data analytical functions.

10. The method of claim 9, wherein said library or other shared object is included in the processing segments as deployed.

11. The method of claim 9, wherein said library or other shared object embodies said one or more data analytical functions in the form of one or more of the following:

compiled C++ code, compiled Java, compiled Fortran, or other compiled code.

12. The method of claim 1, wherein the plurality of distributed processing segments comprise a subset of parallel processing segments comprising a massively parallel processing (MPP) database system.

13. A system, comprising:

a communication interface; and

a processor coupled to the communication interface and configured to;

receive a data analysis request;

create a plan to generate a response to the request; and

assign to each of a plurality of distributed processing segments, via a communication sent via the communication interface, a corresponding portion of the plan to be performed by that segment, including by invoking as indicated in the assignment one or more data analytical functions embedded in the processing segment.

14. The system of claim 13, wherein the data analysis request comprises one or more SQL statements.

15. The system of claim 13, wherein the data analysis request is received at a master node of a large scale distributed system.

16. The system of claim 13, wherein the processor is configured to create the plan to generate a response to the request at least in part by creating a query plan, slicing the query plan into a plurality of slices, and identifying for each slice a group of processing segments to perform tasks comprising that slice of the query plan.

17. The system of claim 13, wherein the processor is configured to assign to each of a plurality of distributed processing segments a corresponding portion of the plan to be performed by that segment at least in part by embedding in the communication to be sent via the communication interface to one or more of said plurality of distributed processing segments a metadata indicating a location, within a distributed data storage layer, of data to be processed by that segment.

18. The system of claim 17, wherein each of said distributed processing segments is configured to use the metadata to access said data to be processed by that segment.

19. The system of claim 13, wherein each of said distributed processing segments has embedded therein a library or other shared object comprising said one or more data analytical functions.

20. The system of claim 13, wherein the plurality of distributed processing segments is comprise a subset of parallel processing segments comprising a massively parallel processing (MPP) database system.

21. A computer program product embodied in a tangible, non-transitory computer readable storage medium, comprising computer instructions for:

receiving a data analysis request;

creating a plan to generate a response to the request; and

assigning to each of a plurality of distributed processing segments a corresponding portion of the plan to be performed by that segment, including by invoking as indicated in the assignment one or more data analytical functions embedded in the processing segment.

22. The computer program product of claim 21, wherein assigning to each of a plurality of distributed processing segments a corresponding portion of the plan to be performed by that segment includes embedding in an assignment communication to be sent to one or more of said plurality of distributed processing segments a metadata indicating a location, within a distributed data storage layer, of data to be processed by that segment.