Distributed Multiple Tier Multi-Node Serverless Framework for Complex Analytics Task Execution

• US 20190317808A1
• Filed: 10/12/2018
• Published: 10/17/2019
• Est. Priority Date: 04/11/2018
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Inventors: Srinivasan, Madhan Kumar, Purushothaman, Arun, Achanta, Vijaya Tapaswi
• Granted Patent: US 10,719,360 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 9/4887 involving deadlines, e.g. rate based, periodic
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70

Abstract

A system may support distributed multiple tier multi-node serverless analytics task execution. At a data ingestion tier, data ingestion serverless tasks may receive detail data for analytic processing. data integration serverless tasks, executing at a data integration and consolidation tier and initiated by the data ingestion serverless tasks, may sort the detail data and identify patterns within the detail data to generate grouped pre-processed data. The data integration serverless tasks may initiate partitioning serverless tasks which may divide the grouped pre-processed data into data chunks. Multi-node analytic serverless tasks at an analytic tier, at least some of which being initiated by the partitioning serverless tasks, may analyze the data chunks and generate prescriptive outputs.

Claims

1. A system including:

network interface circuitry configured to;

receive ingestion detail data from multiple data sources; and

sending first and second prescriptive outputs to a storage host server; and

tiered analytic processing circuitry configured to execute multiple tiers of serverless tasks, the multiple tiers including;

a data ingestion tier;

a data integration and consolidation tier;

a partitioning tier; and

an analytic tier;

the tiered analytic processing circuitry configured to;

establish, at the data ingestion tier, a data ingestion serverless task configured to;

access, via the network interface circuitry, the ingestion detail data; and

generate an ingestion inter-tier communication link; and

via the ingestion inter-tier communication link;

initiate a data integration serverless task;

responsive to initiation from the data ingestion serverless task, establish, at the data integration and consolidation tier, the data integration serverless task, the data integration serverless task configured to;

sort the ingestion detail data into an association grouping to generate grouped detail data;

parse the grouped detail data to identify patterns present within the grouped detail data to generate grouped pre-processed data;

generate a data integration inter-tier communication link; and

via the data integration inter-tier communication link;

initiate a partitioning serverless task;

responsive to initiation from the data integration serverless task, establish, at the partitioning tier, the partitioning serverless task, the partitioning serverless task configured to;

identify a partitioning criterion and a concurrency criterion based on a serverless task execution limit and a size of the grouped pre-processed data;

partition the grouped pre-processed data in accord with the partitioning criterion to generate a first data chunk and a second data chunk;

generate a partitioning inter-tier communication link; and

via the partitioning inter-tier communication link;

initiate a first multi-node analytic serverless task;

responsive to initiation from the partitioning serverless task, establish, at the analytic tier, the first multi-node analytic serverless task configured to;

analyze the first data chunk and at least a first portion the patterns present within the grouped detail data to generate the first prescriptive output; and

initiate a second multi-node analytic serverless task, the period based on the concurrency criterion; and

responsive to initiation from the first multi-node analytic serverless task, establish, at the analytic tier, the second multi-node analytic serverless task, the second multi-node analytic serverless task configured to analyze the second data chunk and at least a second portion the patterns present within the grouped detail data to generate the second prescriptive output.

2. The system of claim 1, where:

the multiple tiers further include a data merge tier; and

the tiered analytic processing circuitry is further configured to;

establish, in accord with the concurrency criterion, a final multi-node analytic serverless task configured to;

generate an analytic inter-tier communication link; and

via the analytic inter-tier communication link, initiate a data merge serverless task; and

responsive to initiation from the final multi-node analytic serverless task, establish, at the data merge tier, a data merge serverless task configured to merge the first and second prescriptive outputs.

3. The system of claim 2, where the second multi-node analytic serverless task is the final multi-node analytic serverless task.

4. The system of claim 1, where the data integration serverless task is configured to sort the ingestion detail data into multiple association groupings based on multiple predetermined associations.

5. The system of claim 4, where the storage host server is configured to control multiple databases each defined for a respective one of the multiple association groupings.

6. The system of claim 4, where the multiple association groupings include one or more from among:

a client association, a sensor type association, a geographic association, a tagging association, and a utility association.

7. The system of claim 1, where the data integration serverless task is further configured to:

prior to sorting the ingestion detail data, exclude ingestion detail data outside of a defined analysis window.

8. The system of claim 1, where the serverless task execution limit includes a pre-defined serverless task timeout.

9. The system of claim 1, where the size of the grouped pre-processed data includes a number of entries within the grouped pre-processed data.

10. The system of claim 1, where one or more from among:

the data ingestion serverless task, the data integration serverless task, and the partitioning serverless task include a series of serverless tasks.

11. The system of claim 10, where each serverless task within the series is partially concurrent with another one of the serverless tasks in the series.

12. The system of claim 1, where the partitioning serverless task is configured to determine the partitioning criterion and the concurrency criterion further based on a target processing completion schedule for the analytics tier.

13. The system of claim 12, where the partitioning serverless task is configured to determine the partitioning criterion and the concurrency criterion further based on the target processing completion schedule for the analytics tier by determining a number of multi-node analytic serverless tasks with a capacity to finish processing the grouped pre-processed data prior to a completion target time.

14. The system of claim 1, where the data ingestion serverless task is configured to pass the ingestion detail data to the data integration serverless task by providing the data integration serverless task with a storage location for the ingestion detail data.

15. A method including:

at network interface circuitry;

receiving ingestion detail data from multiple data; and

sending first and second prescriptive outputs to a storage host server; and

at tiered analytic processing circuitry configured to execute multiple tiers of serverless tasks;

establishing, at a data ingestion tier of the multiple tiers, a data ingestion serverless task configured to;

accessing, via execution of the data ingestion serverless task, the ingestion detail data;

generating, via execution of the data ingestion serverless task, an ingestion inter-tier communication link;

via the ingestion inter-tier communication link;

initiating, via execution of the data ingestion serverless task, a data integration serverless task;

responsive to initiation from the data ingestion serverless task, establishing, at a data integration and consolidation tier of the multiple tiers, the data integration serverless task;

sorting, via execution of the data integration serverless task, the ingestion detail data into an association grouping to generate grouped detail data;

parsing, via execution of the data integration serverless task, the grouped detail data to identify patterns present within the grouped detail data to generate grouped pre-processed data;

generating, via execution of the data integration serverless task, a data integration inter-tier communication link;

via the data integration inter-tier communication link;

initiating, via execution of the data integration serverless task, a partitioning serverless task; and

responsive to initiation from the data integration serverless task, establishing, at a partitioning tier of the multiple tiers, the partitioning serverless task;

identifying, via execution of the partitioning serverless task, a partitioning criterion and a concurrency criterion based on a serverless task execution limit and a size of the grouped pre-processed data;

partitioning, via execution of the partitioning serverless task, the grouped pre-processed data in accord with the partitioning criterion to generate a first data chunk and a second data chunk;

generating, via execution of the partitioning serverless task, a partitioning inter-tier communication link;

via the partitioning inter-tier communication link;

initiating, via execution of the partitioning serverless task, a first multi-node analytic serverless task; and

responsive to initiation from the partitioning serverless task, establishing, at an analytic tier of the multiple tiers, the first multi-node analytic serverless task;

analyzing, via execution of the first multi-node analytic serverless task, the first data chunk and at least a first portion the patterns present within the grouped detail data to generate the first prescriptive output;

initiating a second multi-node analytic serverless task, the period based on the concurrency criterion;

responsive to initiation from the first multi-node analytic serverless task, establishing, at the analytic tier, the second multi-node analytic serverless task; and

analyzing, via execution of the second multi-node analytic serverless task, the second data chunk and at least a second portion the patterns present within the grouped detail data to generate the second prescriptive output.

16. The method of claim 15, further including

at the tiered analytic processing circuitry:

establishing, in accord with the concurrency criterion, a final multi-node analytic serverless task;

generating, via execution of the final multi-node analytic serverless task, an analytic inter-tier communication link;

via the analytic inter-tier communication link and via execution of the final multi-node analytic serverless task, initiating a data merge serverless task;

responsive to initiation from the final multi-node analytic serverless task, establishing, at a data merge tier of the multiple tiers, a data merge serverless task; and

merging, via execution of the data merge serverless task, the first and second prescriptive outputs.

17. The method of claim 15, where one or more from among:

the data ingestion serverless task, the data integration serverless task, and the partitioning serverless task include a series of serverless tasks.

18. A product including:

machine-readable media other than a transitory signal; and

instructions stored on the machine-readable media, the instructions configured to, when executed, cause a machine to;

at network interface circuitry;

receive ingestion detail data from multiple data; and

send first and second prescriptive outputs to a storage host server; and

at tiered analytic processing circuitry configured to execute multiple tiers of serverless tasks;

establish, at a data ingestion tier of the multiple tiers, a data ingestion serverless task configured to;

access, via execution of the data ingestion serverless task, the ingestion detail data;

generate, via execution of the data ingestion serverless task, an ingestion inter-tier communication link;

via the ingestion inter-tier communication link;

initiate, via execution of the data ingestion serverless task, a data integration serverless task;

responsive to initiation from the data ingestion serverless task, establish, at a data integration and consolidation tier of the multiple tiers, the data integration serverless task;

sort, via execution of the data integration serverless task, the ingestion detail data into an association grouping to generate grouped detail data;

parse, via execution of the data integration serverless task, the grouped detail data to identify patterns present within the grouped detail data to generate grouped pre-processed data;

generate, via execution of the data integration serverless task, a data integration inter-tier communication link;

via the data integration inter-tier communication link;

initiate, via execution of the data integration serverless task, a partitioning serverless task;

responsive to initiation from the data integration serverless task, establish, at a partitioning tier of the multiple tiers, the partitioning serverless task;

identify, via execution of the partitioning serverless task, a partitioning criterion and a concurrency criterion based on a serverless task execution limit and a size of the grouped pre-processed data;

partition, via execution of the partitioning serverless task, the grouped pre-processed data in accord with the partitioning criterion to generate a first data chunk and a second data chunk;

generate, via execution of the partitioning serverless task, a partitioning inter-tier communication link;

via the partitioning inter-tier communication link;

initiate, via execution of the partitioning serverless task, a first multi-node analytic serverless task;

responsive to initiation from the partitioning serverless task, establish, at an analytic tier of the multiple tiers, the first multi-node analytic serverless task;

analyze, via execution of the first multi-node analytic serverless task, the first data chunk and at least a first portion the patterns present within the grouped detail data to generate the first prescriptive output;

initiate a second multi-node analytic serverless task, the period based on the concurrency criterion;

responsive to initiation from the first multi-node analytic serverless task, establish, at the analytic tier, the second multi-node analytic serverless task; and

analyze, via execution of the second multi-node analytic serverless task, the second data chunk and at least a second portion the patterns present within the grouped detail data to generate the second prescriptive output.

19. The product of claim 18, where the instructions are further configured to cause the machine to determine, via execution of the partitioning serverless task, the partitioning criterion and the concurrency criterion further based on a target processing completion schedule for the analytics tier.

20. The product of claim 19, where the instructions are further configured to cause the machine to determine, via execution of the partitioning serverless task, the partitioning criterion and the concurrency criterion further based on the target processing completion schedule for the analytics tier by determining a number of multi-node analytic serverless tasks with a capacity to finish processing the grouped pre-processed data prior to a completion target date.