Data pipeline architecture for analytics processing stack

• US 10,698,625 B2
• Filed: 05/07/2018
• Issued: 06/30/2020
• Est. Priority Date: 05/15/2017
• Status: Active Grant

• Current Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Original Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Inventors: Das, Jagaran, Yelisetty, Srinivas, Tung, Teresa Sheausan
• Primary Examiner(s): Dao, Tuan C
• Assistant Examiner(s): Wood, William C
• Application Number:
  US15/973,200
• Publication Number:
  US 20180329644A1
• Time in Patent Office: 785 Days
• Field of Search: None
• CPC Class Codes: G06F 13/382 using universal interface adapter
• CPC Class Codes: G06F 13/4068 Electrical coupling
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/3869 Implementation aspects, e.g. pipeline latches; pipeline synchronisation and clocking
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/547 Remote procedure calls [RPC]; Web services

Abstract

A data pipeline architecture is integrated with an analytics processing stack. The data pipeline architecture may receive incoming data streams from multiple diverse endpoint systems. The data pipeline architecture may include converter interface circuitry with multiple dynamic converters configured to convert the diverse incoming data stream into one or more interchange formats for processing by the analytics processing stack. The analytics processing stack may include multiple layers with insight processing layer circuitry above analysis layer circuitry. The analysis layer circuitry may control analytics models and rule application. The insight processing layer circuitry may monitor output from the analysis layer circuitry and generate insight adjustments responsive to rule changes and analytics model parameter changes produced at the analysis layer circuitry.

Claims

1. A system comprising:

a data stream processing pipeline comprising sequential multiple input processing stages, including;

an ingestion stage comprising multiple ingestion processors configured to receive input streams from multiple different input sources;

an integration stage comprising converter interface circuitry configured to;

perform a classification on the input streams; and

responsive to the classification, assign the input streams to one or more dynamic converters of the converter interface circuitry configured to output data from multiple different stream types into a stream data in a predefined interchange format, the one or more dynamic converters comprising;

a structured data dynamic converter configured to handle input streams in a predetermined predictor data format;

a natural language dynamic converter configured to handle natural language input streams; and

a media dynamic converter configured to handle media input stream by;

selecting among speech-to-text processing for audio streams and computer vision processing for video streams; and

causing execution of selected processing for the media input streams, when present, to generate the output data; and

a storage stage comprising a memory hierarchy, the memory hierarchy configured to store the stream data in the pre-defined interchange format; and

an analytics processing stack coupled to the data stream processing pipeline, the analytics processing stack configured to;

access the stream data, via a hardware memory resource provided by storage layer circuitry of the analytics processing stack;

process the stream data at processing engine layer circuitry of the analytics processing stack to determine whether to provide the stream data to analytics model logic of analysis layer circuitry of the analytic processing stack, rule logic of the analysis layer circuitry of the analytic processing stack, or both;

when the stream data is passed to the analytics model logic;

determine a model change to a model parameter for a predictive data model for the stream data; and

pass a model change indicator of the model change to the storage layer circuitry for storage within the memory hierarchy;

when the stream data is passed to the rule logic;

determine a rule change for a rule governing response to content of the stream data; and

pass a rule change indicator of the rule change to the storage layer circuitry for storage within the memory hierarchy;

at insight processing layer circuitry above the analysis layer circuitry within the analytics processing stack;

access the model change indicator, the rule change indicator, the stream data, or any combination thereof via the hardware memory resource provided by the storage layer circuitry;

determine an insight adjustment responsive to the model change, the rule change, the stream data, or any combination thereof;

generate an insight adjustment indicator responsive to the insight adjustment; and

pass the insight adjustment indicator to the storage layer circuitry for storage within the memory hierarchy.

2. The system of claim 1, where the predetermined predictor data format comprises a predictive model markup language (PMML) format, a portable format for Analytics (PFA), or both.

3. The system of claim 1, where the analytics processing stack further includes configuration layer circuitry configured to accept a configuration input.

4. The system of claim 3, where the configuration input is implemented via an application programming interface (API) configured to provide access to the analytics model logic, the rule logic, the insight processing layer circuitry, or any combination thereof.

5. The system of claim 4, where the API is configured to support parallel management of multiple instances of the analytics model logic, the rule logic, or both.

6. The system of claim 4, where the API is configured to accept a scripted configuration file comprising a configuration setting for the analytics model logic, the rule logic, the insight processing layer circuitry, or any combination thereof.

7. The system of claim 1, where the analytics processing stack further comprises presentation layer circuitry configured to expose the model change indicator, the rule change indicator, the insight adjustment indicator, or any combination thereof to a representational state transfer (REST) server.

8. The system of claim 1, where:

the analytic processing stack further includes presentation layer circuitry; and

the insight processing layer circuitry is further configured to;

generate a notification responsive to the insight adjustment indicator; and

send the notification to the presentation layer circuitry to indicate status for the model parameter, the rule, or both.

9. A method comprising:

in a data stream processing pipeline comprising sequential multiple input processing stages;

receiving, at an ingestion stage of a data stream processing pipeline, input streams from multiple different input sources, the ingestion stage comprising multiple ingestion processors;

at an integration stage comprising converter interface circuitry;

performing a classification on the input streams; and

responsive to the classification, assigning the input streams to one or more dynamic converters configured to output data from multiple different stream types into a stream data in a pre-defined interchange format, the one or more dynamic converters comprising;

a structured data dynamic converter configured to handle input streams in a predetermined predictor data format;

a natural language dynamic converter configured to handle natural language input streams; and

a media dynamic converter configured to handle media input stream by;

selecting among speech-to-text processing for audio streams and computer vision processing for video streams; and

causing execution of selected processing for the media input streams, when present, to generate the output data; and

at a storage stage comprising a memory hierarchy, storing the stream data in the pre-defined interchange format; and

in an analytics processing stack coupled to the data stream processing pipeline;

accessing the stream data, via a hardware memory resource provided by storage layer circuitry of the analytics processing stack;

processing the stream data at processing engine layer circuitry of the analytics processing stack to determine whether to provide the stream data to analytics model logic of analysis layer circuitry of the analytic processing stack, rule logic of the analysis layer circuitry of the analytic processing stack, or both;

when the stream data is passed to the analytics model logic;

determining a model change to a model parameter for a predictive data model for the stream data; and

passing a model change indicator of the model change to the storage layer circuitry for storage within the memory hierarchy;

when the stream data is passed to the rule logic;

determining a rule change for a rule governing response to content of the stream data; and

passing a rule change indicator of the rule change to the storage layer circuitry for storage within the memory hierarchy;

at insight processing layer circuitry above the analysis layer circuitry within the analytics processing stack;

accessing the model change indicator, the rule change indicator, the stream data, or any combination thereof via the hardware memory resource provided by the storage layer circuitry;

determining an insight adjustment responsive to the model change, the rule change, the stream data, or any combination thereof;

generating an insight adjustment indicator responsive to the insight adjustment; and

passing the insight adjustment indicator to the storage layer circuitry for storage within the memory hierarchy.

10. The method of claim 9, further comprising accepting, at configuration layer circuitry of the analytics processing stack, a configuration input implemented via an application programming interface (API) configured to provide access to the analytics model logic, the rule logic, the insight processing layer circuitry, or any combination thereof.

11. The method of claim 9, where the predetermined predictor data format comprises a predictive model markup language (PMML) format, a portable format for Analytics (PFA), or both.

12. The method of claim 9, further comprising exposing, at presentation layer circuitry of the analytics processing stack, the model change indicator, the rule change indicator, the insight adjustment indicator, or any combination thereof to a representational state transfer (REST) server.

13. The method of claim 9, where the method further comprises:

at the insight processing layer circuitry;

generating a notification responsive to the insight adjustment indicator; and

sending the notification to presentation layer circuitry of the analytics processing stack to indicate status for the model parameter, the rule, or both.

14. A product comprising:

a machine readable medium other than a transitory signal; and

instructions stored on the machine readable medium, the instructions configured to, when executed, cause a processor to;

in a data stream processing pipeline comprising sequential multiple input processing stages;

receive, at an ingestion stage of a data stream processing pipeline, input streams from multiple different input sources, the ingestion stage comprising multiple ingestion processors;

at an integration stage comprising converter interface circuitry;

perform a classification on the input streams; and

responsive to the classification, assign the input streams to one or more dynamic converters configured to output data from multiple different stream types into a stream data in a pre-defined interchange format, the one or more dynamic converters comprising;

a structured data dynamic converter configured to handle input streams in a predetermined predictor data format;

a natural language dynamic converter configured to handle natural language input streams; and

a media dynamic converter configured to handle media input streams by;

selecting among speech-to-text processing for audio streams and computer vision processing for video streams; and

causing execution of selected processing for the media input streams, when present, to generate the output data; and

at a storage stage comprising a memory hierarchy, store the stream data in the pre-defined interchange format; and

in an analytics processing stack coupled to the data stream processing pipeline;

access the stream data, via a hardware memory resource provided by storage layer circuitry of the analytics processing stack;

process the stream data at processing engine layer circuitry of the analytics process stack to determine whether to provide the stream data to analytics model logic of analysis layer circuitry of the analytic processing stack, rule logic of the analysis layer circuitry of the analytic processing stack, or both;

when the stream data is passed to the analytics model logic;

determine a model change to a model parameter for a predictive data model for the stream data; and

pass a model change indicator of the model change to the storage layer circuitry for storage within the memory hierarchy;

when the stream data is passed to the rule logic;

determine a rule change for a rule governing response to content of the stream data; and

pass a rule change indicator of the rule change to the storage layer circuitry for storage within the memory hierarchy;

at insight processing layer circuitry above the analysis layer circuitry within the analytics processing stack;

access the model change indicator, the rule change indicator, the stream data, or any combination thereof via the hardware memory resource provided by the storage layer circuitry;

determine an insight adjustment responsive to the model change, the rule change, the stream data, or any combination thereof;

generate an insight adjustment indicator responsive to the insight adjustment; and

pass the insight adjustment indicator to the storage layer circuitry for storage within the memory hierarchy.

15. The product of claim 14, where the instructions are further configured to cause the processor to expose, at presentation layer circuitry of the analytics processing stack, the model change indicator, the rule change indicator, the insight adjustment indicator, or any combination thereof to a representational state transfer (REST) server.

16. The product of claim 14, where the instructions are further configured to cause the processor to:

at the insight processing layer circuitry;

generate a notification responsive to the insight adjustment indicator; and

send the notification to presentation layer circuitry of the analytics processing stack to indicate status for the model parameter, the rule, or both.

17. The product of claim 14, where the instructions are further configured to cause the processor to accept, configuration layer circuitry of the analytics processing stack, a configuration input implemented via an application programming interface (API) configured to provide access to the analytics model logic, the rule logic, the insight processing layer circuitry, or any combination thereof.

18. The product of claim 14, where the predetermined predictor data format comprises a predictive model markup language (PMML) format, a portable format for Analytics (PFA), or both.

19. The product of claim 14, where the instructions are further configured to cause the processor to expose, at presentation layer circuitry of the analytics processing stack, the model change indicator, the rule change indicator, the insight adjustment indicator, or any combination thereof to a representational state transfer (REST) server.

20. The product of claim 14, where the instructions are further configured to cause the processor to, at the insight processing layer circuitry:

generate a notification responsive to the insight adjustment indicator; and

send the notification to presentation layer circuitry of the analytics processing stack to indicate status for the model parameter, the rule, or both.