System and method for inferencing of data transformations through pattern decomposition

• US 10,705,812 B2
• Filed: 08/22/2017
• Issued: 07/07/2020
• Est. Priority Date: 08/22/2016
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Seetharaman, Ganesh, Stojanovic, Alexander Sasha, Namarvar, Hassan Heidari, Allan, David
• Primary Examiner(s): Rivera, Anibal
• Application Number:
  US15/683,556
• Publication Number:
  US 20180067732A1
• Time in Patent Office: 1,050 Days
• CPC Class Codes: G06F 16/144 Query formulation
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 8/10 Requirements analysis; Specification techniques
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/433 Dependency analysis; Data or control flow analysis
• CPC Class Codes: G06F 8/4452 Software pipelining
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06Q 10/0637 : Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals

Abstract

In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can provide a service to recommend actions and transformations, on an input data, based on patterns identified from the functional decomposition of a data flow for a software application, including determining possible transformations of the data flow in subsequent applications. Data flows can be decomposed into a model describing transformations of data, predicates, and business rules applied to the data, and attributes used in the data flows.

Claims

1. A method for use with a data integration or other computing environment comprising:

providing, at a computer including a processor, a design-time system for creating software applications that perform data processing, wherein the design-time system includes;

a software development component having a graphical user interface for creation of data flows associated with the software applications, including specification of input hubs and output hubs comprising datasets that are data structures having attributes and associated with one or more of the hubs; and

a system hub that stores metadata associated with processing the data flows associated with the software applications, including functional and business data types;

wherein the software applications are deployed to a run-time system that executes the software applications, and that receives input from the design-time system;

accessing a data flow for each of one or more software applications that;

receive input data from one or more input hub sources of data, and

publish output data to one or more output hub destinations, according to the data flow associated with the one or more software applications;

processing the data flow for a first software application of the one or more software applications to generate one or more functional expressions representing the data flow for the first software application, wherein the one or more functional expressions are generated based on a determination of one or more semantic actions or rules identified in the data flow, including wherein;

an application represents a top level data flow transformation; and

an action represents an operator on one or more datasets;

identifying a pattern of transformation in the data flow for the first software application, as determined by the one or more functional expressions that are generated as representing the data flow for the first software application; and

subsequent to identifying the pattern of transformation identified in the data flow for the first software application, providing, as an output, a recommendation of one or more data transformations for incorporation within at least one of a modified data flow for the first software application or a data flow of a second software application;

wherein the pattern is used in displaying, at a graphical user interface, selected ones of the semantic actions enabled for the accessed data, for selection and use with the accessed data, including automatically providing or updating a list of the selected ones of the semantic actions enabled for the accessed data, during the processing of the accessed data.

2. The method of claim 1, wherein the application is a dataflow application.

3. The method of claim 1, wherein an application represents a top level data flow transformation;

an action represents an operator on one or more datasets; and

actions reference a base semantic action or function declaratively defined in the system, and can have one or more action parameters.

4. The method of claim 3, wherein action parameters are owned by actions and have specific functional or business types, and represent a specific upstream dataset to be processed.

5. The method of claim 3, wherein a scope resolver is used to resolve the reference to a particular attribute or embedded object.

6. The method of claim 1, wherein the method is performed in a cloud or cloud-based computing environment.

7. A system for providing recommendations of actions and transformations on an input data for use with a data integration or other computing environment, comprising:

a design-time system for creating software applications that perform data processing, wherein the design-time system includes;

a software development component having a graphical user interface for creation of data flows associated with the software applications, including specification of input hubs and output hubs comprising datasets that are data structures having attributes and associated with one or more of the hubs; and

a system hub that stores metadata associated with processing the data flows associated with the software applications, including functional and business data types;

wherein the software applications are deployed to a run-time system that executes the software applications, and that receives input from the design-time system;

one or more processors operable to;

access a data flow for each of one or more software applications that;

receive input data from one or more input hub sources of data, and

publish output data to one or more output hub destinations, according to the data flow associated with the one or more software applications;

process the data flow for a first software application of the one or more software applications to generate one or more functional expressions representing the data flow for the first software application, wherein the one or more functional expressions are generated based on a determination of one or more semantic actions or business rules identified in the data flow, including wherein;

an application represents a top level data flow transformation; and

an action represents an operator on one or more datasets;

identify a pattern of transformation in the data flow for the first software application, as determined by the one or more functional expressions that are generated as representing the data flow for the first software application; and

subsequent to identifying the pattern of transformation identified in the data flow for the first software application, provide, as an output, a recommendation of one or more data transformations for incorporation within at least one of a modified data flow for the first software application or a data flow of a second software application;

wherein the pattern is used in displaying, at a graphical user interface, selected ones of the semantic actions enabled for the accessed data, for selection and use with the accessed data, including automatically providing or updating a list of the selected ones of the semantic actions enabled for the accessed data, during the processing of the accessed data.

8. The system of claim 7, wherein the application is a dataflow application.

9. The system of claim 7, wherein an application represents a top level data flow transformation;

an action represents an operator on one or more datasets; and

actions reference a base semantic action or function declaratively defined in the system, and can have one or more action parameters.

10. The system of claim 9, wherein action parameters are owned by actions and have specific functional or business types, and represent a specific upstream dataset to be processed.

11. The system of claim 9, wherein a scope resolver is used to resolve the reference to a particular attribute or embedded object.

12. The system of claim 7, wherein the system is provided in a cloud or cloud-based computing environment.

13. A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers cause the one or more computers to perform a method comprising:

providing a design-time system for creating software applications that perform data processing, wherein the design-time system includes;

a software development component having a graphical user interface for creation of data flows associated with the software applications, including specification of input hubs and output hubs comprising datasets that are data structures having attributes and associated with one or more of the hubs; and

a system hub that stores metadata associated with processing the data flows associated with the software applications, including functional and business data types;

wherein the software applications are deployed to a run-time system that executes the software applications, and that receives input from the design-time system;

accessing a data flow for each of one or more software applications that;

receive input data from one or more input hub sources of data, and

publish output data to one or more output hub destinations,

according to the data flow associated with the one or more software applications;

processing the data flow for a first software application of the one or more software applications to generate one or more functional expressions representing the data flow for the first software application, wherein the one or more functional expressions are generated based on a determination of one or more semantic actions or rules identified in the data flow, including wherein;

an application represents a top level data flow transformation; and

an action represents an operator on one or more datasets;

identifying a pattern of transformation in the data flow for the first software application, as determined by the one or more functional expressions that are generated as representing the data flow for the first software application; and

subsequent to identifying the pattern of transformation identified in the data flow for the first software application, providing, as an output, a recommendation of one or more data transformations for incorporation within at least one of a modified data flow for the first software application or a data flow of a second software application;

wherein the pattern is used in displaying, at a graphical user interface, selected ones of the semantic actions enabled for the accessed data, for selection and use with the accessed data, including automatically providing or updating a list of the selected ones of the semantic actions enabled for the accessed data, during the processing of the accessed data.

14. The non-transitory computer readable storage medium of claim 13, wherein the application is a dataflow application.

15. The non-transitory computer readable storage medium of claim 13, wherein an application represents a top level data flow transformation;

an action represents an operator on one or more datasets; and

actions reference a base semantic action or function declaratively defined in the system, and can have one or more action parameters.

16. The non-transitory computer readable storage medium of claim 15, wherein action parameters are owned by actions and have specific functional or business types, and represent a specific upstream dataset to be processed.

17. The non-transitory computer readable storage medium of claim 15, wherein a scope resolver is used to resolve the reference to a particular attribute or embedded object.