Managing Interfaces for Sub-Graphs

• US 20190095176A1
• Filed: 11/27/2018
• Published: 03/28/2019
• Est. Priority Date: 12/05/2013
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Larson, Brond, Bay, Paul, Bromley, H. Mark
• Granted Patent: US 10,901,702 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 8/10 Requirements analysis; Specification techniques
• CPC Class Codes: G06F 8/20 Software design
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 9/4494 data driven

Abstract

Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Claims

1. (canceled)

2. A method for configuring a dataflow graph template that includes one or more subgraph interfaces, the method including:

generating in a development environment a dataflow graph template that includes a dataflow graph that includes one or more subgraph interfaces, with the one or more subgraph interfaces each having one or more flow junctions;

generating in the development environment an implementation subgraph that conforms with a given subgraph interface; and

configuring in the development environment the dataflow graph template by;

dynamically linking the implementation subgraph in place of the given subgraph interface in the dataflow graph; and

determining, for each flow junction of the given subgraph interface, a direction of metadata propagation, where the determined direction is one of an inward transfer of metadata from the flow junction on the given subgraph interface to a component of the dataflow graph or an outward transfer of the metadata from the flow junction on the given subgraph interface to the component of the dataflow graph.

3. The method of claim 2 wherein the one or more flow junctions define a point of connection between a flow in the dataflow graph template and the implementation subgraph.

4. The method of claim 2 wherein the one or more flow junctions represent corresponding one or more connections between a flow of data to or from a port on a component of the dataflow graph and a flow of data to or from a port on a component of the implementation subgraph.

5. The method of claim 2, further including:

receiving datasets from a database or from some other data storage or data queueing system;

wherein the dataflow graph includes dataset components representing the received datasets and data processing components; and

wherein ports of the dataset components and the data processing components and the one or more flow junctions of the one or more sub-graph interfaces are interconnected by flows that define how data propagates among the dataset components, the data processing components, and the one or more sub-graph interfaces.

6. The method of claim 2, wherein the dataflow graph template is read-only, encrypted or protected from being viewed and/or altered.

7. The method of claim 2, wherein the implementation subgraph is read-only, encrypted or protected from being viewed and/or altered.

8. The method of claim 2, wherein the dataflow graph template is configured to accommodate different types of record formats from the implementation subgraph without requiring user intervention.

9. The method of claim 2, wherein a flow junction is an output flow function or an input flow junction, further including:

adding an input flow junction or an output flow junction that are represented as one or more menu items from a click menu in a graphical user interface.

10. The method of claim 2, wherein a flow junction is an output flow function or an input flow junction, further including:

removing an input flow junction or an output flow junction that is represented as one or more menu items from a click menu in a graphical user interface.

11. The method of claim 2, wherein a flow junction is an output flow function or an input flow junction, further including:

adding an input and/or an output flow junction to the one or more sub-graph interfaces.

12. The method of claim 2, wherein a flow junction is an output flow function or an input flow junction, further including:

generating one or more links between input flow junctions and output flow junctions, with a generated link between a given one of the input flow junctions and a given one of the output flow junctions indicative of relationships between metadata associated with the given one of the input flow junctions and metadata associated with the given one of the output flow junctions.

13. The method of claim 2, further including generating a generated link through a user interface.

14. The method of claim 2 further including:

generating a graphical user interface to render a flow junction configuration tab to configure the one or more flow junctions of the one or more sub-graph interfaces, with the flow junction configuration tab including a flow junction list and a propagation control.

15. The method of claim 14 wherein the flow junction list renders a list of flow junctions and the propagation control includes one or more controls to select the direction of metadata propagation as an inward transfer direction or an outward transfer direction.

16. The method of claim 2, further including analyzing at least one of the one or more flow junctions of the one or more subgraph interfaces to determine the direction of metadata propagation.

17. One or more machine-readable hardware storage devices for configuring a dataflow graph template that includes one or more subgraph interfaces, the one or more machine-readable hardware storage devices storing instructions that are executable by one or more processing devices to perform operations including:

generating in a development environment a dataflow graph template that includes a dataflow graph that includes one or more subgraph interfaces, with the one or more subgraph interfaces each having one or more flow junctions;

generating in the development environment an implementation subgraph that conforms with a given subgraph interface; and

configuring in the development environment the dataflow graph template by;

dynamically linking the implementation subgraph in place of the given subgraph interface in the dataflow graph; and

determining, for each flow junction of the given subgraph interface, a direction of metadata propagation, where the determined direction is one of an inward transfer of metadata from the flow junction on the given subgraph interface to a component of the dataflow graph or an outward transfer of the metadata from the flow junction on the given subgraph interface to the component of the dataflow graph.

18. The one or more machine-readable hardware storage devices of claim 17, wherein the one or more flow junctions define a point of connection between a flow in the dataflow graph template and the implementation subgraph.

19. The one or more machine-readable hardware storage devices of claim 17, wherein the one or more flow junctions represent corresponding one or more connections between a flow of data to or from a port on a component of the dataflow graph and a flow of data to or from a port on a component of the implementation subgraph.

20. The one or more machine-readable hardware storage devices of claim 17, wherein the operations further include:

receiving datasets from a database or from some other data storage or data queueing system;

wherein the dataflow graph includes dataset components representing the received datasets and data processing components; and

wherein ports of the dataset components and the data processing components and the one or more flow junctions of the one or more sub-graph interfaces are interconnected by flows that define how data propagates among the dataset components, the data processing components, and the one or more sub-graph interfaces.

21. The one or more machine-readable hardware storage devices of claim 17, wherein the dataflow graph template is read-only, encrypted or protected from being viewed and/or altered.

22. The one or more machine-readable hardware storage devices of claim 17, wherein the implementation subgraph is read-only, encrypted or protected from being viewed and/or altered.

23. The one or more machine-readable hardware storage devices of claim 17, wherein the dataflow graph template is configured to accommodate different types of record formats from the implementation subgraph without requiring user intervention.

24. The one or more machine-readable hardware storage devices of claim 17, wherein a flow junction is an output flow function or an input flow junction, the operations further including:

adding an input flow junction or an output flow junction that are represented as one or more menu items from a click menu in a graphical user interface.

25. The one or more machine-readable hardware storage devices of claim 17, wherein a flow junction is an output flow function or an input flow junction, the operations further including:

removing an input flow junction or an output flow junction that is represented as one or more menu items from a click menu in a graphical user interface.

26. The one or more machine-readable hardware storage devices of claim 17, wherein a flow junction is an output flow function or an input flow junction, the operations further including:

adding an input and/or an output flow junction to the one or more sub-graph interfaces.

27. The one or more machine-readable hardware storage devices of claim 17, wherein a flow junction is an output flow function or an input flow junction, the operations further including:

generating one or more links between input flow junctions and output flow junctions, with a generated link between a given one of the input flow junctions and a given one of the output flow junctions indicative of relationships between metadata associated with the given one of the input flow junctions and metadata associated with the given one of the output flow junctions.

28. The one or more machine-readable hardware storage devices of claim 27, the operations further including generating a generated link through a user interface.

29. The one or more machine-readable hardware storage devices of claim 17, the operations further including:

generating a graphical user interface to render a flow junction configuration tab to configure the one or more flow junctions of the one or more sub-graph interfaces, with the flow junction configuration tab including a flow junction list and a propagation control.

30. The one or more machine-readable hardware storage devices of claim 17, the operations further including wherein the flow junction list renders a list of flow junctions and the propagation control includes one or more controls to select the direction of metadata propagation as an inward transfer direction or an outward transfer direction.

31. The one or more machine-readable hardware storage devices of claim 17, the operations further including analyzing at least one of the one or more flow junctions of the one or more subgraph interfaces to determine the direction of metadata propagation.

32. A system for configuring a dataflow graph template that includes one or more subgraph interfaces, including:

one or more processing devices; and

one or more machine-readable hardware storage devices storing instructions that are executable by the one or more processing devices to perform operations including;

generating in a development environment a dataflow graph template that includes a dataflow graph that includes one or more subgraph interfaces, with the one or more subgraph interfaces each having one or more flow junctions;

generating in the development environment an implementation subgraph that conforms with a given subgraph interface; and

configuring in the development environment the dataflow graph template by;

dynamically linking the implementation subgraph in place of the given subgraph interface in the dataflow graph; and

determining, for each flow junction of the given subgraph interface, a direction of metadata propagation, where the determined direction is one of an inward transfer of metadata from the flow junction on the given subgraph interface to a component of the dataflow graph or an outward transfer of the metadata from the flow junction on the given subgraph interface to the component of the dataflow graph.

33. The system of claim 32, wherein the one or more flow junctions define a point of connection between a flow in the dataflow graph template and the implementation subgraph.

34. The system of claim 32, wherein the one or more flow junctions represent corresponding one or more connections between a flow of data to or from a port on a component of the dataflow graph and a flow of data to or from a port on a component of the implementation subgraph.

35. The system of claim 32, wherein the operations further include:

receiving datasets from a database or from some other data storage or data queueing system;

wherein the dataflow graph includes dataset components representing the received datasets and data processing components; and

wherein ports of the dataset components and the data processing components and the one or more flow junctions of the one or more sub-graph interfaces are interconnected by flows that define how data propagates among the dataset components, the data processing components, and the one or more sub-graph interfaces.

36. The system of claim 32, wherein the dataflow graph template is read-only, encrypted or protected from being viewed and/or altered.

37. The system of claim 32, wherein the implementation subgraph is read-only, encrypted or protected from being viewed and/or altered.

38. The system of claim 32, wherein the dataflow graph template is configured to accommodate different types of record formats from the implementation subgraph without requiring user intervention.

39. The system of claim 32, wherein a flow junction is an output flow function or an input flow junction, the operations further including:

adding an input flow junction or an output flow junction that are represented as one or more menu items from a click menu in a graphical user interface.

40. The system of claim 32, wherein a flow junction is an output flow function or an input flow junction, the operations further including:

removing an input flow junction or an output flow junction that is represented as one or more menu items from a click menu in a graphical user interface.

41. The system of claim 32, wherein a flow junction is an output flow function or an input flow junction, the operations further including:

adding an input and/or an output flow junction to the one or more sub-graph interfaces.

42. The system of claim 32, wherein a flow junction is an output flow function or an input flow junction, the operations further including:

generating one or more links between input flow junctions and output flow junctions, with a generated link between a given one of the input flow junctions and a given one of the output flow junctions indicative of relationships between metadata associated with the given one of the input flow junctions and metadata associated with the given one of the output flow junctions.

43. The system of claim 42, the operations further including generating a generated link through a user interface.

44. The system of claim 32, the operations further including:

generating a graphical user interface to render a flow junction configuration tab to configure the one or more flow junctions of the one or more sub-graph interfaces, with the flow junction configuration tab including a flow junction list and a propagation control.

45. The system of claim 32, the operations further including wherein the flow junction list renders a list of flow junctions and the propagation control includes one or more controls to select the direction of metadata propagation as an inward transfer direction or an outward transfer direction.

46. The system of claim 32, the operations further including analyzing at least one of the one or more flow junctions of the one or more subgraph interfaces to determine the direction of metadata propagation.