Generation of job flow objects in federated areas from data structure

• US 10,394,890 B2
• Filed: 12/20/2018
• Issued: 08/27/2019
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Jin, Minna, Chen, Huina, Du, Juan, Bequet, Henry Gabriel Victor
• Primary Examiner(s): Patel, Hiren P
• Application Number:
  US16/226,828
• Publication Number:
  US 20190146998A1
• Time in Patent Office: 250 Days
• Field of Search: None
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/9014 hash tables
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 9/46 Multiprogramming arrangements
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An apparatus includes a processor to: receive a request to generate a DAG of a job flow of multiple tasks of an analysis based on data table(s) and formulae of a spreadsheet data structure; correlate each indication of data required as input or output to at least a subpart of a data table; identify data dependencies and determine an order of performance among the multiple tasks based on the formulae; generate, within the specified federated area, a job flow definition that specifies the order of performance of the multiple tasks; for each task of the multiple tasks, generate, within the specified federated area, a corresponding macro data structure of multiple macro data structures; and generate the requested visualization based on the job flow definition and the multiple macro data structures.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

receive, by the processor and from an input device, a request to generate a visualization of a directed acyclic graph (DAG) of a job flow of multiple tasks of an analysis that is based on data and multiple formulae incorporated into a spreadsheet data structure specified in the request, wherein;

the data incorporated into the spreadsheet data structure is organized into at least one data table within the spreadsheet data structure;

each data table is divisible into multiple data subparts that each comprise at least one row or at least one column of the data table;

the multiple formulae are organized into at least one formula table within the spreadsheet data structure; and

each formula of the multiple formulae specifies a task of the multiple tasks, and incorporates at least one indication of data required as input to perform the task and at least one indication of data that is generated as output when the task is performed;

correlate each indication of data required as input to a subpart of a data table of the at least one data table;

correlate each indication of data generated as output to a subpart of a data table of the at least one data table;

among the multiple formulae, correlate the indications of data required as input to the indications of data generated as output to identify data dependencies among the multiple tasks;

identify at least one pair of tasks of the multiple tasks that are able to be performed in parallel due to a lack of data dependencies therebetween;

determine an order of performance of the multiple tasks based on the identification of the data dependencies and the at least one pair of tasks;

generate, within a federated area specified in the request, a job flow definition that specifies the order of performance of the multiple tasks, wherein;

each task of the multiple tasks is specified with a unique flow task identifier of multiple flow task identifiers; and

the job flow definition includes an indication of each identified pair of tasks able to be performed in parallel;

for each task of the multiple tasks, generate, within the specified federated area, a corresponding macro data structure of multiple macro data structures, wherein each macro data structure comprises;

the flow task identifier of the task;

indications of characteristics of at least one input interface for each input that is required to perform the task; and

indications of characteristics of at least one output interface for each output that is generated when the task is performed; and

for each indication of data required as an input to a task of the multiple tasks, the processor is caused to;

in response to an indication of data required as an input to a task that is not able to be correlated to a subpart of a data table, augment the macro data structure that corresponds to the task to include an indication of missing data required as an input to the task to enable the inclusion of a corresponding visual error indication in a visual representation of the task;

or

in response to a data type mismatch between a data type of specified in an indication of data required as an input to a task and a data type of the subpart of the data table of the at least one data table that is correlated to the indication, augment the macro data structure that corresponds to the task to include an indication of a data type mismatch error to enable the inclusion of a corresponding visual error indication in a visual representation of the task;

generate the requested visualization based on the job flow definition and the multiple macro data structures, wherein;

the visualization includes, for each macro data structure of the multiple macro data structures, a visual representation of the corresponding task of the multiple tasks; and

each representation of a task comprises;

a task graph object;

at least one input data graph object that represents an input, that has a connection to the task graph object in the representation, and that comprises a visual indication of the at least one characteristic of the input; and

at least one output data graph object that represents an output, that has a connection to the task graph object in the representation, and that comprises an indication of the at least one characteristic of the output.

2. The apparatus of claim 1, wherein, for each indication of data required as an input to a task of the multiple tasks, the processor is caused to

compare the data type specified in the indication to the data type of the subpart of a data table of the at least one data table that is correlated to the indication.

3. The apparatus of claim 1, wherein, in response to identification of an error in dependency among at least two tasks of the multiple tasks as a result of the correlation of the indications of data required as input to the indications of data generated as output, the processor is caused to augment the macro data structure that corresponds to each task of the at least two tasks to include an indication of an error in dependency thereamong to enable the inclusion of a corresponding visual error indication in visual representations of each of the tasks.

4. The apparatus of claim 1, wherein:

the request includes a request to provide, within the specified area, a set of objects that enable a performance of the job flow that is based on the data and multiple formulae incorporated into the specified spreadsheet data structure, wherein the set of objects comprises at least one data object comprising data required as an input to the job flow, at least one task routine comprising executable instructions to perform a task of the multiple tasks when executed, and the job flow definition; and

the processor is caused to perform operations comprising;

allocate storage space among one or more storage devices for the specified federated area;

generate a unique federated area identifier for the specified federated area; and

augment each macro data structure of the multiple macro data structures to include the federated area identifier.

5. The apparatus of claim 4, wherein for each formula of the multiple formulae, the processor is caused to perform operations comprising:

generate, within the specified federated area, a corresponding task routine of the at least one task routine that includes executable instructions to perform the corresponding task of the multiple tasks and an indication of the corresponding flow task identifier;

generate a unique task routine identifier for the task routine; and

augment the corresponding macro data structure to include the unique task routine identifier.

6. The apparatus of claim 5, wherein the processor is caused to, for each data table of the at least one data table that includes at least one subpart that is correlated to an indication of data required as input to a task, and that is not correlated to an indication of data generated as output, perform operations comprising:

generate, within the specified federated area, a data object of the at least one data object that includes the data within the at least one subpart;

generate a unique data object identifier for the data object; and

augment the macro data structure that corresponds to the task to include the unique data object identifier.

7. The apparatus of claim 6, wherein, in response to identification of at least one subpart of a data table of the at least one data table that is correlated to an indication of data generated as output, and is not correlated to an indication of data required as input, the processor is caused to perform operations comprising:

generate, within the specified federated area, at least one result report that includes the data within the identified at least one subpart; and

generate an instance log to specify aspects of a past performance of the analysis based on the data and the multiple formulae incorporated into the specified spreadsheet structure, wherein the instance log comprises;

the unique data object identifier for each data object of the at least one data object;

the unique task routine identifier of each task routine of the at least one task routine;

a unique result report identifier for each result report of the at least one result report; and

a unique job flow identifier for the job flow definition.

8. The apparatus of claim 1, wherein for a dependency identified between an output of a first task and an input of a second task, the visualization includes a dependency marker that visually links the output of the first task to the input of the second task.

9. The apparatus of claim 1, wherein the processor is caused to transmit the job flow definition and the multiple macro data structures to another device via a network to enable the other device to generate the visualization based on the job flow definition and the multiple macro data structures.

10. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

receive, by the processor and from an input device, a request to generate a visualization of a directed acyclic graph (DAG) of a job flow of multiple tasks of an analysis that is based on data and multiple formulae incorporated into a spreadsheet data structure specified in the request, wherein;

the data incorporated into the spreadsheet data structure is organized into at least one data table within the spreadsheet data structure;

each data table is divisible into multiple data subparts that each comprise at least one row or at least one column of the data table;

the multiple formulae are organized into at least one formula table within the spreadsheet data structure; and

each formula of the multiple formulae specifies a task of the multiple tasks, and incorporates at least one indication of data required as input to perform the task and at least one indication of data that is generated as output when the task is performed;

correlate each indication of data required as input to a subpart of a data table of the at least one data table;

correlate each indication of data generated as output to a subpart of a data table of the at least one data table;

among the multiple formulae, correlate the indications of data required as input to the indications of data generated as output to identify data dependencies among the multiple tasks;

identify at least one pair of tasks of the multiple tasks that are able to be performed in parallel due to a lack of data dependencies therebetween;

determine an order of performance of the multiple tasks based on the identification of the data dependencies and the at least one pair of tasks;

generate, within a federated area specified in the request, a job flow definition that specifies the order of performance of the multiple tasks, wherein;

each task of the multiple tasks is specified with a unique flow task identifier of multiple flow task identifiers; and

the job flow definition includes an indication of each identified pair of tasks able to be performed in parallel;

for each task of the multiple tasks, generate, within the specified federated area, a corresponding macro data structure of multiple macro data structures, wherein each macro data structure comprises;

the flow task identifier of the task;

indications of characteristics of at least one input interface for each input that is required to perform the task; and

indications of characteristics of at least one output interface for each output that is generated when the task is performed; and

for each indication of data required as an input to a task of the multiple tasks, the processor is caused to;

in response to an indication of data required as an input to a task that is not able to be correlated to a subpart of a data table, augment the macro data structure that corresponds to the task to include an indication of missing data required as an input to the task to enable the inclusion of a corresponding visual error indication in a visual representation of the task;

or

in response to a data type mismatch between a data type of specified in an indication of data required as an input to a task and a data type of the subpart of the data table of the at least one data table that is correlated to the indication, augment the macro data structure that corresponds to the task to include an indication of a data type mismatch error to enable the inclusion of a corresponding visual error indication in a visual representation of the task;

generate the requested visualization based on the job flow definition and the multiple macro data structures, wherein;

the visualization includes, for each macro data structure of the multiple macro data structures, a visual representation of the corresponding task of the multiple tasks; and

each representation of a task comprises;

a task graph object;

at least one input data graph object that represents an input, that has a connection to the task graph object in the representation, and that comprises a visual indication of the at least one characteristic of the input; and

at least one output data graph object that represents an output, that has a connection to the task graph object in the representation, and that comprises an indication of the at least one characteristic of the output.

11. The computer-program product of claim 10, wherein, for each indication of data required as an input to a task of the multiple tasks, the processor is caused to:

compare the data type specified in the indication to the data type of the subpart of a data table of the at least one data table that is correlated to the indication.

12. The computer-program product of claim 10, wherein, in response to identification of an error in dependency among at least two tasks of the multiple tasks as a result of the correlation of the indications of data required as input to the indications of data generated as output, the processor is caused to augment the macro data structure that corresponds to each task of the at least two tasks to include an indication of an error in dependency thereamong to enable the inclusion of a corresponding visual error indication in visual representations of each of the tasks.

13. The computer-program product of claim 10, wherein:

the request includes a request to provide, within the specified area, a set of objects that enable a performance of the job flow that is based on the data and multiple formulae incorporated into the specified spreadsheet data structure, wherein the set of objects comprises at least one data object comprising data required as an input to the job flow, at least one task routine comprising executable instructions to perform a task of the multiple tasks when executed, and the job flow definition; and

the processor is caused to perform operations comprising;

allocate storage space among one or more storage devices for the specified federated area;

generate a unique federated area identifier for the specified federated area; and

augment each macro data structure of the multiple macro data structures to include the federated area identifier.

14. The computer-program product of claim 13, wherein for each formula of the multiple formulae, the processor is caused to perform operations comprising:

generate, within the specified federated area, a corresponding task routine of the at least one task routine that includes executable instructions to perform the corresponding task of the multiple tasks and an indication of the corresponding flow task identifier;

generate a unique task routine identifier for the task routine; and

augment the corresponding macro data structure to include the unique task routine identifier.

15. The computer-program product of claim 14, wherein the processor is caused to, for each data table of the at least one data table that includes at least one subpart that is correlated to an indication of data required as input to a task, and that is not correlated to an indication of data generated as output, perform operations comprising:

generate, within the specified federated area, a data object of the at least one data object that includes the data within the at least one subpart;

generate a unique data object identifier for the data object; and

augment the macro data structure that corresponds to the task to include the unique data object identifier.

16. The computer-program product of claim 15, wherein, in response to identification of at least one subpart of a data table of the at least one data table that is correlated to an indication of data generated as output, and is not correlated to an indication of data required as input, the processor is caused to perform operations comprising:

generate, within the specified federated area, at least one result report that includes the data within the identified at least one subpart; and

generate an instance log to specify aspects of a past performance of the analysis based on the data and the multiple formulae incorporated into the specified spreadsheet structure, wherein the instance log comprises;

the unique data object identifier for each data object of the at least one data object;

the unique task routine identifier of each task routine of the at least one task routine;

a unique result report identifier for each result report of the at least one result report; and

a unique job flow identifier for the job flow definition.

17. The computer-program product of claim 10, wherein for each formula of the multiple formulae, the processor is caused to perform operations comprising:

employ the flow task identifier to identify a task routine that includes executable instructions to perform the task; and

augment the corresponding macro data structure to include the unique task routine identifier.

18. The computer-program product of claim 10, wherein the processor is caused to transmit the job flow definition and the multiple macro data structures to another device via the network to enable the other device to generate the visualization based on the job flow definition and the multiple macro data structures.

19. A computer-implemented method comprising:

receiving, by a processor and from a requesting device, a request to generate a visualization of a directed acyclic graph (DAG) of a job flow of multiple tasks of an analysis that is based on data and multiple formulae incorporated into a spreadsheet data structure specified in the request, wherein;

the data incorporated into the spreadsheet data structure is organized into at least one data table within the spreadsheet data structure;

each data table is divisible into multiple data subparts that each comprise at least one row or at least one column of the data table;

the multiple formulae are organized into at least one formula table within the spreadsheet data structure; and

each formula of the multiple formulae specifies a task of the multiple tasks, and incorporates at least one indication of data required as input to perform the task and at least one indication of data that is generated as output when the task is performed;

correlating, by the processor, each indication of data required as input to a subpart of a data table of the at least one data table;

correlating, by the processor, each indication of data generated as output to a subpart of a data table of the at least one data table;

among the multiple formulae, correlating, by the processor, the indications of data required as input to the indications of data generated as output to identify data dependencies among the multiple tasks;

identifying, by the processor, at least one pair of tasks of the multiple tasks that are able to be performed in parallel due to a lack of data dependencies therebetween;

determining, by the processor, an order of performance of the multiple tasks based on the identification of the data dependencies and the at least one pair of tasks;

generating, by the processor and within a federated area specified in the request, a job flow definition that specifies the order of performance of the multiple tasks, wherein;

each task of the multiple tasks is specified with a unique flow task identifier of multiple flow task identifiers; and

the job flow definition includes an indication of each identified pair of tasks able to be performed in parallel;

for each task of the multiple tasks, generating, by the processor and within the specified federated area, a corresponding macro data structure of multiple macro data structures, wherein each macro data structure comprises;

the flow task identifier of the task;

indications of characteristics of at least one input interface for each input that is required to perform the task; and

indications of characteristics of at least one output interface for each output that is generated when the task is performed; and

for each indication of data required as an input to a task of the multiple tasks, performing operations comprising;

in response to an indication of data required as an input to a task that is not able to be correlated to a subpart of a data table, augmenting the macro data structure that corresponds to the task to include an indication of missing data required as an input to the task to enable the inclusion of a corresponding visual error indication in a visual representation of the task;

or

in response to a data type mismatch between a data type of specified in an indication of data required as an input to a task and a data type of the subpart of the data table of the at least one data table that is correlated to the indication, augmenting the macro data structure that corresponds to the task to include an indication of a data type mismatch error to enable the inclusion of a corresponding visual error indication in a visual representation of the task;

generating, by the processor, the requested visualization based on the job flow definition and the multiple macro data structures, wherein;

the visualization includes, for each macro data structure of the multiple macro data structures, a visual representation of the corresponding task of the multiple tasks; and

each representation of a task comprises;

a task graph object;

at least one input data graph object that represents an input, that has a connection to the task graph object in the representation, and that comprises a visual indication of the at least one characteristic of the input; and

at least one output data graph object that represents an output, that has a connection to the task graph object in the representation, and that comprises an indication of the at least one characteristic of the output.

20. The computer-implemented method of claim 19, further comprising, for each indication of data required as an input to a task of the multiple tasks,

comparing, by the processor, the data type specified in the indication to the data type of the subpart of a data table of the at least one data table that is correlated to the indication.

21. The computer-implemented method of claim 19, further comprising, in response to identification of an error in dependency among at least two tasks of the multiple tasks as a result of the correlation of the indications of data required as input to the indications of data generated as output, augmenting the macro data structure that corresponds to each task of the at least two tasks to include an indication of an error in dependency thereamong to enable the inclusion of a corresponding visual error indication in visual representations of each of the tasks.

22. The computer-implemented method of claim 19, wherein:

the request includes a request to provide, within the specified area, a set of objects that enable a performance of the job flow that is based on the data and multiple formulae incorporated into the specified spreadsheet data structure, wherein the set of objects comprises at least one data object comprising data required as an input to the job flow, at least one task routine comprising executable instructions to perform a task of the multiple tasks when executed, and the job flow definition; and

the method further comprises performing operations comprising;

allocate storage space among one or more storage devices for the specified federated area;

generate a unique federated area identifier for the specified federated area; and

augment each macro data structure of the multiple macro data structures to include the federated area identifier.

23. The computer-implemented method of claim 22, further comprising, for each formula of the multiple formulae, performing operations comprising:

generating, by the processor and within the specified federated area, a corresponding task routine of the at least one task routine that includes executable instructions to perform the corresponding task of the multiple tasks and an indication of the corresponding flow task identifier;

generating, by the processor, a unique task routine identifier for the task routine; and

augmenting the corresponding macro data structure to include the unique task routine identifier.

24. The computer-implemented method of claim 23, further comprising, for each data table of the at least one data table that includes at least one subpart that is correlated to an indication of data required as input to a task, and that is not correlated to an indication of data generated as output, performing operations comprising:

generating, by the processor and within the specified federated area, a data object of the at least one data object that includes the data within the at least one subpart;

generating a unique data object identifier for the data object; and

augmenting the macro data structure that corresponds to the task to include the unique data object identifier.

25. The computer-implemented method of claim 24, further comprising, in response to identification of at least one subpart of a data table of the at least one data table that is correlated to an indication of data generated as output, and is not correlated to an indication of data required as input, performing operations comprising:

generating, by the processor and within the specified federated area, at least one result report that includes the data within the identified at least one subpart; and

generating, by the processor, an instance log to specify aspects of a past performance of the analysis based on the data and the multiple formulae incorporated into the specified spreadsheet structure, wherein the instance log comprises;

the unique data object identifier for each data object of the at least one data object;

the unique task routine identifier of each task routine of the at least one task routine;

a unique result report identifier for each result report of the at least one result report; and

a unique job flow identifier for the job flow definition.

26. The computer-implemented method of claim 19, further comprising, for each formula of the multiple formulae, performing operations comprising:

employing the flow task identifier to identify a task routine that includes executable instructions to perform the task; and

augmenting the corresponding macro data structure to include the unique task routine identifier.

27. The computer-implemented method of claim 19, further comprising transmitting the job flow definition and the multiple macro data structures to another device via the network to enable the other device to generate the visualization based on the job flow definition and the multiple macro data structures.