Generation of job flow objects in federated areas from data structure

• US 10,380,185 B2
• Filed: 12/18/2018
• Issued: 08/13/2019
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Jin, Minna, Chen, Huina, Du, Juan, Bequet, Henry Gabriel Victor
• Primary Examiner(s): Patel, Hiren P
• Application Number:
  US16/223,518
• Publication Number:
  US 20190146997A1
• Time in Patent Office: 238 Days
• Field of Search: None
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/9014 hash tables
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 9/46 Multiprogramming arrangements
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An apparatus includes a processor to: receive a request to provide, within a specified federated area, a set of objects that enable a performance of a job flow to perform multiple tasks of an analysis based on data table(s) and formulae of a spreadsheet data structure, wherein the set of objects includes at least one task routine to perform a task of the multiple tasks; correlate each indication of data required as input or output to at least a subpart of a data table; identify data dependencies and determine an order of performance among the multiple tasks based on the formulae; generate, within the specified federated area, a job flow definition that specifies the order of performance of the multiple tasks; and for each task routine of the at least one task routine, generate, within the specified federated area, a corresponding macro data structure.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

receive, by the processor and from a requesting device, a request to provide, within a federated area specified in the request, a set of objects that enable a performance of a job flow to perform multiple tasks of an analysis that is based on data and multiple formulae incorporated into a spreadsheet data structure specified in the request, wherein;

the set of objects comprises at least one data object comprising data required as an input to the job flow, at least one task routine comprising executable instructions to perform a task of the multiple tasks when executed, and a job flow definition that specifies an order of performance of the multiple tasks;

the data incorporated into the spreadsheet data structure is organized into at least one data table within the spreadsheet data structure;

each data table is divisible into multiple data subparts that each comprise at least one row or at least one column of the data table;

the multiple formulae are organized into at least one formula table within the spreadsheet data structure; and

each formula of the multiple formulae specifies a task of the multiple tasks, and incorporates at least one indication of data required as input to perform the task and at least one indication of data that is generated as output when the task is performed;

correlate each indication of data required as input to a subpart of a data table of the at least one data table;

correlate each indication of data generated as output to a subpart of a data table of the at least one data table;

among the multiple formulae, correlate the indications of data required as input to each formula to the indications of data generated as output to identify data dependencies among the multiple tasks, and to identify an error in dependency among at least two tasks of the multiple tasks;

identify at least one pair of tasks of the multiple tasks that are able to be performed in parallel due to a lack of data dependencies therebetween;

determine an order of performance of the multiple tasks based on the identification of the data dependencies and the at least one pair of tasks;

for each data table of the at least one data table that includes at least one subpart that is correlated to an indication of data required as input, and that is not correlated to an indication of data generated as output, generate, within the specified federated area, a data object of the at least one data object that includes the data within the at least one subpart;

generate, within the specified federated area, a job flow definition that specifies the order of performance of the multiple tasks, wherein;

each task of the multiple tasks is specified with a unique flow task identifier of multiple flow task identifiers; and

the job flow definition includes an indication of each identified pair of tasks able to be performed in parallel;

for each task routine of the at least one task routine that is to perform a task of the multiple tasks when executed, generate, within the specified federated area, a corresponding macro data structure of multiple macro data structures, wherein each macro data structure comprises;

the flow task identifier of the task performed by the corresponding task routine;

a task routine identifier that uniquely identifies the corresponding task routine;

indications of characteristics of at least one input interface for each input that is required to perform the task; and

indications of characteristics of at least one output interface for each output that is generated when the task is performed; and

in response to the identification of an error in dependency among at least two tasks of the multiple tasks, augment the macro data structure that corresponds to one task of the at least two tasks to include an indication of the error in dependency to enable an inclusion of a corresponding visual error indication in a visual representation of the corresponding task routine.

2. The apparatus of claim 1, wherein for each formula of the multiple formulae, the processor is caused to perform operations comprising:

generate, within the specified federated area, a corresponding task routine of the at least one task routine that includes executable instructions to perform the task specified by the formula and an indication of the flow task identifier of the task specified by the formula; and

generate the unique task routine identifier for the task routine.

3. The apparatus of claim 2, wherein, in response to identification of at least one subpart of a data table of the at least one data table that is correlated to an indication of data generated as output, and is not correlated to an indication of data required as input, the processor is caused to perform operations comprising:

generate, within the specified federated area, at least one result report that includes the data within the identified at least one subpart; and

generate an instance log to specify aspects of a past performance of the analysis based on the data and the multiple formulae incorporated into the specified spreadsheet structure, wherein the instance log comprises;

a unique data object identifier for each data object of the at least one data object;

the unique task routine identifier of each task routine of the at least one task routine;

a unique result report identifier for each result report of the at least one result report; and

a unique job flow identifier for the job flow definition.

4. The apparatus of claim 1, wherein for each formula of the multiple formulae, the processor is caused to perform operations comprising:

correlate the task specified by the formula to a flow task identifier of the multiple flow task identifiers; and

employ the flow task identifier to identify a task routine among the at least one task routine that includes executable instructions to perform the task.

5. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

allocate storage space among one or more storage devices for the specified federated area;

generate a unique federated area identifier for the specified federated area; and

augment each macro data structure of the multiple macro data structures to include the federated area identifier.

6. The apparatus of claim 1, wherein, for each indication of data required as an input to a task routine of the at least one task routine that is not able to be correlated to a subpart of a data table, the processor is caused to augment the macro data structure that corresponds to the task routine to include an indication of missing data required as an input to the task routine to enable the inclusion of a corresponding visual error indication in a visual representation of the task routine.

7. The apparatus of claim 1, wherein, for each indication of data required as an input to a task routine of the at least one task routine, the processor is caused to perform operations comprising:

compare a data type specified in the indication to a data type of the subpart of a data table of the at least one data table that is correlated to the indication; and

in response to identification of a data type mismatch between the indication and the correlated subpart, augment the macro data structure that corresponds to the task routine to include an indication of a data type mismatch error to enable the inclusion of a corresponding visual error indication in a visual representation of the task routine.

8. The apparatus of claim 1, wherein, in response to the identification of an error in dependency among the at least two tasks, the processor is caused to augment the macro data structure that corresponds to each task of the at least two tasks to include an indication of an error in dependency thereamong to enable the inclusion of a corresponding visual error indication in visual representations of each of the corresponding task routines.

9. The apparatus of claim 1, wherein the processor is caused to transmit the multiple macro data structures to the requesting device via the network to enable the requesting device to generate a visualization of a directed acyclic graph (DAG) of the job flow based at least on the multiple macro data structures, wherein:

the DAG includes, for each macro data structure of the multiple macro data structures, a visual representation of the corresponding task routine of the at least one task routine;

each representation of a task routine comprises;

a task graph object comprising an identifier of the task routine;

at least one input data graph object that represents an input to the task routine, that is visually connected to the task graph object in the representation, and that comprises a visual indication of the at least one characteristic of the input; and

at least one output data graph object that represents an output of the task routine, that is visually connected to the task graph object in the representation, and that comprises an indication of the at least one characteristic of the output; and

for each dependency identified between an output of a first task routine and an input of a second task routine, the visualization includes a dependency marker that visually links the output of the first task routine to the input of the second task routine.

10. The apparatus of claim 1, wherein the processor is caused to generate a visualization of a directed acyclic graph (DAG) of the job flow based at least on the multiple macro data structures, wherein:

the DAG includes, for each macro data structure of the multiple macro data structures, a visual representation of the corresponding task routine of the at least one task routine;

each representation of a task routine comprises;

a task graph object comprising an identifier of the task routine;

at least one input data graph object that represents an input to the task routine, that is visually connected to the task graph object in the representation, and that comprises a visual indication of the at least one characteristic of the input; and

at least one output data graph object that represents an output of the task routine, that is visually connected to the task graph object in the representation, and that comprises an indication of the at least one characteristic of the output; and

for each dependency identified between an output of a first task routine and an input of a second task routine, the visualization includes a dependency marker that visually links the output of the first task routine to the input of the second task routine.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

receive, by the processor and from a requesting device, a request to provide, within a federated area specified in the request, a set of objects that enable a performance of a job flow to perform multiple tasks of an analysis that is based on data and multiple formulae incorporated into a spreadsheet data structure specified in the request, wherein;

the set of objects comprises at least one data object comprising data required as an input to the job flow, at least one task routine comprising executable instructions to perform a task of the multiple tasks when executed, and a job flow definition that specifies an order of performance of the multiple tasks;

the data incorporated into the spreadsheet data structure is organized into at least one data table within the spreadsheet data structure;

each data table is divisible into multiple data subparts that each comprise at least one row or at least one column of the data table;

the multiple formulae are organized into at least one formula table within the spreadsheet data structure; and

each formula of the multiple formulae specifies a task of the multiple tasks, and incorporates at least one indication of data required as input to perform the task and at least one indication of data that is generated as output when the task is performed;

correlate each indication of data required as input to a subpart of a data table of the at least one data table;

correlate each indication of data generated as output to a subpart of a data table of the at least one data table;

among the multiple formulae, correlate the indications of data required as input to each formula to the indications of data generated as output to identify data dependencies among the multiple tasks, and to identify an error in dependency among at least two tasks of the multiple tasks;

identify at least one pair of tasks of the multiple tasks that are able to be performed in parallel due to a lack of data dependencies therebetween;

determine an order of performance of the multiple tasks based on the identification of the data dependencies and the at least one pair of tasks;

for each data table of the at least one data table that includes at least one subpart that is correlated to an indication of data required as input, and that is not correlated to an indication of data generated as output, generate, within the specified federated area, a data object of the at least one data object that includes the data within the at least one subpart;

generate, within the specified federated area, a job flow definition that specifies the order of performance of the multiple tasks, wherein;

each task of the multiple tasks is specified with a unique flow task identifier of multiple flow task identifiers; and

the job flow definition includes an indication of each identified pair of tasks able to be performed in parallel;

for each task routine of the at least one task routine that is to perform a task of the multiple tasks when executed, generate, within the specified federated area, a corresponding macro data structure of multiple macro data structures, wherein each macro data structure comprises;

the flow task identifier of the task performed by the corresponding task routine;

a task routine identifier that uniquely identifies the corresponding task routine;

indications of characteristics of at least one input interface for each input that is required to perform the task; and

indications of characteristics of at least one output interface for each output that is generated when the task is performed; and

in response to the identification of an error in dependency among at least two tasks of the multiple tasks, augment the macro data structure that corresponds to one task of the at least two tasks to include an indication of the error in dependency to enable an inclusion of a corresponding visual error indication in a visual representation of the corresponding task routine.

12. The computer-program product of claim 11, wherein for each formula of the multiple formulae, the processor is caused to perform operations comprising:

generate, within the specified federated area, a corresponding task routine of the at least one task routine that includes executable instructions to perform the task specified by the formula and an indication of the flow task identifier of the task specified by the formula; and

generate the unique task routine identifier for the task routine.

13. The computer-program product of claim 12, wherein, in response to identification of at least one subpart of a data table of the at least one data table that is correlated to an indication of data generated as output, and is not correlated to an indication of data required as input, the processor is caused to perform operations comprising:

generate, within the specified federated area, at least one result report that includes the data within the identified at least one subpart; and

generate, within the specified federated area, an instance log to specify aspects of a past performance of the analysis based on the data and the multiple formulae incorporated into the specified spreadsheet structure, wherein the instance log comprises;

a unique data object identifier for each data object of the at least one data object;

the unique task routine identifier of each task routine of the at least one task routine;

a unique result report identifier for each result report of the at least one result report; and

a unique job flow identifier for the job flow definition.

14. The computer-program product of claim 11, wherein for each formula of the multiple formulae, the processor is caused to perform operations comprising:

correlate the task specified by the formula to a flow task identifier of the multiple flow task identifiers; and

employ the flow task identifier to identify a task routine among the at least one task routine that includes executable instructions to perform the task.

15. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

allocate storage space among one or more storage devices for the specified federated area;

generate a unique federated area identifier for the specified federated area; and

augment each macro data structure of the multiple macro data structures to include the federated area identifier.

16. The computer-program product of claim 11, wherein, for each indication of data required as an input to a task routine of the at least one task routine that is not able to be correlated to a subpart of a data table, the processor is caused to augment the macro data structure that corresponds to the task routine to include an indication of missing data required as an input to the task routine to enable the inclusion of a corresponding visual error indication in a visual representation of the task routine.

17. The computer-program product of claim 11, wherein, for each indication of data required as an input to a task routine of the at least one task routine, the processor is caused to perform operations comprising:

compare a data type specified in the indication to a data type of the subpart of a data table of the at least one data table that is correlated to the indication; and

in response to identification of a data type mismatch between the indication and the correlated subpart, augment the macro data structure that corresponds to the task routine to include an indication of a data type mismatch error to enable the inclusion of a corresponding visual error indication in a visual representation of the task routine.

18. The computer-program product of claim 11, wherein, in response to the identification of an error in dependency among the at least two tasks, the processor is caused to augment the macro data structure that corresponds to each task of the at least two tasks to include an indication of an error in dependency thereamong to enable the inclusion of a corresponding visual error indication in visual representations of each of the corresponding task routines.

19. The computer-program product of claim 11, wherein the processor is caused to transmit the multiple macro data structures to the requesting device via the network to enable the requesting device to generate a visualization of a directed acyclic graph (DAG) of the job flow based at least on the multiple macro data structures, wherein:

the DAG includes, for each macro data structure of the multiple macro data structures, a visual representation of the corresponding task routine of the at least one task routine;

each representation of a task routine comprises;

a task graph object comprising an identifier of the task routine;

at least one input data graph object that represents an input to the task routine, that is visually connected to the task graph object in the representation, and that comprises a visual indication of the at least one characteristic of the input; and

at least one output data graph object that represents an output of the task routine, that is visually connected to the task graph object in the representation, and that comprises an indication of the at least one characteristic of the output; and

for each dependency identified between an output of a first task routine and an input of a second task routine, the visualization includes a dependency marker that visually links the output of the first task routine to the input of the second task routine.

20. The computer-program product of claim 11, wherein the processor is caused generate a visualization of a directed acyclic graph (DAG) of the job flow based at least on the multiple macro data structures, wherein:

the DAG includes, for each macro data structure of the multiple macro data structures, a visual representation of the corresponding task routine of the at least one task routine;

each representation of a task routine comprises;

a task graph object comprising an identifier of the task routine;

at least one input data graph object that represents an input to the task routine, that is visually connected to the task graph object in the representation, and that comprises a visual indication of the at least one characteristic of the input; and

at least one output data graph object that represents an output of the task routine, that is visually connected to the task graph object in the representation, and that comprises an indication of the at least one characteristic of the output; and

for each dependency identified between an output of a first task routine and an input of a second task routine, the visualization includes a dependency marker that visually links the output of the first task routine to the input of the second task routine.

21. A computer-implemented method comprising:

receiving, by a processor and from a requesting device, a request to provide, within a federated area specified in the request, a set of objects that enable a performance of a job flow to perform multiple tasks of an analysis that is based on data and multiple formulae incorporated into a spreadsheet data structure specified in the request, wherein;

the set of objects comprises at least one data object comprising data required as an input to the job flow, at least one task routine comprising executable instructions to perform a task of the multiple tasks when executed, and a job flow definition that specifies an order of performance of the multiple tasks;

the data incorporated into the spreadsheet data structure is organized into at least one data table within the spreadsheet data structure;

each data table is divisible into multiple data subparts that each comprise at least one row or at least one column of the data table;

the multiple formulae are organized into at least one formula table within the spreadsheet data structure; and

each formula of the multiple formulae specifies a task of the multiple tasks, and incorporates at least one indication of data required as input to perform the task and at least one indication of data that is generated as output when the task is performed;

correlating, by the processor, each indication of data required as input to a subpart of a data table of the at least one data table;

correlating, by the processor, each indication of data generated as output to a subpart of a data table of the at least one data table;

among the multiple formulae, correlating, by the processor, the indications of data required as input to each formula to the indications of data generated as output to identify data dependencies among the multiple tasks, and to identify an error in dependency among at least two tasks of the multiple tasks;

identifying, by the processor, at least one pair of tasks of the multiple tasks that are able to be performed in parallel due to a lack of data dependencies therebetween;

determining, by the processor, an order of performance of the multiple tasks based on the identification of the data dependencies and the at least one pair of tasks;

for each data table of the at least one data table that includes at least one subpart that is correlated to an indication of data required as input, and that is not correlated to an indication of data generated as output, generating, by the processor and within the specified federated area, a data object of the at least one data object that includes the data within the at least one subpart;

generating, by the processor and within the specified federated area, the job flow definition that specifies the order of performance of the multiple tasks, wherein;

each task of the multiple tasks is specified with a unique flow task identifier of multiple flow task identifiers; and

the job flow definition includes an indication of each identified pair of tasks able to be performed in parallel;

for each task routine of the at least one task routine that is to perform a task of the multiple tasks when executed, generating, by the processor and within the specified federated area, a corresponding macro data structure of multiple macro data structures, wherein each macro data structure comprises;

the flow task identifier of the task performed by the corresponding task routine;

a task routine identifier that uniquely identifies the corresponding task routine;

indications of characteristics of at least one input interface for each input that is required to perform the task; and

indications of characteristics of at least one output interface for each output that is generated when the task is performed; and

in response to the identification of an error in dependency among at least two tasks of the multiple tasks, augmenting the macro data structure that corresponds to one task of the at least two tasks to include an indication of the error in dependency to enable an inclusion of a corresponding visual error indication in a visual representation of the corresponding task routine.

22. The computer-implemented method of claim 21, further comprising, for each formula of the multiple formulae, performing operations comprising:

generating, by the processor and within the specified federated area, a corresponding task routine of the at least one task routine that includes executable instructions to perform the task specified by the formula and an indication of the flow task identifier of the task specified by the formula; and

generating, by the processor, the unique task routine identifier for the task routine.

23. The computer-implemented method of claim 22, further comprising, in response to identification of at least one subpart of a data table of the at least one data table that is correlated to an indication of data generated as output, and is not correlated to an indication of data required as input, performing operations comprising:

generating, by the processor and within the specified federated area, at least one result report that includes the data within the identified at least one subpart; and

generating, by the processor and within the specified federated area, an instance log to specify aspects of a past performance of the analysis based on the data and the multiple formulae incorporated into the specified spreadsheet structure, wherein the instance log comprises;

a unique data object identifier for each data object of the at least one data object;

the unique task routine identifier of each task routine of the at least one task routine;

a unique result report identifier for each result report of the at least one result report; and

a unique job flow identifier for the job flow definition.

24. The computer-implemented method of claim 21, further comprising, for each formula of the multiple formulae, performing operations comprising:

correlating, by the processor, the task specified by the formula to a flow task identifier of the multiple flow task identifiers; and

employing the flow task identifier to identify a task routine among the at least one task routine that includes executable instructions to perform the task.

25. The computer-implemented method of claim 21, further comprising:

allocating storage space among one or more storage devices for the specified federated area;

generating, by the processor, a unique federated area identifier for the specified federated area; and

augmenting each macro data structure of the multiple macro data structures to include the federated area identifier.

26. The computer-implemented method of claim 21, further comprising, for each indication of data required as an input to a task routine of the at least one task routine that is not able to be correlated to a subpart of a data table, augmenting the macro data structure that corresponds to the task routine to include an indication of missing data required as an input to the task routine to enable the inclusion of a corresponding visual error indication in a visual representation of the task routine.

27. The computer-implemented method of claim 21, further comprising, for each indication of data required as an input to a task routine of the at least one task routine, performing operations comprising:

comparing, by the processor, a data type specified in the indication to a data type of the subpart of a data table of the at least one data table that is correlated to the indication; and

in response to identification of a data type mismatch between the indication and the correlated subpart, augmenting the macro data structure that corresponds to the task routine to include an indication of a data type mismatch error to enable the inclusion of a corresponding visual error indication in a visual representation of the task routine.

28. The computer-implemented method of claim 21, further comprising, in response to the identification of an error in dependency among the at least two tasks, augmenting the macro data structure that corresponds to each task of the at least two tasks to include an indication of an error in dependency thereamong to enable the inclusion of a corresponding visual error indication in visual representations of each of the corresponding task routines.

29. The computer-implemented method of claim 21, further comprising, transmitting the multiple macro data structures to the requesting device via the network to enable the requesting device to generate a visualization of a directed acyclic graph (DAG) of the job flow based at least on the multiple macro data structures, wherein:

the DAG includes, for each macro data structure of the multiple macro data structures, a visual representation of the corresponding task routine of the at least one task routine;

each representation of a task routine comprises;

a task graph object comprising an identifier of the task routine;

at least one input data graph object that represents an input to the task routine, that is visually connected to the task graph object in the representation, and that comprises a visual indication of the at least one characteristic of the input; and

at least one output data graph object that represents an output of the task routine, that is visually connected to the task graph object in the representation, and that comprises an indication of the at least one characteristic of the output; and

for each dependency identified between an output of a first task routine and an input of a second task routine, the visualization includes a dependency marker that visually links the output of the first task routine to the input of the second task routine.

30. The computer-implemented method of claim 21, further comprising, generating, by the processor, a visualization of a directed acyclic graph (DAG) of the job flow based at least on the multiple macro data structures, wherein:

the DAG includes, for each macro data structure of the multiple macro data structures, a visual representation of the corresponding task routine of the at least one task routine;

each representation of a task routine comprises;

a task graph object comprising an identifier of the task routine;

at least one input data graph object that represents an input to the task routine, that is visually connected to the task graph object in the representation, and that comprises a visual indication of the at least one characteristic of the input; and

at least one output data graph object that represents an output of the task routine, that is visually connected to the task graph object in the representation, and that comprises an indication of the at least one characteristic of the output; and

for each dependency identified between an output of a first task routine and an input of a second task routine, the visualization includes a dependency marker that visually links the output of the first task routine to the input of the second task routine.