Automated exchanges of job flow objects between federated area and external storage space

• US 10,649,750 B2
• Filed: 08/13/2019
• Issued: 05/12/2020
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Bequet, Henry Gabriel Victor, Arfaoui, Kais
• Primary Examiner(s): Patel, Hiren P
• Application Number:
  US16/539,222
• Publication Number:
  US 20190369974A1
• Time in Patent Office: 273 Days
• Field of Search: None
• CPC Class Codes: G06F 16/9014 hash tables
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 8/51 Source to source
• CPC Class Codes: G06F 9/46 Multiprogramming arrangements
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An apparatus includes a processor to: receive a job flow definition; retrieve the most recent versions of a set of task routines for the defined job flow; translate, into an intermediate representation, executable instructions of each task routine implementing an interface for data input and/or output during execution; translate executable instructions of the job flow definition that defines the interface for each task routine into an intermediate representation; compare each intermediate representation from a task routine to the corresponding intermediate representation from the job flow definition to determine if there is a match; and in response to there being a match for each comparison and to the executable instructions of the job flow definition being written in a secondary programming language, translate the executable instructions of the job flow definition into a primary programming language, and store the resulting translated form of the job flow definition in a federated area.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

receive, at the processor, and from a remote device via a network, a job flow definition to be stored in a federated area of at least one federated area, wherein;

the job flow definition defines a job flow as a set of tasks to be performed by execution of a corresponding set of task routines to perform the job flow;

the job flow definition employs a set of flow task identifiers to identify the set of tasks; and

the at least one federated area is maintained within at least one storage device to store the job flow definition, multiple task routines and multiple data sets as objects;

retrieve the set of flow task identifiers from the job flow definition;

for each retrieved flow task identifier, retrieve, from among the multiple task routines, a most recent version of a task routine of the set of task routines that performs the corresponding task of the set of tasks when executed;

translate a portion of executable instructions within each retrieved task routine of the set of task routines that implements an interface by which a data set is accepted as an input or is output during execution of the task routine into an intermediate representation;

analyze executable instructions of the job flow definition to determine whether the executable instructions of the job flow definition are written in a primary programming language;

translate a portion of the executable instructions within the job flow definition that defines the interface for each task routine of the set of task routines into an intermediate representation;

compare each intermediate representation generated from one of the retrieved task routines to the corresponding intermediate representation generated from the job flow definition to determine if there is a match;

in response to a determination that there is a match for each comparison of intermediate representations, and in response to a determination that the executable instructions of the job flow definition are written in a secondary programming language, perform operations comprising;

translate the portion of the executable instructions of the job flow definition that defines the interface for each task routine of the set of task routines into the primary programming language to generate a translated form of the job flow definition; and

store the translated form of the job flow definition within the federated area; and

in response to a determination that there is a lack of a match for at least one comparison of intermediate representations, transmit an indication of a lack of match to the remote device.

2. The apparatus of claim 1, wherein the processor is caused to:

maintain a first transfer area within the federated area;

cooperate with the remote device via the network to exchange objects via the network to synchronize objects between the first transfer area and a second transfer area maintained by the remote device;

cooperate with the remote device to receive the job flow definition in an exchange of objects via the network to synchronize the objects between the first transfer area and the second transfer area in response to the job flow definition having been stored within the second transfer area or in response to the a more recent version of the job flow definition having been stored within the second transfer area; and

in response to a determination that there is a match for each comparison of intermediate representations, and in response to a determination that the executable instructions of the job flow definition are written in a secondary programming language, store the translated form of the job flow definition within the first transfer area.

3. The apparatus of claim 2, wherein the processor is caused to, in response to a change having been made to the translated form of the job flow definition stored within the first transfer area, perform operations comprising:

reverse-translate the portion of the executable instructions of the changed translated form of the job flow definition that defines the interface for each task routine of the set of task routines from the primary programming language into the secondary programming language to generate a reverse-translated form of the job flow definition; and

cooperate with the remote device via the network to transmit the reverse-translated form of the job flow definition to the remote device in an exchange of objects via the network to synchronize the objects between the first transfer area and the second transfer area.

4. The apparatus of claim 3, wherein:

the executable instructions of the job flow definition include a portion of executable instructions to implement a graphical user interface (GUI) when executed;

in response to a determination that there is a match for each comparison of intermediate representations, and in response to a determination that the executable instructions of the job flow definition are written in a secondary programming language, the processor is caused to translate the portion of the executable instructions that implement the GUI from the secondary programming language into GUI instructions within the translated form of the job flow definition in the primary programming language; and

in response to a change having been made to the translated form of the job flow definition stored within the first transfer area, the processor is caused to reverse-translate the GUI instructions within the changed translated form of the job flow definition into the secondary language in a corresponding portion of the executable instructions of the reverse-translated form of the job flow definition.

5. The apparatus of claim 2, wherein the processor is caused, in response to a determination that there is a lack of a match for at least one comparison of intermediate representations, to perform operations comprising:

generate a directed acyclic graph (DAG) that depicts the lack of a match for the at least one comparison; and

transmit the DAG as the indication of a lack of match to the remote device.

6. The apparatus of claim 2, wherein:

the first transfer area and the second transfer area are used cooperatively to store and exchange objects as part of collaborative development of a set of objects of the job flow;

the processor receives an indication that the job flow definition has been committed to become part of a set of objects required to perform the job flow; and

in response to the receipt of the indication, the processor cooperates with the remote device to receive the job flow definition in an exchange of objects.

7. The apparatus of claim 2, wherein the processor is caused to perform operations comprising:

receive, at the processor, and from the remote device via the network, security credentials from the remote device as the remote device logs into the federated area as a user;

analyze the security credentials to determine whether the remote device is authorized to log into the federated area; and

in response to a determination that the remote device is authorized to log into the federated area, the processor grants access to the federated area to the remote device to enable receipt of the job flow definition.

8. The apparatus of claim 1, the processor is caused to perform operations comprising:

use the set of flow task identifiers retrieved from the job flow definition to search the at least one federated area for at least one task routine to perform each task of the set of tasks; and

in response to a lack of a task routine being stored within the one or more federated for at least one task of the set of tasks, perform operations comprising;

generate a directed acyclic graph (DAG) of the job flow definition that identifies the at least one task; and

transmit the DAG to the remote device.

9. The apparatus of claim 1, wherein the translation of the portion of the executable instructions of the job flow definition that defines the interface for each task routine of the set of task routines into the primary programming language comprises translating the intermediate expression for the definition of the interface for each task routine into executable instructions in the primary programming language.

10. The apparatus of claim 1, wherein:

the intermediate expression comprises executable instructions generated in an intermediate programming language; and

one of the primary programming language, the secondary programming language and the intermediate programming language is selected from a group consisting of;

SAS programming language;

Python;

JSON;

Pascal;

Fortran;

BASIC;

C;

C++;

R; and

CUDA.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

receive, at the processor, and from a remote device via a network, a job flow definition to be stored in a federated area of at least one federated area, wherein;

the job flow definition defines a job flow as a set of tasks to be performed by execution of a corresponding set of task routines to perform the job flow;

the job flow definition employs a set of flow task identifiers to identify the set of tasks; and

the at least one federated area is maintained within at least one storage device to store the job flow definition, multiple task routines and multiple data sets as objects;

retrieve the set of flow task identifiers from the job flow definition;

for each retrieved flow task identifier, retrieve, from among the multiple task routines, a most recent version of a task routine of the set of task routines that performs the corresponding task of the set of tasks when executed;

translate a portion of executable instructions within each retrieved task routine of the set of task routines that implements an interface by which a data set is accepted as an input or is output during execution of the task routine into an intermediate representation;

analyze executable instructions of the job flow definition to determine whether the executable instructions of the job flow definition are written in a primary programming language;

translate a portion of the executable instructions within the job flow definition that defines the interface for each task routine of the set of task routines into an intermediate representation;

compare each intermediate representation generated from one of the retrieved task routines to the corresponding intermediate representation generated from the job flow definition to determine if there is a match;

in response to a determination that there is a match for each comparison of intermediate representations, and in response to a determination that the executable instructions of the job flow definition are written in a secondary programming language, perform operations comprising;

translate the portion of the executable instructions of the job flow definition that defines the interface for each task routine of the set of task routines into the primary programming language to generate a translated form of the job flow definition; and

store the translated form of the job flow definition within the federated area; and

in response to a determination that there is a lack of a match for at least one comparison of intermediate representations, transmit an indication of a lack of match to the remote device.

12. The computer-program product of claim 11, wherein the processor is caused to:

maintain a first transfer area within the federated area;

cooperate with the remote device via the network to exchange objects via the network to synchronize objects between the first transfer area and a second transfer area maintained by the remote device;

cooperate with the remote device to receive the job flow definition in an exchange of objects via the network to synchronize the objects between the first transfer area and the second transfer area in response to the job flow definition having been stored within the second transfer area or in response to the a more recent version of the job flow definition having been stored within the second transfer area; and

in response to a determination that there is a match for each comparison of intermediate representations, and in response to a determination that the executable instructions of the job flow definition are written in a secondary programming language, store the translated form of the job flow definition within the first transfer area.

13. The computer-program product of claim 12, wherein the processor is caused to, in response to a change having been made to the translated form of the job flow definition stored within the first transfer area, perform operations comprising:

reverse-translate the portion of the executable instructions of the changed translated form of the job flow definition that defines the interface for each task routine of the set of task routines from the primary programming language into the secondary programming language to generate a reverse-translated form of the job flow definition; and

cooperate with the remote device via the network to transmit the reverse-translated form of the job flow definition to the remote device in an exchange of objects via the network to synchronize the objects between the first transfer area and the second transfer area.

14. The computer-program product of claim 13, wherein:

the executable instructions of the job flow definition include a portion of executable instructions to implement a graphical user interface (GUI) when executed;

in response to a determination that there is a match for each comparison of intermediate representations, and in response to a determination that the executable instructions of the job flow definition are written in a secondary programming language, the processor is caused to translate the portion of the executable instructions that implement the GUI from the secondary programming language into GUI instructions within the translated form of the job flow definition in the primary programming language; and

in response to a change having been made to the translated form of the job flow definition stored within the first transfer area, the processor is caused to reverse-translate the GUI instructions within the changed translated form of the job flow definition into the secondary language in a corresponding portion of the executable instructions of the reverse-translated form of the job flow definition.

15. The computer-program product of claim 12, wherein the processor is caused, in response to a determination that there is a lack of a match for at least one comparison of intermediate representations, to perform operations comprising:

generate a directed acyclic graph (DAG) that depicts the lack of a match for the at least one comparison; and

transmit the DAG as the indication of a lack of match to the remote device.

16. The computer-program product of claim 12, wherein:

the first transfer area and the second transfer area are used cooperatively to store and exchange objects as part of collaborative development of a set of objects of the job flow;

the processor receives an indication that the job flow definition has been committed to become part of a set of objects required to perform the job flow; and

in response to the receipt of the indication, the processor cooperates with the remote device to receive the job flow definition in an exchange of objects.

17. The computer-program product of claim 12, wherein the processor is caused to perform operations comprising:

receive, at the processor, and from the remote device via the network, security credentials from the remote device as the remote device logs into the federated area as a user;

analyze the security credentials to determine whether the remote device is authorized to log into the federated area; and

in response to a determination that the remote device is authorized to log into the federated area, the processor grants access to the federated area to the remote device to enable receipt of the job flow definition.

18. The computer-program product of claim 11, the processor is caused to perform operations comprising:

use the set of flow task identifiers retrieved from the job flow definition to search the at least one federated area for at least one task routine to perform each task of the set of tasks; and

in response to a lack of a task routine being stored within the one or more federated for at least one task of the set of tasks, perform operations comprising;

generate a directed acyclic graph (DAG) of the job flow definition that identifies the at least one task; and

transmit the DAG to the remote device.

19. The computer-program product of claim 11, wherein the translation of the portion of the executable instructions of the job flow definition that defines the interface for each task routine of the set of task routines into the primary programming language comprises translating the intermediate expression for the definition of the interface for each task routine into executable instructions in the primary programming language.

20. The computer-program product of claim 11, wherein:

the intermediate expression comprises executable instructions generated in an intermediate programming language; and

one of the primary programming language, the secondary programming language and the intermediate programming language is selected from a group consisting of;

SAS programming language;

Python;

JSON;

Pascal;

Fortran;

BASIC;

C;

C++;

R; and

CUDA.

21. A computer-implemented method comprising:

receiving, by a processor, and from a remote device via a network, a job flow definition to be stored in a federated area of at least one federated area, wherein;

the job flow definition defines a job flow as a set of tasks to be performed by execution of a corresponding set of task routines to perform the job flow;

the job flow definition employs a set of flow task identifiers to identify the set of tasks; and

the at least one federated area is maintained within at least one storage device to store the job flow definition, multiple task routines and multiple data sets as objects;

retrieving the set of flow task identifiers from the job flow definition;

for each retrieved flow task identifier, retrieving, from among the multiple task routines, a most recent version of a task routine of the set of task routines that performs the corresponding task of the set of tasks when executed;

translating, by the processor, a portion of executable instructions within each retrieved task routine of the set of task routines that implements an interface by which a data set is accepted as an input or is output during execution of the task routine into an intermediate representation;

analyzing, by the processor, executable instructions of the job flow definition to determine whether the executable instructions of the job flow definition are written in a primary programming language;

translating, by the processor, a portion of the executable instructions within the job flow definition that defines the interface for each task routine of the set of task routines into an intermediate representation;

comparing, by the processor, each intermediate representation generated from one of the retrieved task routines to the corresponding intermediate representation generated from the job flow definition to determine if there is a match;

in response to a determination that there is a match for each comparison of intermediate representations, and in response to a determination that the executable instructions of the job flow definition are written in a secondary programming language, performing operations comprising;

translating, by the processor, the portion of the executable instructions of the job flow definition that defines the interface for each task routine of the set of task routines into the primary programming language to generate a translated form of the job flow definition; and

storing the translated form of the job flow definition within the federated area;

or

in response to a determination that there is a lack of a match for at least one comparison of intermediate representations, transmitting, from the processor, an indication of a lack of match to the remote device.

22. The computer-implemented method of claim 21, comprising:

maintaining a first transfer area within the federated area;

cooperating, by the processor, with the remote device via the network to exchange objects via the network to synchronize objects between the first transfer area and a second transfer area maintained by the remote device;

cooperating, by the processor, with the remote device to receive the job flow definition in an exchange of objects via the network to synchronize the objects between the first transfer area and the second transfer area in response to the job flow definition having been stored within the second transfer area or in response to the a more recent version of the job flow definition having been stored within the second transfer area; and

in response to a determination that there is a match for each comparison of intermediate representations, and in response to a determination that the executable instructions of the job flow definition are written in a secondary programming language, store the translated form of the job flow definition within the first transfer area.

23. The computer-implemented method of claim 22, comprising, in response to a change having been made to the translated form of the job flow definition stored within the first transfer area, performing operations comprising:

reverse-translating, by the processor, the portion of the executable instructions of the changed translated form of the job flow definition that defines the interface for each task routine of the set of task routines from the primary programming language into the secondary programming language to generate a reverse-translated form of the job flow definition; and

cooperating, by the processor, with the remote device via the network to transmit the reverse-translated form of the job flow definition to the remote device in an exchange of objects via the network to synchronize the objects between the first transfer area and the second transfer area.

24. The computer-implemented method of claim 23, wherein:

the executable instructions of the job flow definition include a portion of executable instructions to implement a graphical user interface (GUI) when executed;

the method comprises, in response to a determination that there is a match for each comparison of intermediate representations, and in response to a determination that the executable instructions of the job flow definition are written in a secondary programming language, translating, by the processor, the portion of the executable instructions that implement the GUI from the secondary programming language into GUI instructions within the translated form of the job flow definition in the primary programming language; and

the method comprises, in response to a change having been made to the translated form of the job flow definition stored within the first transfer area, reverse-translating, by the processor, the GUI instructions within the changed translated form of the job flow definition into the secondary language in a corresponding portion of the executable instructions of the reverse-translated form of the job flow definition.

25. The computer-implemented method of claim 22, comprising, in response to a determination that there is a lack of a match for at least one comparison of intermediate representations, performing operations comprising:

generating, by the processor, a directed acyclic graph (DAG) that depicts the lack of a match for the at least one comparison; and

transmitting, from the processor, the DAG as the indication of a lack of match to the remote device.

26. The computer-implemented method of claim 22, wherein:

the first transfer area and the second transfer area are used cooperatively to store and exchange objects as part of collaborative development of a set of objects of the job flow;

the processor receives an indication that the job flow definition has been committed to become part of a set of objects required to perform the job flow; and

the method comprises, in response to the receipt of the indication, cooperating, by the processor, with the remote device to receive the job flow definition in an exchange of objects.

27. The computer-implemented method of claim 22, comprising:

receiving, at the processor, and from the remote device via the network, security credentials from the remote device as the remote device logs into the federated area as a user;

analyzing, by the processor, the security credentials to determine whether the remote device is authorized to log into the federated area; and

in response to a determination that the remote device is authorized to log into the federated area, granting access to the federated area to the remote device to enable receipt of the job flow definition.

28. The computer-implemented method of claim 21, comprising:

using the set of flow task identifiers retrieved from the job flow definition to search the at least one federated area for at least one task routine to perform each task of the set of tasks; and

in response to a lack of a task routine being stored within the one or more federated for at least one task of the set of tasks, performing operations comprising;

generating, by the processor, a directed acyclic graph (DAG) of the job flow definition that identifies the at least one task; and

transmitting, from the processor, the DAG to the remote device.

29. The computer-implemented method of claim 21, wherein the translation of the portion of the executable instructions of the job flow definition that defines the interface for each task routine of the set of task routines into the primary programming language comprises translating the intermediate expression for the definition of the interface for each task routine into executable instructions in the primary programming language.

30. The computer-implemented method of claim 21, wherein:

the intermediate expression comprises executable instructions generated in an intermediate programming language; and

one of the primary programming language, the secondary programming language and the intermediate programming language is selected from a group consisting of;

SAS programming language;

Python;

JSON;

Pascal;

Fortran;

BASIC;

C;

C++;

R; and

CUDA.