Handling of data sets during execution of task routines of multiple languages

• US 10,642,896 B2
• Filed: 08/12/2019
• Issued: 05/05/2020
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Bequet, Henry Gabriel Victor, Zhang, Chaowang "Ricky", Stogner, Ronald Earl, Yang, Eric Jian
• Primary Examiner(s): Patel, Hiren P
• Application Number:
  US16/538,734
• Publication Number:
  US 20190370288A1
• Time in Patent Office: 267 Days
• Field of Search: None
• CPC Class Codes: G06F 16/9014 hash tables
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 8/51 Source to source
• CPC Class Codes: G06F 9/46 Multiprogramming arrangements
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An apparatus includes a processor to: receive a request to perform a job flow defined in a job flow definition; retrieve the most recent versions of a set of task routines to perform a set of tasks of the job flow; analyze each interface of each task routine by which a data set is input or output during execution to identify dependencies where two task routines exchange a data set, and are written in different programming languages; execute the set of task routines to perform the job flow; in response to identifying such a dependency, convert the data set from a form supported by the programming language of one of the two task routines and into a form supported by the programming language of the other; store one of the forms within a federated area; and transmit a result report output during the performance to a remote device.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

receive, at the processor and from a remote device, a request to perform a job flow defined in a job flow definition stored in at least one federated area, wherein;

the job flow definition specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;

at least one flow input data set is to be employed as an input to the job flow performance;

at least one mid-flow data set is to be exchanged between at least two of the set of task routines;

at least one result report is to be output during the job flow performance; and

the at least one federated area is maintained within at least one storage device to store the job flow definition, and multiple task routines, data sets and result reports;

retrieve, from among the multiple task routines, a most recent version of each task routine of the set of task routines to perform a corresponding task of the set of tasks when executed;

analyze each interface of each task routine of the set of task routines by which a data set is accepted as an input or is output during execution of the task routine to identify at least one dependency among at least two task routines in which a first task routine of the at least two task routines outputs a mid-flow data set that a second task routine of the at least two task routines accepts as an input, and in which the first task routine and the second task routine include executable instructions written in different programming languages;

execute the executable instructions of the set of task routines to perform the set of tasks to thereby perform the job flow;

in response to having identified at least one dependency in which the first task routine outputs a mid-flow data set that the second task routine accepts as an input, and in which the first task routine and the second task routine include executable instructions written in different programming languages, perform operations comprising;

instantiate a shared memory space to store at least one mid-flow data set during the job flow performance;

for each identified dependency of the at least one identified dependency;

convert the mid-flow data set from a first form supported by the programming language in which the executable instructions of the first task routine are written, and into a second form supported by the programming language in which the executable instructions of the second task routine are written;

store one of the first form and the second form of the mid-flow data set within the at least one federated area as one of the multiple data sets; and

store another of the first form and the second form of the mid-flow data set within the shared memory space as the mid-flow data set is converted;

un-instantiate the shared memory space; and

transmit the at least one result report output during the performance of the job flow to the remote device.

2. The apparatus of claim 1, wherein:

in a dependency of the at least one identified dependency, a selected one of the programming language in which the executable instructions of the first task routine are written and the programming language in which the executable instructions of the second task routine are written is designated as a primary programming language, and the non-selected one is designated as a secondary programming language; and

the multiple data sets and result reports are stored in the at least one federated area in the one of the first form and the second form that is supported by the primary programming language.

3. The apparatus of claim 2, wherein the processor is caused to perform operations comprising:

analyze each interface of each task routine of the set of task routines to identify at least one other dependency among at least two task routines in which a first task routine of the at least two task routines outputs a mid-flow data set that a second task routine of the at least two task routines accepts as an input, and in which the first task routine and the second task routine include executable instructions written in the same programming language; and

in response to having identified at least one other dependency in which the first task routine outputs a mid-flow data set that the second task routine accepts as an input, and in which the first task routine and the second task routine include executable instructions written in the same programming language, perform operations comprising;

for each identified dependency of the at least one other identified dependency;

refrain from converting the mid-flow data set between the first form and the second form;

analyze the form of the mid-flow data set to determine whether it comprises the one of the first form and the second form that is supported by the primary programming language; and

in response to a determination that the form of the mid-flow data set comprises the one of the first form and the second form that is supported by the primary programming language, store the mid-flow data set within the at least one federated area as one of the multiple data sets.

4. The apparatus of claim 2, wherein, in response to having identified at least one dependency in which the first task routine outputs a mid-flow data set that the second task routine accepts as an input, and in which the first task routine and the second task routine include executable instructions written in different programming languages, the processor is caused to perform operations comprising:

for each identified dependency of the at least one identified dependency;

store the one of the first form and the second form of the mid-flow data set within the at least one federated area as one of the multiple data sets based on which of the first form and the second form is supported by the primary programming language; and

store another of the first form and the second form of the mid-flow data set that is supported by the secondary programming language within the shared memory space as the mid-flow data set is converted.

5. The apparatus of claim 2, wherein one of the primary programming language and the secondary programming language is selected from a group consisting of:

SAS programming language;

Python;

JSON;

Pascal;

Fortran;

BASIC;

C;

C++;

R; and

CUDA.

6. The apparatus of claim 1, wherein the conversion of the mid-flow data set for each identified dependency of the at least one identified dependency comprises a conversion selected from a group consisting of:

a change between data types;

a change between byte orderings;

a change between delimiters separating data values;

a change between big Endian and little Endian;

a change between byte widths of data values;

a change in encoding of data values;

a reordering of data values between starting with a highest index value and starting with a lowest index value;

a change between a row-column organization and a column-row organization;

a serialization from structured data to un-structured data;

a de-serialization from unstructured data to structured data;

a serialization from an array to comma-separated variables; and

a de-serialization from comma-separated variables to an array.

7. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

for each task routine of the set of task routines that includes executable instructions written in a first programming language, execute a first runtime interpreter or compiler to execute, by the processor, the executable instructions written in the first programming language; and

for each task routine of the set of task routines that includes executable instructions written in a second programming language, execute a second runtime interpreter or compiler to execute, by the processor, the executable instructions written in the second programming language.

8. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

receive, at the processor, a task routine from another device;

retrieve a flow task identifier from the received task routine that identifies the task that the received task routine performs when the executable instructions of the received task routine are executed;

analyze each task routine of the multiple task routines to identify at least one task routine of the multiple task routines that performs the same task when the executable instructions of the at least one task routine are executed;

in response to identifying at least one other task routine of the multiple task routines that performs the same task, perform operations comprising;

analyze the received task routine to identify the programming language in which the executable instructions are written;

analyze each task routine of the at least one task routine to identify the programming language in which the executable instructions of each are written;

for the received task routine and for each task routine of the at least one task routine, select an intermediate translator based on the programming language in which the executable instructions are written, and translate a portion of the executable instructions that implements the interface into an intermediate representation;

compare the intermediate representations generated from the executable instructions of the received task routine and each task routine of the at least one task routine to determine if there is a match; and

in response to a determination that there is a match, store the received task routine among the multiple task routines in the at least one federated area.

9. The apparatus of claim 8, wherein the processor is caused to, in response to a determination that there is not a match, perform operations comprising:

generate a directed acyclic graph (DAG) that depicts a difference between the interface of the received task routine and the interface of the at least one task routine; and

transmit the DAG to the other device.

10. The apparatus of claim 8, wherein each intermediate representation comprises executable instructions written in an intermediate programming language.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

receive, at the processor and from a remote device, a request to perform a job flow defined in a job flow definition stored in at least one federated area, wherein;

the job flow definition specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;

at least one flow input data set is to be employed as an input to the job flow performance;

at least one mid-flow data set is to be exchanged between at least two of the set of task routines;

at least one result report is to be output during the job flow performance; and

the at least one federated area is maintained within at least one storage device to store the job flow definition, and multiple task routines, data sets and result reports;

retrieve, from among the multiple task routines, a most recent version of each task routine of the set of task routines to perform a corresponding task of the set of tasks when executed;

analyze each interface of each task routine of the set of task routines by which a data set is accepted as an input or is output during execution of the task routine to identify at least one dependency among at least two task routines in which a first task routine of the at least two task routines outputs a mid-flow data set that a second task routine of the at least two task routines accepts as an input, and in which the first task routine and the second task routine include executable instructions written in different programming languages;

execute the executable instructions of the set of task routines to perform the set of tasks to thereby perform the job flow;

in response to having identified at least one dependency in which the first task routine outputs a mid-flow data set that the second task routine accepts as an input, and in which the first task routine and the second task routine include executable instructions written in different programming languages, perform operations comprising;

instantiate a shared memory space to store at least one mid-flow data set during the job flow performance;

for each identified dependency of the at least one identified dependency;

convert the mid-flow data set from a first form supported by the programming language in which the executable instructions of the first task routine are written, and into a second form supported by the programming language in which the executable instructions of the second task routine are written;

store one of the first form and the second form of the mid-flow data set within the at least one federated area as one of the multiple data sets; and

store another of the first form and the second form of the mid-flow data set within the shared memory space as the mid-flow data set is converted;

un-instantiate the shared memory space; and

transmit the at least one result report output during the performance of the job flow to the remote device.

12. The computer-program product of claim 11, wherein:

in a dependency of the at least one identified dependency, a selected one of the programming language in which the executable instructions of the first task routine are written and the programming language in which the executable instructions of the second task routine are written is designated as a primary programming language, and the non-selected one is designated as a secondary programming language; and

the multiple data sets and result reports are stored in the at least one federated area in the one of the first form and the second form that is supported by the primary programming language.

13. The computer-program product of claim 12, wherein the processor is caused to perform operations comprising:

analyze each interface of each task routine of the set of task routines to identify at least one other dependency among at least two task routines in which a first task routine of the at least two task routines outputs a mid-flow data set that a second task routine of the at least two task routines accepts as an input, and in which the first task routine and the second task routine include executable instructions written in the same programming language; and

in response to having identified at least one other dependency in which the first task routine outputs a mid-flow data set that the second task routine accepts as an input, and in which the first task routine and the second task routine include executable instructions written in the same programming language, perform operations comprising;

for each identified dependency of the at least one other identified dependency;

refrain from converting the mid-flow data set between the first form and the second form;

analyze the form of the mid-flow data set to determine whether it comprises the one of the first form and the second form that is supported by the primary programming language; and

in response to a determination that the form of the mid-flow data set comprises the one of the first form and the second form that is supported by the primary programming language, store the mid-flow data set within the at least one federated area as one of the multiple data sets.

14. The computer-program product of claim 12, wherein, in response to having identified at least one dependency in which the first task routine outputs a mid-flow data set that the second task routine accepts as an input, and in which the first task routine and the second task routine include executable instructions written in different programming languages, the processor is caused to perform operations comprising:

for each identified dependency of the at least one identified dependency;

store the one of the first form and the second form of the mid-flow data set within the at least one federated area as one of the multiple data sets based on which of the first form and the second form is supported by the primary programming language; and

store another of the first form and the second form of the mid-flow data set that is supported by the secondary programming language within the shared memory space as the mid-flow data set is converted.

15. The computer-program product of claim 12, wherein one of the primary programming language and the secondary programming language is selected from a group consisting of:

SAS programming language;

Python;

JSON;

Pascal;

Fortran;

BASIC;

C;

C++;

R; and

CUDA.

16. The computer-program product of claim 11, wherein the conversion of the mid-flow data set for each identified dependency of the at least one identified dependency comprises a conversion selected from a group consisting of:

a change between data types;

a change between byte orderings;

a change between delimiters separating data values;

a change between big Endian and little Endian;

a change between byte widths of data values;

a change in encoding of data values;

a reordering of data values between starting with a highest index value and starting with a lowest index value;

a change between a row-column organization and a column-row organization;

a serialization from structured data to un-structured data;

a de-serialization from unstructured data to structured data;

a serialization from an array to comma-separated variables; and

a de-serialization from comma-separated variables to an array.

17. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

for each task routine of the set of task routines that includes executable instructions written in a first programming language, execute a first runtime interpreter or compiler to execute, by the processor, the executable instructions written in the first programming language; and

for each task routine of the set of task routines that includes executable instructions written in a second programming language, execute a second runtime interpreter or compiler to execute, by the processor, the executable instructions written in the second programming language.

18. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

receive, at the processor, a task routine from another device;

retrieve a flow task identifier from the received task routine that identifies the task that the received task routine performs when the executable instructions of the received task routine are executed;

analyze each task routine of the multiple task routines to identify at least one task routine of the multiple task routines that performs the same task when the executable instructions of the at least one task routine are executed;

in response to identifying at least one other task routine of the multiple task routines that performs the same task, perform operations comprising;

analyze the received task routine to identify the programming language in which the executable instructions are written;

analyze each task routine of the at least one task routine to identify the programming language in which the executable instructions of each are written;

for the received task routine and for each task routine of the at least one task routine, select an intermediate translator based on the programming language in which the executable instructions are written, and translate a portion of the executable instructions that implements the interface into an intermediate representation;

compare the intermediate representations generated from the executable instructions of the received task routine and each task routine of the at least one task routine to determine if there is a match; and

in response to a determination that there is a match, store the received task routine among the multiple task routines in the at least one federated area.

19. The computer-program product of claim 18, wherein the processor is caused to, in response to a determination that there is not a match, perform operations comprising:

generate a directed acyclic graph (DAG) that depicts a difference between the interface of the received task routine and the interface of the at least one task routine; and

transmit the DAG to the other device.

20. The computer-program product of claim 18, wherein each intermediate representation comprises executable instructions written in an intermediate programming language.

21. A computer-implemented method comprising:

receiving, by a processor, and from a remote device, a request to perform a job flow defined in a job flow definition stored in at least one federated area, wherein;

the job flow definition specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;

at least one flow input data set is to be employed as an input to the job flow performance;

at least one mid-flow data set is to be exchanged between at least two of the set of task routines;

at least one result report is to be output during the job flow performance; and

the at least one federated area is maintained within at least one storage device to store the job flow definition, and multiple task routines, data sets and result reports;

retrieving, from among the multiple task routines, a most recent version of each task routine of the set of task routines to perform a corresponding task of the set of tasks when executed;

analyzing, by the processor, each interface of each task routine of the set of task routines by which a data set is accepted as an input or is output during execution of the task routine to identify at least one dependency among at least two task routines in which a first task routine of the at least two task routines outputs a mid-flow data set that a second task routine of the at least two task routines accepts as an input, and in which the first task routine and the second task routine include executable instructions written in different programming languages;

executing, by the processor the executable instructions of the set of task routines to perform the set of tasks to thereby perform the job flow;

in response to having identified at least one dependency in which the first task routine outputs a mid-flow data set that the second task routine accepts as an input, and in which the first task routine and the second task routine include executable instructions written in different programming languages, performing operations comprising;

instantiating, by the processor, a shared memory space to store at least one mid-flow data set during the job flow performance;

for each identified dependency of the at least one identified dependency;

converting, by the processor, the mid-flow data set from a first form supported by the programming language in which the executable instructions of the first task routine are written, and into a second form supported by the programming language in which the executable instructions of the second task routine are written;

storing one of the first form and the second form of the mid-flow data set within the at least one federated area as one of the multiple data sets; and

storing another of the first form and the second form of the mid-flow data set within the shared memory space as the mid-flow data set is converted;

un-instantiating, by the processor, the shared memory space; and

transmitting, from the processor, the at least one result report output during the performance of the job flow to the remote device.

22. The computer-implemented method of claim 21, wherein:

in a dependency of the at least one identified dependency, a selected one of the programming language in which the executable instructions of the first task routine are written and the programming language in which the executable instructions of the second task routine are written is designated as a primary programming language, and the non-selected one is designated as a secondary programming language; and

the multiple data sets and result reports are stored in the at least one federated area in the one of the first form and the second form that is supported by the primary programming language.

23. The computer-implemented method of claim 22, comprising:

analyzing, by the processor, each interface of each task routine of the set of task routines to identify at least one other dependency among at least two task routines in which a first task routine of the at least two task routines outputs a mid-flow data set that a second task routine of the at least two task routines accepts as an input, and in which the first task routine and the second task routine include executable instructions written in the same programming language; and

in response to having identified at least one other dependency in which the first task routine outputs a mid-flow data set that the second task routine accepts as an input, and in which the first task routine and the second task routine include executable instructions written in the same programming language, performing operations comprising;

for each identified dependency of the at least one other identified dependency;

refraining from converting the mid-flow data set between the first form and the second form;

analyzing, by the processor, the form of the mid-flow data set to determine whether it comprises the one of the first form and the second form that is supported by the primary programming language; and

in response to a determination that the form of the mid-flow data set comprises the one of the first form and the second form that is supported by the primary programming language, storing the mid-flow data set within the at least one federated area as one of the multiple data sets.

24. The computer-implemented method of claim 22, comprising, in response to having identified at least one dependency in which the first task routine outputs a mid-flow data set that the second task routine accepts as an input, and in which the first task routine and the second task routine include executable instructions written in different programming languages, performing operations comprising:

for each identified dependency of the at least one identified dependency;

storing the one of the first form and the second form of the mid-flow data set within the at least one federated area as one of the multiple data sets based on which of the first form and the second form is supported by the primary programming language; and

storing another of the first form and the second form of the mid-flow data set that is supported by the secondary programming language within the shared memory space as the mid-flow data set is converted.

25. The computer-implemented method of claim 22, wherein one of the primary programming language and the secondary programming language is selected from a group consisting of:

SAS programming language;

Python;

JSON;

Pascal;

Fortran;

BASIC;

C;

C++;

R; and

CUDA.

26. The computer-implemented method of claim 21, wherein the conversion of the mid-flow data set for each identified dependency of the at least one identified dependency comprises a conversion selected from a group consisting of:

a change between data types;

a change between byte orderings;

a change between delimiters separating data values;

a change between big Endian and little Endian;

a change between byte widths of data values;

a change in encoding of data values;

a reordering of data values between starting with a highest index value and starting with a lowest index value;

a change between a row-column organization and a column-row organization;

a serialization from structured data to un-structured data;

a de-serialization from unstructured data to structured data;

a serialization from an array to comma-separated variables; and

a de-serialization from comma-separated variables to an array.

27. The computer-implemented method of claim 21, comprising:

for each task routine of the set of task routines that includes executable instructions written in a first programming language, executing, by the processor, a first runtime interpreter or compiler to execute, by the processor, the executable instructions written in the first programming language; and

for each task routine of the set of task routines that includes executable instructions written in a second programming language, executing, by the processor, a second runtime interpreter or compiler to execute, by the processor, the executable instructions written in the second programming language.

28. The computer-implemented method of claim 21, comprising:

receiving, at the processor, a task routine from another device;

retrieving a flow task identifier from the received task routine that identifies the task that the received task routine performs when the executable instructions of the received task routine are executed;

analyzing, by the processor, each task routine of the multiple task routines to identify at least one task routine of the multiple task routines that performs the same task when the executable instructions of the at least one task routine are executed;

in response to identifying at least one other task routine of the multiple task routines that performs the same task, performing operations comprising;

analyzing, by the processor, the received task routine to identify the programming language in which the executable instructions are written;

analyzing, by the processor, each task routine of the at least one task routine to identify the programming language in which the executable instructions of each are written;

for the received task routine and for each task routine of the at least one task routine, selecting, by the processor, an intermediate translator based on the programming language in which the executable instructions are written, and translate a portion of the executable instructions that implements the interface into an intermediate representation;

comparing, by the processor, the intermediate representations generated from the executable instructions of the received task routine and each task routine of the at least one task routine to determine if there is a match; and

in response to a determination that there is a match, storing the received task routine among the multiple task routines in the at least one federated area.

29. The computer-implemented method of claim 28, comprising, in response to a determination that there is not a match, performing operations comprising:

generating, by the processor, a directed acyclic graph (DAG) that depicts a difference between the interface of the received task routine and the interface of the at least one task routine; and

transmitting, from the processor, the DAG to the other device.

30. The computer-implemented method of claim 28, wherein each intermediate representation comprises executable instructions written in an intermediate programming language.