Distributed neuromorphic processing performance accountability

• US 10,338,968 B2
• Filed: 07/19/2018
• Issued: 07/02/2019
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Bequet, Henry Gabriel Victor, Chen, III, Huina, Du, Juan
• Primary Examiner(s): Patel, Hiren P
• Application Number:
  US16/039,863
• Publication Number:
  US 20190012403A1
• Time in Patent Office: 348 Days
• CPC Class Codes: G06F 16/9014 hash tables
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

An apparatus includes a processor to: receive a request to repeat an earlier performance of a first job flow described in a job flow definition; analyze the job flow definition to determine whether the first job flow uses a neural network; in response to a determination that the first job flow uses a neural network, analyze an object associated with the first job flow to determine whether the neural network was trained using training data from a second job flow that does not use a neural network; and in response to a determination that such training data was so used, repeat the earlier performance of the first job flow, perform the second job flow with the same input data values as used in the repeated performance of the first job flow, and analyze corresponding output data values of both performances to determine a degree of accuracy of the neural network.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

receive, at a portal, and from a remote device via a network, a request to repeat an earlier performance, described in a first instance log of multiple instance logs stored in one or more federated areas, of a first job flow defined in a first job flow definition of multiple job flow definitions stored in the one or more federated areas, or to provide objects to the remote device to enable the remote device to repeat the earlier performance, wherein;

the portal is provided on the network to control access by the remote device to the one or more federated areas via the network;

the one or more federated areas are maintained within one or more storage devices to store at least the multiple job flow definitions and the multiple instance logs; and

the request specifies a first instance log identifier of the first instance log;

use the first instance log identifier to retrieve the first instance log from among the multiple instance logs stored in the one or more federated areas, wherein the first instance log comprises a first job flow identifier of the first job flow definition, a task routine identifier for each task routine used to perform a task specified in the first job flow definition, and a data object identifier for each data object associated with the earlier performance of the first job flow;

analyze the first job flow definition to determine whether performances of the first job flow comprise use of a neural network;

in response to a determination that performances of the first job flow do comprise use of the neural network, analyze an object associated with the first job flow to determine whether the neural network was trained to perform an analytical function using a training data set derived from at least one performance of a second job flow defined by a second job flow definition stored in the one or more federated areas, wherein;

the object associated with the first job flow comprises at least one of the first job flow definition, the first instance log, or a task routine executed during the earlier performance of the first job flow; and

performances of the second job flow comprise performances of the analytical function in a manner that does not use any neural network; and

in response to the request comprising a request to repeat the earlier performance, in response to a determination that performances of the first job flow do comprise use of the neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, the processor is caused to perform operations comprising;

repeat the earlier performance of the first job flow with one or more data sets associated with the earlier performance of the first job flow, wherein the repetition of the earlier performance of the first job flow comprises execution, by the processor, of each task routine identified by a task routine identifier in the first instance log;

perform the second job flow with the one or more data sets associated with the earlier performance of the first job flow, wherein the performance of the second job flow comprises execution, by the processor, of a most recent version of a task routine to perform each task identified by a flow task identifier in the second job flow definition;

analyze an output of the repetition of the earlier performance of the first job flow relative to a corresponding output of the performance of the second job flow to determine a degree of accuracy of the first job flow in performing the analytical function relative to a predetermined threshold of accuracy to determine whether the second job flow is to be used in place of the first job flow to perform the analytical function; and

transmit at least the output of the repetition of the earlier performance of the first job flow and an indication of the degree of accuracy or the results of the comparison to the requesting device.

2. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

use the first job flow identifier within the first instance log to retrieve the first job flow definition from the one or more federated areas;

for each task routine executed during the earlier performance of the first job flow to perform a corresponding task specified in the first job flow definition, use the corresponding task routine identifier within the first instance log to retrieve the task routine from the one or more federated areas; and

for each data object employed as an input to the earlier performance of the first job flow, use the corresponding data object identifier within the first instance log to retrieve the data object from the one or more federated areas.

3. The apparatus of claim 2, wherein the processor is caused to, in response to the request comprising a request to provide objects to the remote device to enable the remote device to repeat the earlier performance, transmit, to the requesting device, at least the first job flow definition, each task routine executed during the earlier performance of the first job flow, and each data object employed as an input to the earlier performance of the first job flow.

4. The apparatus of claim 2, wherein the processor is caused to, in response to the request comprising a request to repeat the earlier performance of the first job flow, perform operations comprising:

for each data object generated during the earlier performance of the first job flow, use the corresponding data object identifier within the first instance log to retrieve the data object from the one or more federated areas, and analyze a corresponding data object generated during the repetition of the earlier performance relative to a corresponding data object generated during the earlier performance; and

transmit, to the requesting device, an indication of results of the analysis of at least one data object generated during the repetition of the earlier performance relative to at least one corresponding data object generated during the earlier performance.

5. The apparatus of claim 2, wherein the processor is caused, in response to the determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, to perform operations comprising:

use a second job flow identifier provided within the selected object associated with the first job flow to retrieve the second job flow definition from the one or more federated areas;

for each task specified by a corresponding flow task identifier in the second job flow definition, use the corresponding flow task identifier to retrieve a most recent version of a corresponding task routine from the one or more federated areas; and

in response to the request comprising a request to provide objects to the remote device to enable the remote device to repeat the earlier performance, transmit, to the requesting device, at least the second job flow definition and the most recent version of task routine corresponding to each task specified in the second job flow definition.

6. The apparatus of claim 2, wherein the processor is caused, in response to the determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, to perform operations comprising:

use a second instance log identifier provided within the selected object associated with the first job flow to retrieve a second instance log that describes the at least one performance of the second job flow from the one or more federated areas;

use a second job flow identifier provided within the second instance log to retrieve the second job flow definition from the one or more federated areas;

for each task routine executed during the at least one performance of the second job flow to perform a corresponding task specified in the second job flow definition, use the corresponding task routine identifier within the second instance log to retrieve the task routine from the one or more federated areas; and

in response to the request comprising a request to provide objects to the remote device to enable the remote device to repeat the earlier performance, transmit, to the requesting device, at least the second job flow definition and each task routine executed during the at least one performance of the second job flow.

7. The apparatus of claim 6, wherein the processor is caused, in response to the request comprising a request to repeat the earlier performance, in response to a determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, to perform operations comprising:

for each data object associated with the at least one performance of the second job flow and that became part of the training data set used to train the neural network of the first job flow, use the corresponding data object identifier within the second instance log to retrieve the data object from the one or more federated areas;

perform a regression analysis of the training data set to identify one or more characteristics of the training data set; and

transmit an indication of the identified one or more characteristics to the requesting device.

8. The apparatus of claim 1, wherein the processor is caused, in response to the determination that performances of the first job flow do comprise use of a neural network, to perform operations comprising:

perform a regression analysis with a first one of the one or more data sets associated with the earlier performance of the first job flow that was employed as an input to the earlier performance, and with a second one of the one or more data sets associated with the earlier performance that was generated during the earlier performance to identify one or more characteristics of the combination of the first and second ones of the one or more data sets; and

transmit an indication of the identified one or more characteristics to the requesting device.

9. The apparatus of claim 1, wherein:

at least one data set associated with the earlier performance of the first job flow comprises neural network configuration data that comprises at least one of a hyperparameter or a trained parameter; and

in response to the determination that performances of the first job flow do comprise use of a neural network, the processor is caused to transmit the neural network configuration data to the requesting device.

10. The apparatus of claim 1, wherein the processor is caused, in response to a determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network of the first job flow was trained using the training data set derived from at least one performance of the second job flow, to perform operations comprising:

use a third job flow identifier provided within the selected object associated with the first job flow to retrieve, from the one or more federated areas, a third job flow definition that defines a third job flow to test the neural network or a third instance log identifier that identifies an instance log that describes a performance of the third job flow to test the neural network;

retrieve, from the one or more storage devices, data objects associated with the performance of the third job flow and that became part of a testing data set used to test the neural network; and

transmit, to the requesting device, the testing data set or an indication of one or more characteristics of the testing set that are identified in a regression analysis of the testing data set.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

receive, at a portal, and from a remote device via a network, a request to repeat an earlier performance, described in a first instance log of multiple instance logs stored in one or more federated areas, of a first job flow defined in a first job flow definition of multiple job flow definitions stored in the one or more federated areas, or to provide objects to the remote device to enable the remote device to repeat the earlier performance, wherein;

the portal is provided on the network to control access by the remote device to the one or more federated areas via the network;

the one or more federated areas are maintained within one or more storage devices to store at least the multiple job flow definitions and the multiple instance logs; and

the request specifies a first instance log identifier of the first instance log;

use the first instance log identifier to retrieve the first instance log from among the multiple instance logs stored in the one or more federated areas, wherein the first instance log comprises a first job flow identifier of the first job flow definition, a task routine identifier for each task routine used to perform a task specified in the first job flow definition, and a data object identifier for each data object associated with the earlier performance of the first job flow;

analyze the first job flow definition to determine whether performances of the first job flow comprise use of a neural network;

in response to a determination that performances of the first job flow do comprise use of the neural network, analyze an object associated with the first job flow to determine whether the neural network was trained to perform an analytical function using a training data set derived from at least one performance of a second job flow defined by a second job flow definition stored in the one or more federated areas, wherein;

the object associated with the first job flow comprises at least one of the first job flow definition, the first instance log, or a task routine executed during the earlier performance of the first job flow; and

performances of the second job flow comprise performances of the analytical function in a manner that does not use any neural network; and

in response to the request comprising a request to repeat the earlier performance, in response to a determination that performances of the first job flow do comprise use of the neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, the processor is caused to perform operations comprising;

repeat the earlier performance of the first job flow with one or more data sets associated with the earlier performance of the first job flow, wherein the repetition of the earlier performance of the first job flow comprises execution, by the processor, of each task routine identified by a task routine identifier in the first instance log;

perform the second job flow with the one or more data sets associated with the earlier performance of the first job flow, wherein the performance of the second job flow comprises execution, by the processor, of a most recent version of a task routine to perform each task identified by a flow task identifier in the second job flow definition;

analyze an output of the repetition of the earlier performance of the first job flow relative to a corresponding output of the performance of the second job flow to determine a degree of accuracy of the first job flow in performing the analytical function relative to a predetermined threshold of accuracy to determine whether the second job flow is to be used in place of the first job flow to perform the analytical function; and

transmit at least the output of the repetition of the earlier performance of the first job flow and an indication of the degree of accuracy or the results of the comparison to the requesting device.

12. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

use the first job flow identifier within the first instance log to retrieve the first job flow definition from the one or more federated areas;

for each task routine executed during the earlier performance of the first job flow to perform a corresponding task specified in the first job flow definition, use the corresponding task routine identifier within the first instance log to retrieve the task routine from the one or more federated areas; and

for each data object employed as an input to the earlier performance of the first job flow, use the corresponding data object identifier within the first instance log to retrieve the data object from the one or more federated areas.

13. The computer-program product of claim 12, wherein the processor is caused to, in response to the request comprising a request to provide objects to the remote device to enable the remote device to repeat the earlier performance, transmit, to the requesting device, at least the first job flow definition, each task routine executed during the earlier performance of the first job flow, and each data object employed as an input to the earlier performance of the first job flow.

14. The computer-program product of claim 12, wherein the processor is caused to, in response to the request comprising a request to repeat the earlier performance of the first job flow, perform operations comprising:

for each data object generated during the earlier performance of the first job flow, use the corresponding data object identifier within the first instance log to retrieve the data object from the one or more federated areas, and analyze a corresponding data object generated during the repetition of the earlier performance relative to a corresponding data object generated during the earlier performance; and

transmit, to the requesting device, an indication of results of the analysis of at least one data object generated during the repetition of the earlier performance relative to at least one corresponding data object generated during the earlier performance.

15. The computer-program product of claim 12, wherein the processor is caused, in response to the determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, to perform operations comprising:

use a second job flow identifier provided within the selected object associated with the first job flow to retrieve the second job flow definition from the one or more federated areas;

for each task specified by a corresponding flow task identifier in the second job flow definition, use the corresponding flow task identifier to retrieve a most recent version of a corresponding task routine from the one or more federated areas; and

in response to the request comprising a request to provide objects to the remote device to enable the remote device to repeat the earlier performance, transmit, to the requesting device, at least the second job flow definition and the most recent version of task routine corresponding to each task specified in the second job flow definition.

16. The computer-program product of claim 12, wherein the processor is caused, in response to the determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, to perform operations comprising:

use a second instance log identifier provided within the selected object associated with the first job flow to retrieve a second instance log that describes the at least one performance of the second job flow from the one or more federated areas;

use a second job flow identifier provided within the second instance log to retrieve the second job flow definition from the one or more federated areas;

for each task routine executed during the at least one performance of the second job flow to perform a corresponding task specified in the second job flow definition, use the corresponding task routine identifier within the second instance log to retrieve the task routine from the one or more federated areas; and

in response to the request comprising a request to provide objects to the remote device to enable the remote device to repeat the earlier performance, transmit, to the requesting device, at least the second job flow definition and each task routine executed during the at least one performance of the second job flow.

17. The computer-program product of claim 16, wherein the processor is caused, in response to the request comprising a request to repeat the earlier performance, in response to a determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, to perform operations comprising:

for each data object associated with the at least one performance of the second job flow and that became part of the training data set used to train the neural network of the first job flow, use the corresponding data object identifier within the second instance log to retrieve the data object from the one or more federated areas;

perform a regression analysis of the training data set to identify one or more characteristics of the training data set; and

transmit an indication of the identified one or more characteristics to the requesting device.

18. The computer-program product of claim 11, wherein the processor is caused, in response to the determination that performances of the first job flow do comprise use of a neural network, to perform operations comprising:

perform a regression analysis with a first one of the one or more data sets associated with the earlier performance of the first job flow that was employed as an input to the earlier performance, and with a second one of the one or more data sets associated with the earlier performance that was generated during the earlier performance to identify one or more characteristics of the combination of the first and second ones of the one or more data sets; and

transmit an indication of the identified one or more characteristics to the requesting device.

19. The computer-program product of claim 11, wherein:

at least one data set associated with the earlier performance of the first job flow comprises neural network configuration data that comprises at least one of a hyperparameter or a trained parameter; and

in response to the determination that performances of the first job flow do comprise use of a neural network, the processor is caused to transmit the neural network configuration data to the requesting device.

20. The computer-program product of claim 11, wherein the processor is caused, in response to a determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network of the first job flow was trained using the training data set derived from at least one performance of the second job flow, to perform operations comprising:

use a third job flow identifier provided within the selected object associated with the first job flow to retrieve, from the one or more federated areas, a third job flow definition that defines a third job flow to test the neural network or a third instance log identifier that identifies an instance log that describes a performance of the third job flow to test the neural network;

retrieve, from the one or more storage devices, data objects associated with the performance of the third job flow and that became part of a testing data set used to test the neural network; and

transmit, to the requesting device, the testing data set or an indication of one or more characteristics of the testing set that are identified in a regression analysis of the testing data set.

21. A computer-implemented method comprising:

receiving, by a processor at a portal, and from a remote device via a network, a request to repeat an earlier performance, described in a first instance log of multiple instance logs stored in one or more federated areas, of a first job flow defined in a first job flow definition of multiple job flow definitions stored in the one or more federated areas, or to provide objects to the remote device to enable the remote device to repeat the earlier performance, wherein;

the portal is provided on the network to control access by the remote device to the one or more federated areas via the network;

the one or more federated areas are maintained within one or more storage devices to store at least the multiple job flow definitions and the multiple instance logs; and

the request specifies a first instance log identifier of the first instance log;

using the first instance log identifier to retrieve the first instance log from among the multiple instance logs stored in the one or more federated areas, wherein the first instance log comprises a first job flow identifier of the first job flow definition, a task routine identifier for each task routine used to perform a task specified in the first job flow definition, and a data object identifier for each data object associated with the earlier performance of the first job flow;

analyzing, by the processor, the first job flow definition to determine whether performances of the first job flow comprise use of a neural network;

in response to a determination that performances of the first job flow do comprise use of the neural network, analyzing, by the processor, an object associated with the first job flow to determine whether the neural network was trained to perform an analytical function using a training data set derived from at least one performance of a second job flow defined by a second job flow definition stored in the one or more federated areas, wherein;

the object associated with the first job flow comprises at least one of the first job flow definition, the first instance log, or a task routine executed during the earlier performance of the first job flow; and

performances of the second job flow comprise performances of the analytical function in a manner that does not use any neural network;

or

in response to the request comprising a request to repeat the earlier performance, in response to a determination that performances of the first job flow do comprise use of the neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, performing operations comprising;

repeating, by the processor, the earlier performance of the first job flow with one or more data sets associated with the earlier performance of the first job flow, wherein the repetition of the earlier performance of the first job flow comprises execution, by the processor, of each task routine identified by a task routine identifier in the first instance log;

performing, by the processor, the second job flow with the one or more data sets associated with the earlier performance of the first job flow, wherein the performance of the second job flow comprises execution, by the processor, of a most recent version of a task routine to perform each task identified by a flow task identifier in the second job flow definition;

analyzing, by the processor, an output of the repetition of the earlier performance of the first job flow relative to a corresponding output of the performance of the second job flow to determine a degree of accuracy of the first job flow in performing the analytical function relative to a predetermined threshold of accuracy to determine whether the second job flow is to be used in place of the first job flow to perform the analytical function; and

transmitting, from the processor, at least the output of the repetition of the earlier performance of the first job flow and an indication of the degree of accuracy or the results of the comparison to the requesting device.

22. The computer-implemented method of claim 21, comprising:

using the first job flow identifier within the first instance log to retrieve the first job flow definition from the one or more federated areas;

for each task routine executed during the earlier performance of the first job flow to perform a corresponding task specified in the first job flow definition, using the corresponding task routine identifier within the first instance log to retrieve the task routine from the one or more federated areas; and

for each data object employed as an input to the earlier performance of the first job flow, using the corresponding data object identifier within the first instance log to retrieve the data object from the one or more federated areas.

23. The computer-implemented method of claim 22, comprising, in response to the request comprising a request to provide objects to the remote device to enable the remote device to repeat the earlier performance, transmitting, from the processor and to the requesting device, at least the first job flow definition, each task routine executed during the earlier performance of the first job flow, and each data object employed as an input to the earlier performance of the first job flow.

24. The computer-implemented method of claim 22, comprising, in response to the request comprising a request to repeat the earlier performance of the first job flow, performing operations comprising:

for each data object generated during the earlier performance of the first job flow, using the corresponding data object identifier within the first instance log to retrieve the data object from the one or more federated areas, and analyzing, by the processor, a corresponding data object generated during the repetition of the earlier performance relative to a corresponding data object generated during the earlier performance; and

transmitting, from the processor and to the requesting device, an indication of results of the analysis of at least one data object generated during the repetition of the earlier performance relative to at least one corresponding data object generated during the earlier performance.

25. The computer-implemented method of claim 22, comprising, in response to the determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, performing operations comprising:

using a second job flow identifier provided within the selected object associated with the first job flow to retrieve the second job flow definition from the one or more federated areas;

for each task specified by a corresponding flow task identifier in the second job flow definition, using the corresponding flow task identifier to retrieve a most recent version of a corresponding task routine from the one or more federated areas; and

in response to the request comprising a request to provide objects to the remote device to enable the remote device to repeat the earlier performance, transmitting, from the processor and to the requesting device, at least the second job flow definition and the most recent version of task routine corresponding to each task specified in the second job flow definition.

26. The computer-implemented method of claim 22, comprising, in response to the determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, performing operations comprising:

using a second instance log identifier provided within the selected object associated with the first job flow to retrieve a second instance log that describes the at least one performance of the second job flow from the one or more federated areas;

using a second job flow identifier provided within the second instance log to retrieve the second job flow definition from the one or more federated areas;

for each task routine executed during the at least one performance of the second job flow to perform a corresponding task specified in the second job flow definition, using the corresponding task routine identifier within the second instance log to retrieve the task routine from the one or more federated areas; and

in response to the request comprising a request to provide objects to the remote device to enable the remote device to repeat the earlier performance, transmitting, from the processor and to the requesting device, at least the second job flow definition and each task routine executed during the at least one performance of the second job flow.

27. The computer-implemented method of claim 26, comprising, in response to the request comprising a request to repeat the earlier performance, in response to a determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network was trained using the training data set derived from at least one performance of the second job flow, performing operations comprising:

for each data object associated with the at least one performance of the second job flow and that became part of the training data set used to train the neural network of the first job flow, using the corresponding data object identifier within the second instance log to retrieve the data object from the one or more federated areas;

performing, by the processor, a regression analysis of the training data set to identify one or more characteristics of the training data set; and

transmitting, from the processor, an indication of the identified one or more characteristics to the requesting device.

28. The computer-implemented method of claim 21, comprising, in response to the determination that performances of the first job flow do comprise use of a neural network, performing operations comprising:

performing, by the processor, a regression analysis with a first one of the one or more data sets associated with the earlier performance of the first job flow that was employed as an input to the earlier performance, and with a second one of the one or more data sets associated with the earlier performance that was generated during the earlier performance to identify one or more characteristics of the combination of the first and second ones of the one or more data sets; and

transmitting, from the processor, an indication of the identified one or more characteristics to the requesting device.

29. The computer-implemented method of claim 21, wherein:

at least one data set associated with the earlier performance of the first job flow comprises neural network configuration data that comprises at least one of a hyperparameter or a trained parameter; and

the method comprises, in response to the determination that performances of the first job flow do comprise use of a neural network, transmitting the neural network configuration data from the processor and to the requesting device.

30. The computer-implemented method of claim 21, comprising, in response to a determination that performances of the first job flow do comprise use of a neural network, and in response to a determination that the neural network of the first job flow was trained using the training data set derived from at least one performance of the second job flow, performing operations comprising:

using a third job flow identifier provided within the selected object associated with the first job flow to retrieve, from the one or more federated areas, a third job flow definition that defines a third job flow to test the neural network or a third instance log identifier that identifies an instance log that describes a performance of the third job flow to test the neural network;

retrieving, from the one or more storage devices, data objects associated with the performance of the third job flow and that became part of a testing data set used to test the neural network; and

transmitting, from the processor and to the requesting device, the testing data set or an indication of one or more characteristics of the testing set that are identified in a regression analysis of the testing data set.