Distributed data set storage and analysis reproducibility

• US 10,078,710 B2
• Filed: 12/22/2017
• Issued: 09/18/2018
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Bequet, Henry Gabriel Victor, Arfaoui, Kais, Stogner, Ronald Earl
• Primary Examiner(s): Patel, Hiren
• Application Number:
  US15/851,869
• Publication Number:
  US 20180136983A1
• Time in Patent Office: 270 Days
• Field of Search: None
• CPC Class Codes: G06F 16/9014 hash tables
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An apparatus includes a processor and a storage storing instructions causing the processor to: maintain a federated area; receive a request to perform a job flow with a data set from a remote device; retrieve a job flow definition specifying the tasks of the job flow from the federated area; determine whether there is an instance log in the federated area generated by a previous performance of the job flow with the data set; in response to there being such an instance log, retrieve the version specified in the instance log of each task routine for each task from the federated area; in response to there being no such instance log, retrieve the most recent version of each task routine; perform the job flow with the retrieved versions of the task routines and the data set to generate a result report; and provide the result report to the remote device.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

receive, at a portal, and from a remote device via a the network, a first request to execute at least one task routine stored in a federated area to perform at least one corresponding task of a job flow described in a job flow definition stored in the federated area with at least one data set stored in the federated area, wherein;

the portal is provided on the network to control access by the remote device to the federated area via the network;

the federated area is maintained within one or more storage devices to store multiple data sets, multiple job flow definitions, multiple task routines, multiple result reports and multiple instance logs; and

the first request specifies a job flow identifier of the job flow definition and at least one data object identifier of the at least one data set;

use the job flow identifier to retrieve the job flow definition from among the multiple job flow definitions stored in the federated area, wherein the job flow definition comprises a flow task identifier for each task of the at least one task of the job flow and specifies a relative order in which each task of the at least one task is to be performed in the job flow;

use the at least one data object identifier to retrieve the at least one data set from among the multiple data sets stored in the federated area;

use the job flow identifier and the at least one data object identifier as portions of a first index to search for any instance log among the multiple instance logs stored in the federated area to determine whether there is at least one instance log among the multiple instance logs that was generated by a previous performance of the at least one task of the job flow with the at least one data set;

in response to a determination that there is at least one instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, perform operations comprising;

use the first index to retrieve an instance log of the at least one instance log from among the multiple instance logs stored in the federated area, wherein for each task of the at least one task of the job flow, the retrieved instance log comprises a task routine identifier that specifies a version of task routine that was executed during the corresponding previous performance to perform the task; and

for each task of the at least one task of the job flow, use the corresponding task routine identifier of the retrieved instance log to retrieve, from among the multiple task routines stored in the federated area, the version of task routine that was executed during the corresponding previous performance;

in response to a determination that there is no instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, perform operations comprising;

for each task of the at least one task of the job flow, match the corresponding flow task identifier in the job flow definition to a task routine identifier of a most recent version of task routine executable to perform the task that is stored among the multiple task routines stored in the federated area; and

for each task of the at least one task of the job flow, use the task routine identifier of the most recent version of task routine executable to perform the task to retrieve the most recent version of task routine from among the multiple task routines stored in the federated area;

for each task of the at least one task of the job flow, execute the retrieved version of task routine executable to perform the task to generate a new result report and a new instance log;

store the new result report among the multiple result reports in the federated area;

store the new instance log among the multiple instance logs in the federated area; and

provide access to the new result report to the remote device via the portal.

2. The apparatus of claim 1, wherein:

the determination of whether there is at least one instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises a determination, by the processor, of whether there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set; and

in response to a determination that there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set, the processor is caused to perform operations comprising;

provide access to an indication of the more than one instance log and a second request to the remote device to select an instance log from among the more than one instance log via the portal;

receive, at the portal, and from the remote device via the network, an indication of a selected instance log from among the more than one instance log; and

use the first index to retrieve the selected instance log from among the multiple instance logs stored in the federated area as the retrieved instance log.

3. The apparatus of claim 1, wherein:

the determination of whether there is at least one instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises a determination, by the processor, of whether there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set; and

in response to a determination that there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set, the processor is caused to use the first index to retrieve, from among the multiple instance logs stored in the federated area, the most recently generated of the more than one instance log as the retrieved instance log.

4. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

prior to the determination of whether there is an instance log that was generated by a previous performance of the at least one task, determine whether the first request is authorized;

condition performance of the retrieval of the job flow definition, and the search for an instance log that was generated by a previous performance of the at least one task on the determination of whether the first request is authorized; and

in response to a determination that the first request is not authorized, provide an indication of a denial of the first request to the remote device via the portal.

5. The apparatus of claim 1, wherein, in the generation of the new instance log, the processor is caused to perform operations comprising:

take at least a first hash of the at least one data set;

take at least a second hash of the most recent version of a task routine of the at least one task routine;

take a third hash of the new result report; and

concatenate at least the first, second and third hashes to generate a string.

6. The apparatus of claim 5, wherein, in the generation of the new instance log, the processor is caused to generate the new instance log to comprise the string.

7. The apparatus of claim 5, wherein, in the storage of the new instance log among the multiple instance logs within the federated area, the processor is caused to use the string as a second index to a location in the federated area at which the new instance log is stored.

8. The apparatus of claim 1, wherein:

the task routine identifier of each version of each task routine of the multiple task routines stored in the federated area comprises a hash taken the version of the task routine; and

the comparison of the task routine identifier of the most recent version of task routine to the task routine identifier of the version of task routine specified in the instance log for each task comprises a comparison of the corresponding hashes.

9. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

receive, at the portal, and from a source device via the network, a second request to store a task routine among the multiple task routines in the federated area, wherein the task routine comprises a flow task identifier to indicate a corresponding task that is performed when the task routine is executed;

determine whether another task routine is already stored among the multiple task routines in the federated area that comprises the same flow task identifier to indicate that the same corresponding task is performed when the other task routine is executed; and

in response to a determination that another task routine is already stored among the multiple task routines that comprises the same flow task identifier, perform operations comprising;

store the task routine among the multiple task routines; and

store an indication of the storage of more than one task routine comprising the same flow task identifier in the federated area.

10. The apparatus of claim 9, wherein:

the task routine comprises;

an input interface by which the task routine is to receive the at least one data set; and

an output interface; and

in response to a determination that another task routine is already stored among the multiple task routines that comprises the same flow task identifier, perform operations comprising;

compare the input interface to a corresponding input interface of the other task routine;

compare the output interface to a corresponding output interface of the other task routine; and

condition the storage of the task routine and the indication of the storage of more than one task routine on a determination that the input interface matches the corresponding input interface, and a determination that the output interface matches or is a superset of the corresponding output interface.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

receive, at a portal, and from a remote device via a the network, a first request to execute at least one task routine stored in a federated area to perform at least one corresponding task of a job flow described in a job flow definition stored in the federated area with at least one data set stored in the federated area, wherein;

the portal is provided on the network to control access by the remote device to the federated area via the network;

the federated area is maintained within one or more storage devices to store multiple data sets, multiple job flow definitions, multiple task routines, multiple result reports and multiple instance logs; and

the first request specifies a job flow identifier of the job flow definition and at least one data object identifier of the at least one data set;

use the job flow identifier to retrieve the job flow definition from among the multiple job flow definitions stored in the federated area, wherein the job flow definition comprises a flow task identifier for each task of the at least one task of the job flow and specifies a relative order in which each task of the at least one task is to be performed in the job flow;

use the at least one data object identifier to retrieve the at least one data set from among the multiple data sets stored in the federated area;

use the job flow identifier and the at least one data object identifier as portions of a first index to search for any instance log among the multiple instance logs stored in the federated area to determine whether there is at least one instance log among the multiple instance logs that was generated by a previous performance of the at least one task of the job flow with the at least one data set;

in response to a determination that there is at least one instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, perform operations comprising;

use the first index to retrieve an instance log of the at least one instance log from among the multiple instance logs stored in the federated area, wherein for each task of the at least one task of the job flow, the retrieved instance log comprises a task routine identifier that specifies a version of task routine that was executed during the corresponding previous performance to perform the task; and

for each task of the at least one task of the job flow, use the corresponding task routine identifier of the retrieved instance log to retrieve, from among the multiple task routines stored in the federated area, the version of task routine that was executed during the corresponding previous performance;

in response to a determination that there is no instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, perform operations comprising;

for each task of the at least one task of the job flow, match the corresponding flow task identifier in the job flow definition to a task routine identifier of a most recent version of task routine executable to perform the task that is stored among the multiple task routines stored in the federated area; and

for each task of the at least one task of the job flow, use the task routine identifier of the most recent version of task routine executable to perform the task to retrieve the most recent version of task routine from among the multiple task routines stored in the federated area;

for each task of the at least one task of the job flow, execute the retrieved version of task routine executable to perform the task to generate a new result report and a new instance log;

store the new result report among the multiple result reports in the federated area;

store the new instance log among the multiple instance logs in the federated area; and

provide access to the new result report to the remote device via the portal.

12. The computer-program product of claim 11, wherein:

the determination of whether there is at least one instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises a determination, by the processor, of whether there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set; and

in response to a determination that there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set, the processor is caused to perform operations comprising;

provide access to an indication of the more than one instance log and a second request to the remote device to select an instance log from among the more than one instance log via the portal;

receive, at the portal, and from the remote device via the network, an indication of a selected instance log from among the more than one instance log; and

use the first index to retrieve the selected instance log from among the multiple instance logs stored in the federated area as the retrieved instance log.

13. The computer-program product of claim 11, wherein:

the determination of whether there is at least one instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises a determination, by the processor, of whether there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set; and

in response to a determination that there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set, the processor is caused to use the first index to retrieve, from among the multiple instance logs stored in the federated area, the most recently generated of the more than one instance log as the retrieved instance log.

14. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

prior to the determination of whether there is an instance log that was generated by a previous performance of the at least one task, determine whether the first request is authorized;

condition performance of the retrieval of the job flow definition, and the search for an instance log that was generated by a previous performance of the at least one task on the determination of whether the first request is authorized; and

in response to a determination that the first request is not authorized, provide an indication of a denial of the first request to the remote device via the portal.

15. The computer-program product of claim 11, wherein, in the generation of the new instance log, the processor is caused to perform operations comprising:

take at least a first hash of the at least one data set;

take at least a second hash of the most recent version of a task routine of the at least one task routine;

take a third hash of the new result report; and

concatenate at least the first, second and third hashes to generate a string.

16. The computer-program product of claim 15, wherein, in the generation of the new instance log, the processor is caused to generate the new instance log to comprise the string.

17. The computer-program product of claim 15, wherein, in the storage of the new instance log among the multiple instance logs within the federated area, the processor is caused to use the string as a second index to a location in the federated area at which the new instance log is stored.

18. The computer-program product of claim 11, wherein:

the task routine identifier of each version of each task routine of the multiple task routines stored in the federated area comprises a hash taken the version of the task routine; and

the comparison of the task routine identifier of the most recent version of task routine to the task routine identifier of the version of task routine specified in the instance log for each task comprises a comparison of the corresponding hashes.

19. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

receive, at the portal, and from a source device via the network, a second request to store a task routine among the multiple task routines in the federated area, wherein the task routine comprises a flow task identifier to indicate a corresponding task that is performed when the task routine is executed;

determine whether another task routine is already stored among the multiple task routines in the federated area that comprises the same flow task identifier to indicate that the same corresponding task is performed when the other task routine is executed; and

in response to a determination that another task routine is already stored among the multiple task routines that comprises the same flow task identifier, perform operations comprising;

store the task routine among the multiple task routines; and

store an indication of the storage of more than one task routine comprising the same flow task identifier in the federated area.

20. The computer-program product of claim 19, wherein:

the task routine comprises;

an input interface by which the task routine is to receive the at least one data set; and

an output interface; and

in response to a determination that another task routine is already stored among the multiple task routines that comprises the same flow task identifier, perform operations comprising;

compare the input interface to a corresponding input interface of the other task routine;

compare the output interface to a corresponding output interface of the other task routine; and

condition the storage of the task routine and the indication of the storage of more than one task routine on a determination that the input interface matches the corresponding input interface, and a determination that the output interface matches or is a superset of the corresponding output interface.

21. A computer-implemented method comprising:

receiving, at a portal, and from a remote device via a the network, a first request to execute at least one task routine stored in a federated area to perform at least one corresponding task of a job flow described in a job flow definition stored in the federated area with at least one data set stored in the federated area, wherein;

the portal is provided on the network to control access by the remote device to the federated area via the network;

the federated area is maintained within one or more storage devices to store multiple data sets, multiple job flow definitions, multiple task routines, multiple result reports and multiple instance logs; and

the first request specifies a job flow identifier of the job flow definition and at least one data object identifier of the at least one data set;

using, by a processor, the job flow identifier to retrieve the job flow definition from among the multiple job flow definitions stored in the federated area, wherein the job flow definition comprises a flow task identifier for each task of the at least one task of the job flow and specifies a relative order in which each task of the at least one task is to be performed in the job flow;

using, by the processor, the at least one data object identifier to retrieve the at least one data set from among the multiple data sets stored in the federated area;

using, by the processor, the job flow identifier and the at least one data object identifier as portions of a first index to search for any instance log among the multiple instance logs stored in the federated area to determine whether there is at least one instance log among the multiple instance logs that was generated by a previous performance of the at least one task of the job flow with the at least one data set;

in response to a determination that there is at least one instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, performing operations comprising;

using, by the processor, the first index to retrieve an instance log of the at least one instance log from among the multiple instance logs stored in the federated area, wherein for each task of the at least one task of the job flow, the retrieved instance log comprises a task routine identifier that specifies a version of task routine that was executed during the corresponding previous performance to perform the task; and

for each task of the at least one task of the job flow, using, by the processor, the corresponding task routine identifier of the retrieved instance log to retrieve, from among the multiple task routines stored in the federated area, the version of task routine that was executed during the corresponding previous performance;

in response to a determination that there is no instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, performing operations comprising;

for each task of the at least one task of the job flow, matching, by the processor, the corresponding flow task identifier in the job flow definition to a task routine identifier of a most recent version of task routine executable to perform the task that is stored among the multiple task routines stored in the federated area; and

for each task of the at least one task of the job flow, using, by the processor, the task routine identifier of the most recent version of task routine executable to perform the task to retrieve the most recent version of task routine from among the multiple task routines stored in the federated area;

for each task of the at least one task of the job flow, executing, by the processor, the retrieved version of task routine executable to perform the task to generate a new result report and a new instance log;

storing, by the processor, the new result report among the multiple result reports in the federated area;

storing, by the processor, the new instance log among the multiple instance logs in the federated area; and

providing access to the new result report to the remote device via the portal.

22. The computer-implemented method of claim 21, wherein:

determining whether there is at least one instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises determining, by the processor, whether there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set; and

the method comprises, in response to a determination that there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set, performing operations comprising;

providing access to an indication of the more than one instance log and a second request to the remote device to select an instance log from among the more than one instance log via the portal;

receiving, at the portal, and from the remote device via the network, an indication of a selected instance log from among the more than one instance log; and

using, by the processor, the first index to retrieve the selected instance log from among the multiple instance logs stored in the federated area as the retrieved instance log.

23. The computer-implemented method of claim 21, wherein:

determining whether there is at least one instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises determining, by the processor, whether there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set; and

the method comprises, in response to a determination that there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set, using, by the processor, the first index to retrieve, from among the multiple instance logs stored in the federated area, the most recently generated of the more than one instance log as the retrieved instance log.

24. The computer-implemented method of claim 21, comprising:

prior to determining whether there is an instance log that was generated by a previous performance of the at least one task, determining, by the processor, whether the first request is authorized;

conditioning the retrieving of the job flow definition, and the searching for an instance log that was generated by a previous performance of the at least one task on the determining of whether the first request is authorized; and

in response determining that the first request is not authorized, providing an indication of a denial of the first request to the remote device via the portal.

25. The computer-implemented method of claim 21, wherein generating the new instance log comprises:

taking, by the processor, at least a first hash of the at least one data set;

taking, by the processor, at least a second hash of the most recent version of a task routine of the at least one task routine;

taking, by the processor, a third hash of the new result report; and

concatenating, by the processor, at least the first, second and third hashes to generate a string.

26. The computer-implemented method of claim 25, wherein generating the new instance log comprises generating the new instance log to comprise the string.

27. The computer-implemented method of claim 25, wherein storing the new instance log among the multiple instance logs comprises using, by the processor, the string as a second index to a location in the federated area at which the new instance log is stored.

28. The computer-implemented method of claim 21, wherein

the task routine identifier of each version of each task routine of the multiple task routines stored in the federated area comprises a hash taken the version of the task routine;

and

comparing the task routine identifier of the most recent version of task routine to the task routine identifier of the version of task routine specified in the instance log for each task comprises comparing the corresponding hashes.

29. The computer-implemented method of claim 21, comprising:

receiving, at the portal, and from a source device via the network, a second request to store a task routine among the multiple task routines in the federated area, wherein the task routine comprises a flow task identifier to indicate a corresponding task that is performed when the task routine is executed;

determining, by the processor, whether another task routine is already stored among the multiple task routines in the federated area that comprises the same flow task identifier to indicate that the same corresponding task is performed when the other task routine is executed; and

in response to determining that another task routine is already stored among the multiple task routines that comprises the same flow task identifier, performing operations comprising;

storing, by the processor, the task routine among the multiple task routines; and

storing, by the processor, an indication of the storage of more than one task routine comprising the same flow task identifier in the federated area.

30. The computer-implemented method of claim 29, wherein:

the task routine comprises;

an input interface by which the task routine is to receive the at least one data set; and

an output interface; and

the method comprises, in response to a determination that another task routine is already stored among the multiple task routines that comprises the same flow task identifier, performing operations comprising;

comparing, by the processor, the input interface to a corresponding input interface of the other task routine;

comparing, by the processor, the output interface to a corresponding output interface of the other task routine; and

conditioning, by the processor, the storage of the task routine and the indication of the storage of more than one task routine on a determination that the input interface matches the corresponding input interface, and a determination that the output interface matches or is a superset of the corresponding output interface.