Distributed data set storage, retrieval and analysis

• US 9,684,543 B1
• Filed: 02/06/2017
• Issued: 06/20/2017
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Bequet, Henry Gabriel Victor, Yang, Eric Jian, Arfaoui, Kais, Stogner, Ronald Earl
• Primary Examiner(s): PATEL, HIREN P
• Application Number:
  US15/425,749
• Time in Patent Office: 134 Days
• Field of Search: None
• CPC Class Codes: G06F 16/9014 hash tables
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An apparatus includes a processor and a storage storing instructions causing the processor to: maintain a federated area; receive a request to perform a job flow with a data set from a remote device; retrieve a job flow definition specifying the tasks of the job flow from the federated area; determine whether there is an instance log in the federated area generated by a previous performance of the job flow with the data set; in response to there being such an instance log, compare the version specified in the instance log of each task routine for each task to the most recent version stored in the federated area; and in response to each version specified in the instance log matching the most recent version, provide the remote device with access to a result report generated by the previous performance in lieu of generating a new result report.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

maintain, within one or more storage devices, a federated area to store multiple data sets, multiple job flow definitions, multiple task routines, multiple result reports and multiple instance logs;

provide, on a network, a portal to control access by a remote device to the federated area via the network;

receive, at the portal, and from the remote device via the network, a first request to execute at least one task routine stored in the federated area to perform at least one corresponding task of a job flow described in a job flow definition stored in the federated area with at least one data set stored in the federated area, wherein the first request specifies the job flow definition and the at least one data set;

retrieve the job flow definition from among the multiple job flow definitions stored in the federated area, wherein the job flow definition comprises a flow task identifier to identify each task of the job flow and specifies a relative order in which each task is to be performed in the job flow;

for each task of the job flow, retrieve, from among the multiple task routines stored in the federated area, a most recent version of the corresponding task routine of the at least one task routine;

determine whether there is an instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set; and

in response to a determination that there is an instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, perform operations comprising;

retrieve, from among the multiple task routines stored in the federated area, a version specified by the instance log of each task routine of the at least one task routine;

for each task of the at least one task of the job flow, compare the version specified by the instance log of each task routine of the at least one task routine to the most recent version of each task routine of the at least one task routine;

in response to each version specified by the instance log of each task routine of the at least one task routine matching the most recent version of the same task routine, perform operations comprising;

retrieve a result report that was generated by the previous performance of the at least one task of the job flow along with the instance log; and

provide access to the result report to the remote device via the network; and

in response to a determination that there is no instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, perform operations comprising;

retrieve the at least one data set from among the multiple data sets stored in the federated area;

execute the most recent version of each task routine of the at least one task routine to perform the at least one corresponding task of the job flow with the at least one data set to generate a new result report and a new instance log;

store the new result report among the multiple result reports in the federated area;

store the new instance log among the multiple instance logs in the federated area; and

provide access to the new result report to the remote device via the network.

2. The apparatus of claim 1, wherein, in the generation of the new instance log, the processor is caused to perform operations comprising:

take at least a first hash of the at least one data set;

take at least a second hash of the retrieved version of a task routine of the at least one task routine;

take at least a third hash of the new result report;

concatenate at least the first, second and third hashes to generate a string; and

generate the new instance log to comprise the string.

3. The apparatus of claim 1, wherein the processor is caused to, in response to one version specified by the instance log of a task routine of the at least one task routine not matching the most recent version of the same task routine, perform operations comprising:

retrieve the at least one data set from among the multiple data sets stored in the federated area;

starting with an earliest task to be performed of the at least one task of the job flow indicated in the job flow definition, identify the earliest task for which the version of the corresponding task routine specified by the instance log does not match the most recent version of the same task routine;

for each task of the at least one task of the job flow, starting with the identified earliest task, execute the most recent version of the corresponding task routine of the at least one task routine to generate a new result report and a new instance log;

store the new result report among the multiple result reports in the federated area;

store the new instance log among the multiple instance logs in the federated area; and

provide access to the new result report to the remote device via the network.

4. The apparatus of claim 1, wherein:

the determination of whether there is an instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises a determination, by the processor, of whether there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set; and

in response to a determination that there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set, the processor is caused to retrieve, from among the multiple task routines stored in the federated area, a version of each task routine of the at least one task routine specified by the most recently generated one of the more than one instance logs.

5. The apparatus of claim 1, wherein, in the comparison of the version specified by the instance log of each task routine to the most recent version of each task routine for each task of the at least one task of the job flow, the processor is caused to compare a hash taken of the version specified by the instance log of each task routine to a hash taken of the most recent version of each task routine for each task of the at least one task of the job flow.

6. The apparatus of claim 1, wherein, in the determination of whether there is an instance log stored among the multiple instance logs that was generated by a previous performance of the at least one task of the job flow with the at least one data set, the processor is caused to perform operations comprising:

use at least one data set identifier of the at least one data set and a flow identifier of the job flow as portions of an index to a location in the federated area; and

search the location for an instance log that was generated by a previous performance of the at least one task of the job flow with the at least one data set.

7. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

receive, at the portal, and from a source device via the network, a second request to store a task routine among the multiple task routines in the federated area, wherein the task routine comprises a flow task identifier to indicate a corresponding task that is performed when the task routine is executed;

determine whether another task routine is already stored among the multiple task routines in the federated area that comprises the same flow task identifier to indicate that the same corresponding task is performed when the other task routine is executed; and

in response to a determination that another task routine is already stored among the multiple task routines that comprises the same flow task identifier, store the task routine among the multiple task routines and store an indication of the storage of more than one task routine comprising the same flow task identifier in the federated area.

8. The apparatus of claim 1, wherein:

for each task of the at least one task of the job flow, the job flow definition comprises a corresponding flow task identifier; and

in the retrieval of a task routine for each task of the at least one task of the job flow, the processor is caused to use the corresponding task flow identifier as at least a portion of an index to a location in the federated area.

9. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

prior to the determination of whether there is an instance log that was generated by a previous performance of the at least one task, determine whether the first request is authorized;

condition performance of the retrieval of the job flow definition, the retrieval of the most recent version of the at least one task routine, and the determination of whether there is an instance log that was generated by a previous performance of the at least one task, the performance of the at least one task on the determination of whether the first request is authorized; and

in response to a determination that the first request is not authorized, provide an indication of a denial of the first request to the remote device via the network.

10. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

maintain, within one or more storage devices, a federated area to store multiple data sets, multiple job flow definitions, multiple task routines, multiple result reports and multiple instance logs;

provide, on a network, a portal to control access by a remote device to the federated area via the network;

receive, at the portal, and from the remote device via the network, a first request to execute at least one task routine stored in the federated area to perform at least one corresponding task of a job flow described in a job flow definition stored in the federated area with at least one data set stored in the federated area, wherein the first request specifies the job flow definition and the at least one data set;

retrieve the job flow definition from among the multiple job flow definitions stored in the federated area, wherein the job flow definition comprises a flow task identifier to identify each task of the job flow and specifies a relative order in which each task is to be performed in the job flow;

for each task of the job flow, retrieve, from among the multiple task routines stored in the federated area, a most recent version of the corresponding task routine of the at least one task routine;

determine whether there is an instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set; and

in response to a determination that there is an instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, perform operations comprising;

retrieve, from among the multiple task routines stored in the federated area, a version specified by the instance log of each task routine of the at least one task routine;

for each task of the at least one task of the job flow, compare the version specified by the instance log of each task routine of the at least one task routine to the most recent version of each task routine of the at least one task routine;

in response to each version specified by the instance log of each task routine of the at least one task routine matching the most recent version of the same task routine, perform operations comprising;

retrieve a result report that was generated by the previous performance of the at least one task of the job flow along with the instance log; and

provide access to the result report to the remote device via the network; and

in response to a determination that there is no instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, perform operations comprising;

retrieve the at least one data set from among the multiple data sets stored in the federated area;

execute the retrieved version of each task routine of the at least one task routine to perform the at least one corresponding task of the job flow with the at least one data set to generate a new result report and a new instance log;

store the new result report among the multiple result reports in the federated area;

store the new instance log among the multiple instance logs in the federated area; and

provide access to the new result report to the remote device via the network.

11. The computer-program product of claim 10, wherein, in the generation of the new instance log, the processor is caused to perform operations comprising:

take at least a first hash of the at least one data set;

take at least a second hash of the retrieved version of a task routine of the at least one task routine;

take at least a third hash of the new result report;

concatenate at least the first, second and third hashes to generate a string; and

use the string as an index to a location in the federated area at which the new instance log is stored.

12. The computer-program product of claim 10, wherein the processor is caused to, in response to one version specified by the instance log of a task routine of the at least one task routine not matching the most recent version of the same task routine, perform operations comprising:

retrieve the at least one data set from among the multiple data sets stored in the federated area;

starting with an earliest task to be performed of the at least one task of the job flow indicated in the job flow definition, identify the earliest task for which the version of the corresponding task routine specified by the instance log does not match the most recent version of the same task routine;

for each task of the at least one task of the job flow, starting with the identified earliest task, execute the most recent version of the corresponding task routine of the at least one task routine to generate a new result report and a new instance log;

store the new result report among the multiple result reports in the federated area;

store the new instance log among the multiple instance logs in the federated area; and

provide access to the new result report to the remote device via the network.

13. The computer-program product of claim 10, wherein:

the determination of whether there is an instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises a determination, by the processor, of whether there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set; and

in response to a determination that there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set, the processor is caused to retrieve, from among the multiple task routines stored in the federated area, a version of each task routine of the at least one task routine specified by the most recently generated one of the more than one instance logs.

14. The computer-program product of claim 10, wherein, in the comparison of the version specified by the instance log of each task routine to the most recent version of each task routine for each task of the at least one task of the job flow, the processor is caused to compare a hash taken of the version specified by the instance log of each task routine to a hash taken of the most recent version of each task routine for each task of the at least one task of the job flow.

15. The computer-program product of claim 10, wherein, in the determination of whether there is an instance log stored among the multiple instance logs that was generated by a previous performance of the at least one task of the job flow with the at least one data set, the processor is caused to perform operations comprising:

use at least one data set identifier of the at least one data set and a flow identifier of the job flow as portions of an index to a location in the federated area; and

search the location for an instance log that was generated by a previous performance of the at least one task of the job flow with the at least one data set.

16. The computer-program product of claim 10, wherein the processor is caused to perform operations comprising:

receive, at the portal, and from a source device via the network, a second request to store a task routine among the multiple task routines in the federated area, wherein the task routine comprises;

a flow task identifier to indicate a corresponding task that is performed when the task routine is executed;

an input interface by which the task routine is to receive the at least one data set; and

an output interface;

determine whether another task routine is already stored among the multiple task routines in the federated area that comprises the same flow task identifier to indicate that the same corresponding task is performed when the other task routine is executed; and

in response to a determination that another task routine is already stored among the multiple task routines that comprises the same flow task identifier, perform operations comprising;

compare the input interface to a corresponding input interface of the other task routine;

compare the output interface to a corresponding output interface of the other task routine; and

in response to a determination that the input interface matches the corresponding input interface, and in response to a determination that the output interface matches or is a superset of the corresponding output interface, store the task routine among the multiple task routines and store an indication of the storage of more than one task routine comprising the same flow task identifier in the federated area.

17. The computer-program product of claim 10, wherein:

for each task of the at least one task of the job flow, the job flow definition comprises a corresponding flow task identifier; and

in the retrieval of a task routine for each task of the at least one task of the job flow, the processor is caused to use the corresponding task flow identifier as at least a portion of an index to a location in the federated area.

18. The computer-program product of claim 10, wherein the processor is caused to perform operations comprising:

prior to the determination of whether there is an instance log that was generated by a previous performance of the at least one task, determine whether the first request is authorized;

condition performance of the retrieval of the job flow definition, the retrieval of the most recent version of the at least one task routine, and the determination of whether there is an instance log that was generated by a previous performance of the at least one task, the performance of the at least one task on the determination of whether the first request is authorized; and

in response to a determination that the first request is not authorized, provide an indication of a denial of the first request to the remote device via the network.

19. A computer-implemented method comprising:

maintaining, at a server by a processor, and within one or more storage devices, a federated area to store multiple data sets, multiple job flow definitions, multiple task routines, multiple result reports and multiple instance logs;

providing, at the server by the processor, a portal to control access by a remote device to the federated area through a network coupled to the server;

receiving, at the portal at the server, and from the remote device via the network, a first request to execute at least one task routine stored in the federated area to perform at least one corresponding task of a job flow described in a job flow definition stored in the federated area with at least one data set stored in the federated area, wherein the first request specifies the job flow definition and the at least one data set;

retrieving, by the processor, the job flow definition, and from among the multiple job flow definitions stored in the federated area, wherein the job flow definition comprises a flow task identifier to identify each task of the job flow and specifies a relative order in which each task is to be performed in the job flow;

for each task of the job flow, retrieving, by the processor and from among the multiple task routines stored in the federated area, a most recent version of the corresponding task routine of the at least one task routine;

determining, by the processor, whether there is an instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set; and

in response to a determination by the processor that there is an instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, performing operations comprising;

retrieving, by the processor, and from among the multiple task routines stored in the federated area, a version specified by the instance log of each task routine of the at least one task routine;

for each task of the at least one task of the job flow, comparing, by the processor, the version specified by the instance log of each task routine of the at least one task routine to the most recent version of each task routine of the at least one task routine;

in response to each version specified by the instance log of each task routine of the at least one task routine matching the most recent version of the same task routine, performing operations comprising;

retrieving, by the processor, and from among the multiple result reports, a result report that was generated by the previous performance of the at least one task of the job flow along with the instance log; and

providing access to the result report to the remote device via the network; and

in response to a determination that there is no instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set, performing operations comprising;

retrieving, by the processor, the at least one data set from among the multiple data sets stored in the federated area;

executing the retrieved version of each task routine of the at least one task routine to perform the at least one corresponding task of the job flow with the at least one data set to generate a new result report and a new instance log;

storing, by the processor, the new result report among the multiple result reports in the federated area;

storing, by the processor, the new instance log among the multiple instance logs in the federated area; and

providing access to the new result report to the remote device via the network.

20. The computer-implemented method of claim 19, wherein generating the new instance log comprises:

taking, by the processor, at least a first hash of the at least one data set;

taking, by the processor, at least a second hash of the retrieved version of a task routine of the at least one task routine;

taking, by the processor, at least a third hash of the new result report;

concatenating, by the processor, at least the first, second and third hashes to generate a string; and

generating the new instance log to comprise the string.

21. The computer-implemented method of claim 19, comprising, in response to one version specified by the instance log of a task routine of the at least one task routine not matching the most recent version of the same task routine, performing operations comprising:

retrieving, by the processor, the at least one data set from among the multiple data sets stored in the federated area;

starting with an earliest task to be performed of the at least one task of the job flow indicated in the job flow definition, identifying, by the processor, the earliest task for which the version of the corresponding task routine specified by the instance log does not match the most recent version of the same task routine;

for each task of the at least one task of the job flow, starting with the identified earliest task, executing the most recent version of the corresponding task routine of the at least one task routine to generate a new result report and a new instance log;

storing, by the processor, the new result report among the multiple result reports in the federated area;

storing, by the processor, the new instance log among the multiple instance logs in the federated area; and

providing access to the new result report to the remote device via the network.

22. The computer-implemented method of claim 19, wherein:

determining whether there is an instance log among the multiple instance logs stored in the federated area that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises determining, by the processor, whether there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set; and

the method comprises, in response to a determination that there is more than one instance log among the multiple instance logs that were each generated by a previous performance of the at least one task of the job flow with the at least one data set, retrieving, by the processor, and from among the multiple task routines stored in the federated area, a version of each task routine of the at least one task routine specified by the most recently generated one of the more than one instance logs.

23. The computer-implemented method of claim 19, wherein comparing the version specified by the instance log of each task routine to the most recent version of each task routine for each task of the at least one task of the job flow comprises comparing, by the processor, a hash taken of the version specified by the instance log of each task routine to a hash taken of the most recent version of each task routine for each task of the at least one task of the job flow.

24. The computer-implemented method of claim 19, wherein determining whether there is an instance log stored among the multiple instance logs that was generated by a previous performance of the at least one task of the job flow with the at least one data set comprises:

using, by the processor, at least one data set identifier of the at least one data set and a flow identifier of the job flow as portions of an index to a location in the federated area; and

searching, by the processor, the location for an instance log that was generated by a previous performance of the at least one task of the job flow with the at least one data set.

25. The computer-implemented method of claim 19, comprising:

receiving, at the portal at the server, and from a source device via the network, a second request to store a task routine among the multiple task routines in the federated area, wherein the task routine comprises a flow task identifier to indicate a corresponding task that is performed when the task routine is executed;

determining, by the processor, whether another task routine is already stored among the multiple task routines in the federated area that comprises the same flow task identifier to indicate that the same corresponding task is performed when the other task routine is executed; and

in response to determining that another task routine is already stored among the multiple task routines that comprises the same flow task identifier, storing, by the processor, the task routine among the multiple task routines and store an indication of the storage of more than one task routine comprising the same flow task identifier in the federated area.

26. The computer-implemented method of claim 19, wherein:

for each task of the at least one task of the job flow, the job flow definition comprises a corresponding flow task identifier; and

the retrieving of a task routine for each task of the at least one task of the job flow comprises using, by the processor, the corresponding task flow identifier as at least a portion of an index to a location in the federated area.

27. The computer-implemented method of claim 19, comprising:

prior to determining whether there is an instance log that was generated by a previous performance of the at least one task, determining, by the processor, whether the first request is authorized;

conditioning the retrieving of the job flow definition, the retrieving of the most recent version of the at least one task routine, and the determining of whether there is an instance log that was generated by a previous performance of the at least one task, the performance of the at least one task on the determining of whether the first request is authorized; and

in response determining that the first request is not authorized, providing an indication of a denial of the first request to the remote device via the network.