DETERMINING STORAGE TIERS FOR PLACEMENT OF DATA SETS DURING EXECUTION OF TASKS IN A WORKFLOW

• US 20160335134A1
• Filed: 07/29/2016
• Published: 11/17/2016
• Est. Priority Date: 03/31/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gupta, Aayush, Seshadri, Sangeetha
• Granted Patent: US 9,864,642 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5016 the resource being the memory
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores

Abstract

Provided are a computer program product, system, and method for determining storage tiers for placement of data sets during execution of tasks in a workflow. A representation of a workflow execution pattern of tasks for a job indicates a dependency of the tasks and data sets operated on by the tasks. A determination is made of an assignment of the data sets for the tasks to a plurality of the storage tiers based on the dependency of the tasks indicated in the workflow execution pattern. A moving is scheduled of a subject data set of the data sets operated on by a subject task of the tasks that is subject to an event to an assigned storage tier indicated in the assignment for the subject task subject. The moving of the data set is scheduled to be performed in response to the event with respect to the subject task.

Claims

1. A computer program product for assigning tasks to storage tiers to store data sets processed by the tasks, wherein the computer program product comprises a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause operations, the operations comprising:

processing a representation of a workflow execution pattern of tasks for a job indicating a dependency of the tasks and data sets operated on by the tasks;

determining an assignment of the data sets for the tasks to a plurality of the storage tiers based on the dependency of the tasks indicated in the workflow execution pattern, wherein a higher performing storage tier includes faster access storage devices than a relatively lower performing storage tier; and

scheduling a moving of a subject data set of the data sets operated on by a subject task of the tasks that is subject to an event to an assigned storage tier indicated in the assignment for the subject task subject, wherein the moving of the subject data set is scheduled to be performed in response to the event with respect to the subject task.

2. The computer program product of claim 1, wherein the operations further comprise:

determining whether the subject data set is already located on the assigned storage tier, wherein the subject data set is scheduled to be moved to the assigned storage tier in response to determining that the subject data set is not already located on the assigned storage tier.

3. The computer program product of claim 1, wherein an application layer executing the workflow execution pattern of tasks transmits the workflow execution pattern of tasks for the job to a storage layer managing access to the storage tiers, wherein the storage layer performs the operations of determining the assignment and the scheduling of the moving of the subject data set, wherein the operations further comprise:

notifying, by the storage layer, the application layer that a data set for a next task is located on the assigned storage tier; and

executing, by the application layer, the next task in response to receiving the notification from the storage layer that the data set operated on by the next tasks is located on the assigned storage tier.

6. The computer program product of claim 1, wherein the subject task comprises a completed task, wherein the operations further comprise:

determining whether an additional task is currently operating on the a data set operated on by the completed task; and

scheduling the moving of the data set operated on by the completed task to a lower performing storage tier than assigned to the completed task in response to determining that there is no additional task currently operating on the data set operated on by the completed task.

7. The computer program product of claim 1, wherein the determining the assignment comprises:

determining related data sets comprising multiple data sets operated on by a single task, wherein related data sets are located on a same storage tier.

8. The computer program product of claim 1, wherein the determining the assignment comprises:

assigning a same storage tier to multiple tasks that concurrently operate on a same data set.

9. The computer program product of claim 1, wherein the determining the assignment comprises:

assigning a same storage tier to multiple input tasks operating on at least one data set that provide input to a dependent task that cannot execute until receiving the input from the input tasks.

10. The computer program product of claim 9, wherein the same storage tier assigned to the multiple input tasks is assigned to a preferred higher performing storage tier having higher performance than at least one other storage tier.

11. The computer program product of claim 1, wherein the workflow execution pattern comprises a plurality of jobs, wherein each of the jobs includes a plurality of tasks that operate on data sets, wherein the determining the assignment comprises:

determining related data sets comprising at least one of;

multiple data sets operated on by a single task in one of the jobs;

multiple data sets concurrently operated on by a first group of tasks from at least one of the jobs;

multiple data sets that must be operated on by a second group of tasks from at least one of the jobs before at least one dependent task from at least one of the jobs can execute; and

multiple data sets operated by sequential tasks in at least one of the jobs part of a sequential workflow; and

assigning tasks operating on related data sets to a preferred higher performing storage tier having higher performance than at least one other storage tier.

12. The computer program product of claim 11, wherein the operations further comprise:

determining whether there is sufficient space on the preferred higher performing storage tier for the related data sets; and

assigning the tasks operating on the related data sets to a lower performing storage tier than the preferred higher performing storage tier that has sufficient space to store the related data sets in response to determining that the preferred higher performing storage tier does not have the sufficient space to store the related data sets.

13. The computer program product of claim 1,

determining that a data set will be subsequently processed by one of the tasks in the workflow execution pattern;

and

scheduling a prestaging of the data set that will be subsequently processed to a higher performing storage tier that has higher performance than at least one other storage tier to make the data set available on the higher performing storage tier when the data set is processed.

4-5. -5. (canceled)

14. A system coupled to a plurality of storage tiers, comprising:

a plurality of computational nodes; and

a computer readable storage medium having program instructions that when executed by the computational nodes perform operations, the operations comprising;

processing a representation of a workflow execution pattern of tasks for a job indicating a dependency of the tasks and data sets operated on by the tasks;

determining an assignment of the data sets for the tasks to a plurality of the storage tiers based on the dependency of the tasks indicated in the workflow execution pattern, wherein a higher performing storage tier includes faster access storage devices than a relatively lower performing storage tier; and

scheduling a moving of a subject data set of the data sets operated on by a subject task of the tasks that is subject to an event to an assigned storage tier indicated in the assignment for the subject task subject, wherein the moving of the subject data set is scheduled to be performed in response to the event with respect to the subject task.

17. The system of claim 14, wherein the operations further comprise:

determining whether an additional task is currently operating on a data set operated on by a completed task; and

scheduling the moving of the data set operated on by the completed task to a lower performing storage tier than assigned to the completed task in response to determining that there is no additional task currently operating on the data set operated on by the completed task.

18. The system of claim 14, wherein the workflow execution pattern comprises a plurality of jobs, wherein each of the jobs includes a plurality of tasks that operate on data sets, wherein the determining the assignment comprises:

determining related data sets comprising at least one of;

multiple data sets operated on by a single task in one of the jobs;

multiple data sets concurrently operated on by a first group of tasks from at least one of the jobs;

multiple data sets that must be operated on by a second group of tasks from at least one of the jobs before at least one dependent task from at least one of the jobs can execute; and

multiple data sets operated by sequential tasks in at least one of the jobs part of a sequential workflow; and

assigning tasks operating on related data sets to a preferred higher performing storage tier having higher performance than at least one other storage tier.

15-16. -16. (canceled)

19. A method tasks to storage tiers to store data sets processed by the tasks, comprising:

processing a representation of a workflow execution pattern of tasks for a job indicating a dependency of the tasks and data sets operated on by the tasks;

determining an assignment of the data sets for the tasks to a plurality of the storage tiers based on the dependency of the tasks indicated in the workflow execution pattern, wherein a higher performing storage tier includes faster access storage devices than a relatively lower performing storage tier; and

scheduling a moving of a subject data set of the data sets operated on by a subject task of the tasks that is subject to an event to an assigned storage tier indicated in the assignment for the subject task subject, wherein the moving of the subject data set is scheduled to be performed in response to the event with respect to the subject task.

22. The method of claim 19, wherein the subject task comprises a completed task, further comprising:

determining whether an additional task is currently operating on a data set operated on by the completed task; and

scheduling the moving of the data set operated on by the completed task to a lower performing storage tier than assigned to the completed task in response to determining that there is no additional task currently operating on the data set operated on by the completed task.

23. The method of claim 19, wherein the workflow execution pattern comprises a plurality of jobs, wherein each of the jobs includes a plurality of tasks that operate on data sets, wherein the determining the assignment comprises:

determining related data sets comprising at least one of;

multiple data sets operated on by a single task in one of the jobs;

multiple data sets concurrently operated on by a first group of tasks from at least one of the jobs;

multiple data sets that must be operated on by a second group of tasks from at least one of the jobs before at least one dependent task from at least one of the jobs can execute; and

multiple data sets operated by sequential tasks in at least one of the jobs part of a sequential workflow; and

assigning tasks operating on related data sets to a preferred higher performing storage tier having higher performance than at least one other storage tier.

20-21. -21. (canceled)