WORKLOAD MANAGEMENT WITH DATA ACCESS AWARENESS IN A COMPUTING CLUSTER

• US 20190310893A1
• Filed: 04/05/2018
• Published: 10/10/2019
• Est. Priority Date: 04/05/2018
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: ARONOVICH, Lior
• Granted Patent: US 10,768,998 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 13/1689 Synchronisation and timing concerns synchronisation on a memory bus G06F13/4234
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: G06F 9/505 considering the load
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

Embodiments for workload management with data access awareness in a computing cluster. In response to receiving an input workload for scheduling by a workload manager, a set of inputs is retrieved from a storage system by a data requirements evaluator module. The data requirements evaluator module generates a list of cluster hosts ranked for performing the input workload according to data access considerations and provides the ranked list of cluster hosts to a scheduler module. The scheduler module generates a scheduling of the input workload to certain hosts within the computing cluster where the generated scheduling is optimized with the data access considerations.

Claims

1. A method for workload management with data access awareness in a computing cluster, by a processor, comprising:

configuring a workload manager within the computing cluster to include a data requirements evaluator module and a scheduler module; and

in response to receiving an input workload for scheduling by the workload manager;

retrieving, by the data requirements evaluator module, a set of inputs from a storage system, wherein the inputs each include at least one of;

an indication of whether the input workload is intensive in Input/Output (I/O) of new data or intensive in I/O of existing data,

data locality proportions for a set of files associated with the input workload, and

data access costs specified for each pair of hosts in the computing cluster;

generating, by the data requirements evaluator module, a list of cluster hosts ranked for performing the input workload according to data access considerations;

providing the ranked list of cluster hosts to the scheduler module; and

generating, by the scheduler module, a scheduling of the input workload to certain hosts within the computing cluster where the generated scheduling is optimized with the data access considerations.

2. The method of claim 1, wherein:

the data locality proportions input specifies, for the set of files associated with the input workload, a proportion of a total data of the set of files that is respectively stored on each of the cluster hosts; and

the data access costs input specifies, for each cluster host, a cost of accessing data stored on any other cluster host in the computing cluster, wherein the data access costs are computed based on networking latencies between the cluster hosts and storage device access latencies within each of the cluster hosts.

3. The method of claim 1, further including receiving, by the data requirements evaluator module, further inputs including at least one of:

an availability of compute resources in the computing cluster, and

an availability of storage resources in the computing cluster.

4. The method of claim 1, wherein the generating by the data requirements evaluator module the list of cluster hosts ranked for performing the given workload further includes:

when the input workload is intensive in I/O of existing data, generating the ranked list as an ordered list of preferred hosts according to the data locality and data access costs inputs;

when the input workload is intensive in I/O of new data, generating the ordered list of preferred hosts according to available storage; and

when the input workload is not intensive in I/O, generating an empty list of preferred hosts.

5. The method of claim 4, wherein generating the ordered list of preferred hosts according to available storage is based on setting a rank of a first one of the cluster hosts to be higher in the ordered list when compared with other cluster hosts, upon determining more available storage exists within the first one of the cluster hosts when compared with the other cluster hosts.

6. The method of claim 1, wherein the generating, by the scheduler module, the scheduling of the input workload to the cluster hosts includes at least one of:

receiving a resource allocation request for the input workload;

receiving an ordered list of preferred hosts ranked for running the given workload according to the data access considerations from the data requirements evaluator module; and

attempting to allocate compute resources from the preferred hosts according to the resource allocation request and the ordered list of preferred hosts, to satisfy the allocation request.

7. The method of claim 1, wherein the storage system performs at least one of:

storing data on local storage devices, each local storage device being attached to one of the plurality of cluster hosts;

storing data on a shared storage device accessible from the cluster hosts; and

storing data on a combination of the local and shared storage devices.

8. The method of claim 7, wherein locations in the cluster of data of the set of files associated with the input workload are specified for both local storage and shared storage by:

assigning corresponding host names to the shared storage devices;

attaching the local storage devices to each of the cluster hosts, the cluster hosts each having been assigned a host name; and

using a storage identification (ID) within the local or shared storage devices specifying the locations, wherein the host name and storage ID provides a global storage location in the cluster for the local and shared storage devices.

9. A system for workload management with data access awareness in a computing cluster, the system comprising:

a workload manager within the computing cluster, the workload manager including a data requirements evaluator module and a scheduler module; and

a processor executing instructions stored in a memory device, wherein the processor, upon executing the instructions;

in response to receiving an input workload for scheduling by the workload manager;

retrieves, by the data requirements evaluator module, a set of inputs from a storage system, wherein the inputs each include at least one of;

an indication of whether the input workload is intensive in Input/Output (I/O) of new data or intensive in I/O of existing data,

data locality proportions for a set of files associated with the input workload, and

data access costs specified for each pair of hosts in the computing cluster;

generates, by the data requirements evaluator module, a list of cluster hosts ranked for performing the input workload according to data access considerations;

provides the ranked list of cluster hosts to the scheduler module; and

generates, by the scheduler module, a scheduling of the input workload to certain hosts within the computing cluster where the generated scheduling is optimized with the data access considerations.

10. The system of claim 9, wherein:

the data locality proportions input specifies, for the set of files associated with the input workload, a proportion of a total data of the set of files that is respectively stored on each of the cluster hosts; and

the data access costs input specifies, for each cluster host, a cost of accessing data stored on any other cluster host in the computing cluster, wherein the data access costs are computed based on networking latencies between the cluster hosts and storage device access latencies within each of the cluster hosts.

11. The system of claim 9, wherein the processor receives, by the data requirements evaluator module, further inputs including at least one of:

an availability of compute resources in the computing cluster, and

an availability of storage resources in the computing cluster.

12. The system of claim 9, wherein the generating by the data requirements evaluator module the list of cluster hosts ranked for performing the given workload further includes:

when the input workload is intensive in I/O of existing data, generating the ranked list as an ordered list of preferred hosts according to the data locality and data access costs inputs;

when the input workload is intensive in I/O of new data, generating the ordered list of preferred hosts according to available storage; and

when the input workload is not intensive in I/O, generating an empty list of preferred hosts.

13. The system of claim 12, wherein generating the ordered list of preferred hosts according to available storage is based on setting a rank of a first one of the cluster hosts to be higher in the ordered list when compared with other cluster hosts, upon determining more available storage exists within the first one of the cluster hosts when compared with the other cluster hosts.

14. The system of claim 9, wherein the generating, by the scheduler module, the scheduling of the input workload to the cluster hosts includes at least one of:

receiving a resource allocation request for the input workload;

receiving an ordered list of preferred hosts ranked for running the given workload according to the data access considerations from the data requirements evaluator module; and

attempting to allocate compute resources from the preferred hosts according to the resource allocation request and the ordered list of preferred hosts, to satisfy the allocation request.

15. The system of claim 9, wherein the storage system performs at least one of:

storing data on local storage devices, each local storage device being attached to one of the plurality of cluster hosts;

storing data on a shared storage device accessible from the cluster hosts; and

storing data on a combination of the local and shared storage devices.

16. The system of claim 15, wherein locations in the cluster of data of the set of files associated with the input workload are specified for both local storage and shared storage by:

assigning corresponding host names to the shared storage devices;

attaching the local storage devices to each of the cluster hosts, the cluster hosts each having been assigned a host name; and

using a storage identification (ID) within the local or shared storage devices specifying the locations, wherein the host name and storage ID provides a global storage location in the cluster for the local and shared storage devices.

17. A computer program product for workload management with data access awareness in a computing cluster, by a processor, the computer program product embodied on a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

an executable portion that configures a workload manager within the computing cluster to include a data requirements evaluator module and a scheduler module; and

an executable portion that, in response to receiving an input workload for scheduling by the workload manager;

retrieves, by the data requirements evaluator module, a set of inputs from a storage system, wherein the inputs each include at least one of;

an indication of whether the input workload is intensive in Input/Output (I/O) of new data or intensive in I/O of existing data,

data locality proportions for a set of files associated with the input workload, and

data access costs specified for each pair of hosts in the computing cluster;

generates, by the data requirements evaluator module, a list of cluster hosts ranked for performing the input workload according to data access considerations;

provides the ranked list of cluster hosts to the scheduler module; and

generates, by the scheduler module, a scheduling of the input workload to certain hosts within the computing cluster where the generated scheduling is optimized with the data access considerations.

18. The computer program product of claim 17, wherein:

the data locality proportions input specifies, for the set of files associated with the input workload, a proportion of a total data of the set of files that is respectively stored on each of the cluster hosts; and

the data access costs input specifies, for each cluster host, a cost of accessing data stored on any other cluster host in the computing cluster, wherein the data access costs are computed based on networking latencies between the cluster hosts and storage device access latencies within each of the cluster hosts.

19. The computer program product of claim 17, further including an executable portion that receives, by the data requirements evaluator module, further inputs including at least one of:

an availability of compute resources in the computing cluster, and

an availability of storage resources in the computing cluster.

20. The computer program product of claim 17, wherein the generating by the data requirements evaluator module the list of cluster hosts ranked for performing the given workload further includes:

when the input workload is intensive in I/O of existing data, generating the ranked list as an ordered list of preferred hosts according to the data locality and data access costs inputs;

when the input workload is intensive in I/O of new data, generating the ordered list of preferred hosts according to available storage; and

when the input workload is not intensive in I/O, generating an empty list of preferred hosts.

21. The computer program product of claim 20, wherein generating the ordered list of preferred hosts according to available storage is based on setting a rank of a first one of the cluster hosts to be higher in the ordered list when compared with other cluster hosts, upon determining more available storage exists within the first one of the cluster hosts when compared with the other cluster hosts.

22. The computer program product of claim 17, wherein the generating, by the scheduler module, the scheduling of the input workload to the cluster hosts includes at least one of:

receiving a resource allocation request for the input workload;

receiving an ordered list of preferred hosts ranked for running the given workload according to the data access considerations from the data requirements evaluator module; and

attempting to allocate compute resources from the preferred hosts according to the resource allocation request and the ordered list of preferred hosts, to satisfy the allocation request.

23. The computer program product of claim 17, wherein the storage system performs at least one of:

storing data on local storage devices, each local storage device being attached to one of the plurality of cluster hosts;

storing data on a shared storage device accessible from the cluster hosts; and

storing data on a combination of the local and shared storage devices.

24. The computer program product of claim 23, wherein locations in the cluster of data of the set of files associated with the input workload are specified for both local storage and shared storage by:

assigning corresponding host names to the shared storage devices;

attaching the local storage devices to each of the cluster hosts, the cluster hosts each having been assigned a host name; and

using a storage identification (ID) within the local or shared storage devices specifying the locations, wherein the host name and storage ID provides a global storage location in the cluster for the local and shared storage devices.