Operating programs on a computer cluster

• US 9,542,226 B2
• Filed: 06/26/2014
• Issued: 01/10/2017
• Est. Priority Date: 10/07/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bieswanger, Andreas, Leoshkevich, Ilija
• Primary Examiner(s): BROPHY, MATTHEW J
• Application Number:
  US14/315,518
• Publication Number:
  US 20150100968A1
• Time in Patent Office: 929 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455

Abstract

A mechanism is provided for operating programs on a computer cluster comprising cluster resources. The cluster resources comprise non-virtual real hardware resources with variable configurations and virtual resources. Each cluster resource has a configuration description and a type. Each type has a unique type identification and descriptions of operations that can be performed by the cluster resource of the each type. Each program is operable for: requesting usage of the cluster resource specifying the type and the configuration description; and requesting a modification of the variable configuration of the non-virtual real hardware resource with the variable configuration. Execution of each program requires the dedicated execution environment on the computer cluster. The generation of each dedicated execution environment requires one or more dedicated virtual resources and one or more dedicated non-virtual real hardware resources with the variable configurations.

Claims

1. A computer system for operating programs executable on a computer cluster comprising cluster resources, wherein the cluster resources comprise the following:

non-virtual real hardware resources with variable configurations and virtual resources, wherein each cluster resource has a configuration description and a type, wherein each type has a unique type identification and descriptions of operations that can be performed by the cluster resource of the each type,

wherein the computer system comprises a processor and a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

identify cluster resources for generation of one or more dedicated execution environments, wherein execution of each program requires a dedicated execution environment on the computer cluster; and

generate the one or more dedicated execution environments for execution of programs using the identified duster resources, wherein generation of each dedicated execution environment requires a plurality of virtual resources including one or more dedicated virtual resources and one or more dedicated non-virtual real hardware resources with the variable configurations, wherein each dedicated resource has an execution environment specific type and an execution environment specific configuration description, wherein each identified cluster resource matches one dedicated resource, wherein each identified cluster resource and the matching dedicated resource have the same type, and wherein the configuration description of each identified cluster resource comprises the execution environment specific configuration description of the matching dedicated resource,

wherein at least a portion of programs have execution conflicts, wherein each program has a unique identification, wherein the unique identifications of the programs having execution conflicts are stored in a list comprising one or more pairs of the unique identifications of the programs having execution conflicts when both programs of any pair are executed concurrently, wherein the instructions further cause the processor to;

split the programs in a minimum possible number of groups, wherein each group comprises no pairs of the programs having corresponding pairs of the unique identifications in the list;

generate a schedule for execution of the programs, wherein all programs of each group are scheduled for concurrent execution and the groups of the programs are scheduled for consecutive execution, wherein consecutive execution of the groups is prioritized according to a number of the programs in the groups, wherein the group comprising the highest number of programs is scheduled as the first group for execution and the group comprising the least number of programs is scheduled as the last group for execution, wherein responsive to two groups comprising the same number of programs the two groups are prioritized at random;

start execution of the programs according to the schedule in the one or mote generated dedicated execution environments;

detect an execution conflict of a pair of the programs;

abort one of the programs of the pair of the programs having the execution conflict detected;

update the list with the pair the unique identifications of the pair of the programs having the execution conflict detected;

generate a new schedule for execution of the aborted program and programs for which execution was not started yet; and

start execution of the programs according to the new schedule in the one or more generated dedicated execution environments after execution of the group comprising another program of the pair is finished.

2. The computer system of claim 1, wherein the cluster resources are hierarchically allocated within one or more resource parent-child relationship trees and linked by resource parent-child relationships with each other within the one or more resource parent-child relationship trees, wherein each resource parent-child relationship tree matches a type parent-child relationship tree comprising the hierarchically allocated types linked by type parent-child relations within the type parent-child relationship tree, wherein the resource parent-child relationships match the type parent-child relationships of their types, wherein each resource parent-child relationship tree has one top root cluster resource, wherein each type parent-child relationship tree has one top root type, wherein each top root cluster resource of each resource parent-child relationship tree has the top root type of the type parent-child relationship tree matching the each resource parent-child relationship tree, wherein the identifying of the cluster resources for the generation of the one or more dedicated execution environments comprises:

identifying first fragments of the one or more type parent-child relationship trees linking the one or more top root types of the first fragments with the environment specific types allocated at bottoms of the first fragments;

identifying second fragments of the one or more resource parent child-relationship trees matching the first fragments, wherein the identifying of the cluster resources for the generation of the one or more dedicated environments is performed using the cluster resources allocated at bottoms of the second fragments, wherein all cluster resources allocated at bottoms of the second fragments have the environment specific types;

storing the second fragments in a registry; and

identifying one or more first adjacent fragments of one or more resource parent-child relationship trees using a lazy thunk identification, wherein each first adjacent fragment is adjacent to one of the second fragments; and

updating the registry with all identified first adjacent fragments.

3. The computer system of claim 1, wherein the dedicated environment comprises a virtual machine operated by an operating system, wherein the generating of the one or more dedicated environments using the identified cluster resources comprises:

creating the virtual machine using one or more identified virtual resources;

connecting the identified non-virtual real hardware resource with the variable configuration to the virtual machine; and

installing the operating system on the virtual machine.

5. The computer system of claim 1, wherein each program is operable for requesting usage of the cluster resource specifying the type and the configuration description, wherein the instructions further cause the processor to:

identify the cluster resource responsive to a request of the program for usage of the cluster resource, wherein the identified cluster resource has the specified type, wherein the configuration description of the identified cluster resource comprises the specified configuration description.

6. The computer system of claim 1, wherein each program is operable for requesting a modification of the variable configuration of the non-virtual real hardware resource with the variable configuration, wherein the instructions further cause the processor to:

in response to a request of a given program for a modification of the variable configuration of the non-virtual real hardware resource with the variable configuration, reserve the non-virtual real hardware resource with the variable configuration for exclusive usage by the given program and executing the requested modification of the variable configuration of the non-virtual real hardware resource with the variable configuration;

generate a record of the executed modifications of the variable configuration of the non-virtual real hardware with the variable configuration;

roll back the executed modifications of the variable configuration of said non-virtual real hardware resource with the variable configuration using the record responsive to determining execution of the given program is ended; and

responsive to rolling back the executed modifications, cancel the reserving of the non-virtual real hardware resource with the variable configuration.

8. The computer system of claim 1, wherein at least a portion of programs have execution conflicts, wherein each program has a unique identification, wherein the unique identifications of the programs having execution conflicts are stored in a list comprising one or more pairs of the unique identifications of the programs having execution conflicts when both programs of any pair are executed concurrently, wherein the instructions further cause the processor to:

a) generate an additional group;

b) assign one of the not yet assigned programs having no execution conflicts with all programs assigned to the additional group;

c) repeat step b) until there are no programs left that have no execution conflicts with any of the programs of the additional group;

d) iteratively repeat the steps a)-c), wherein the repeating is continued until all programs are assigned to one or more groups, wherein exactly one additional group is generated during each repeating of the steps a)-c), wherein all pairs comprising at least one unique identification of any of the programs assigned to any previously generated additional group are considered as being deleted from the list during the subsequent repeating of the steps a)-c);

generate a schedule for execution of the programs, wherein all programs of each group are scheduled for concurrent execution and the groups of the programs are scheduled for consecutive execution, wherein consecutive execution of the groups is prioritized according to a number of the programs in the groups, wherein the group comprising the highest number of programs is scheduled as the first group for execution and the group comprising the least number of programs is scheduled as the last group for execution, wherein responsive to determining two groups comprise the same number of programs, the two groups are prioritized at random;

start of execution of the programs according to the schedule in the one or more generated dedicated execution environments;

detect an execution conflict of a pair of the programs;

abort one of the programs of the pair of the programs having the execution conflict detected;

update the list with the pair of the unique identifications of the pair of the programs having the execution conflict detected; and

responsive to updating the list with the pair of the unique identifications of the pair of the programs having the execution conflict detected, identify all programs within the pairs in the list that comprise the unique identification of the aborted program;

responsive to identifying that one or more not yet executed groups do not comprise any identified program, assign the aborted program to the not yet executed group not comprising any identified programs and having the highest priority for execution among the not yet executed groups; and

responsive to identifying that one or more not yet executed groups do comprise any identified program, schedule execution of the aborted program after execution of the last group.

9. The computer system of claim 1, wherein the aborted program of the pair of the programs is the program having a larger number of the cluster resources requested for usage in comparison with another program of the pair.

10. The computer system of claim 1, wherein each program has a workload to be processed during execution, wherein the aborted program of the pair of the programs is the program having a lower percentage of the executed workload in comparison with another program of the pair.

11. The computer system of claim 1, wherein the aborted program of the pair of the programs is the program having a shorter duration of execution in comparison with another program of the pair.

12. The computer system of claim 2, wherein identifying the cluster resource upon the request of the program for usage of the cluster resource comprises:

checking whether at least one of the cluster resources allocated in the fragments of the one or more resource parent-child relationship trees stored in the registry has the specified type and has the configuration description comprising the specified configuration description;

responsive to determining that at least one of the luster resources allocated in the fragments of the one or more resource parent-child relationship trees stored in the registry has the specified type and has the configuration description comprising the specified configuration description, the identifying of the cluster resource upon the request of the program for usage of the cluster resource is performed using the fragments of one or more resource parent-child relationship trees stored in the registry; and

responsive to determining that at least one of the luster resources allocated in the fragments of the one or more resource parent-child relationship trees stored in the registry does not have the specified type or does not have the configuration description comprising the specified configuration description;

identifying one or more third fragments of the one or more type parent-child relationship trees linking the one or more top root types of the one or more type parent-child relationship trees with the specified type allocated at one or more bottoms of the one or more third fragments;

identifying one or more fourth fragments of the one or more resource parent child-relationship trees matching the one or more third fragments, wherein the identifying of the cluster resource responsive to the request of the given program for usage of the cluster resource is performed using the cluster resources allocated at one or more bottoms of the one or more fourth fragments, wherein all cluster resources allocated at the one or more bottoms of the one or more fourth fragments have the specified types;

updating the registry with all fourth fragments;

identifying one or more second adjacent fragments of the one or more resource parent-child relationship trees using the lazy thunk identification, wherein each second adjacent fragment is adjacent to one of the one or more fourth fragments; and

updating the registry with the one or more second adjacent fragments.

13. The computer system of claim 2, wherein the identifying of the one or more fragments of the one or more resource parent child-relationship trees is performed applying one or more predefined identification constraints restricting the identifying of the fragments of the one or more resource parent-child relationship trees.

4. A computer program product for operating programs executable on a computer cluster comprising cluster resources, wherein the cluster resources comprise non-virtual real hardware resources with variable configurations and virtual resources, wherein each cluster resource has a configuration description and a type, wherein each type has a unique type identification and descriptions of operations that can be performed by the cluster resource of each type, wherein the computer program produce comprises a computer readable storage medium carrying computer executable code for execution by a processor, wherein execution of the computer executable code causes the processor to:

identify the cluster resources for generation of one or more dedicated execution environments, wherein execution of each program requires the dedicated execution environment on the computer cluster, and

generate the one or more dedicated execution environments for execution of programs using the identified cluster resources, wherein generation of each dedicated execution environment requires a plurality of virtual resources including one or more dedicated virtual resources and one or more dedicated non-virtual real hardware resources with the variable configurations, wherein each dedicated resource has an execution environment specific type and an execution environment specific configuration description, wherein each identified cluster resource matches one dedicated resource, wherein each identified cluster resource and the matching dedicated resource have the same type, and wherein the configuration description of each identified cluster resource comprises the execution environment specific configuration description of the matching dedicated resource,

wherein at least a portion of programs have execution conflicts, wherein each program has a unique identification, wherein the unique identifications of the programs having execution conflicts are stored in a list comprising one or more pairs of the unique identifications of the programs having execution conflicts when both programs of any pair are executed concurrently, wherein the computer executable code further causes the processor to;

split the programs in a minimum possible number of groups, wherein each group comprises no pairs of the programs having corresponding pairs of the unique identifications in the list;

generate a schedule for execution of the programs, wherein all programs of each group are scheduled for concurrent execution and the groups of the programs are scheduled for consecutive execution, wherein consecutive execution of the groups is prioritized according to a number of the programs in the groups, wherein the group comprising the highest number of programs is scheduled as the first group for execution and the group comprising the least number of programs is scheduled as the last group for execution, wherein responsive to two groups comprising the same number of programs the two groups are prioritized at random;

start execution of the programs according to the schedule in the one or more generated dedicated execution environments;

detect an execution conflict of a pair of programs;

abort one of the programs of the pair of the programs having the execution conflict detected;

update the list with the pair the unique identifications of the pair of the programs having the execution conflict detected;

generate a new schedule for execution of the aborted program and programs for which execution was not started yet; and

start execution of the programs according to the new schedule in the one or more generated dedicated execution environments after execution of the group comprising another program of the pair is finished.

7. The computer program product of claim 4, wherein each program is operable for requesting a modification of the variable configuration of the non-virtual real hardware resource with the variable configuration, wherein the computer executable code further causes the processor to:

in response to a request of a given program for a modification of the variable configuration of the non-virtual real hardware resource with the variable configuration, reserve the non-virtual real hardware resource with the variable configuration for exclusive usage by the given program and executing the requested modification of the variable configuration of the non-virtual real hardware resource with the variable configuration;

generate a record of the executed modifications of the variable configuration of the non-virtual real hardware with the variable configuration;

roll back the executed modifications of the variable configuration of said non-virtual real hardware resource with the variable configuration using the record responsive to determining execution of the given program is ended; and

responsive to rolling back the executed modifications, cancel the reserving of the non-virtual real hardware resource with the variable configuration.

14. The computer program product of claim 4, wherein at least a portion of programs have execution conflicts, wherein each program has a unique identification, wherein the unique identifications of the programs having execution conflicts are stored in a list comprising one or more pairs of the unique identifications of the programs having execution conflicts when both programs of any pair are executed concurrently, wherein the computer executable code further causes the processor to:

a) generate an additional group;

b) assign one of the not yet assigned programs having no execution conflicts with all programs assigned to the additional group;

c) repeat step b) until there are no programs left that have no execution conflicts with any of the programs of the additional group;

d) iteratively repeat the steps a)-c), wherein the repeating is continued until all programs are assigned to one or more groups, wherein exactly one additional group is generated during each repeating of the steps a)-c), wherein all pairs comprising at least one unique identification of any of the programs assigned to any previously generated additional group are considered as being deleted from the list during the subsequent repeating of the steps a)-c);

generate a schedule for execution of the programs, wherein all programs of each group are scheduled for concurrent execution and the groups of the programs re scheduled for consecutive execution, wherein consecutive execution of the groups is prioritized according to a number of the programs in the groups, wherein the group comprising the highest number of programs is scheduled as the first group for execution and the group comprising the least number of programs is scheduled as the last group for execution, wherein responsive to determining two groups comprise the same number of programs, the two groups are prioritized at random;

start of execution of the programs according to the schedule in the one or more generated dedicated execution environments;

detect an execution conflict of a pair of the programs;

abort one of the programs of the pair of the programs having the execution conflict detected;

update the list with the pair of the unique identifications of the pair of the programs having the execution conflict detected; and

responsive to updating the list with the pair of the unique identifications of the pair of the programs having the execution conflict detected, identify all programs within the pairs in the list that comprise the unique identification of the aborted program;

responsive to identifying that one or more not yet executed groups do not comprise any identified program, assign the aborted program to the not yet executed group not comprising any identified programs and having the highest priority for execution among the not yet executed groups; and

responsive to identifying that one or more not yet executed groups do comprise any identified program, schedule execution of the aborted program after execution of the last group.

15. The computer program product of claim 4, wherein the cluster resources are hierarchically allocated within one or more resource parent-child relationship trees and linked by resource parent-child relationships with each other within the one or more resource parent-child relationship trees, wherein each resource parent-child relationship tree matches a type parent-child relationship tree comprising the hierarchically allocated types linked by type parent-child relations within the type parent-child relationship tree, wherein the resource parent-child relationships match the type parent-child relationships of their types, wherein each resource parent-child relationship tree has one top root cluster resource, wherein each type parent-child relationship tree has one top root type, wherein each top root cluster resource of each resource parent-child relationship tree has the top root type of the type parent-child relationship tree matching the each resource parent-child relationship tree, wherein the identifying of the cluster resources for the generation of the one or more dedicated execution environments comprises:

identifying first fragments of the one or more type parent-child relationship trees linking the one or more top root types of the first fragments with the environment specific types allocated at bottoms of the first fragments;

identifying second fragments of the one or more resource parent child-relationship trees matching the first fragments, wherein the identifying of the cluster resources for the generation of the one or more dedicated environments is performed using the cluster resources allocated at bottoms of the second fragments, wherein all cluster resources allocated at bottoms of the second fragments have the environment specific types;

storing the second fragments in a registry; and

identifying one or more first adjacent fragments of one or more resource parent-child relationship trees using a lazy thunk identification, wherein each first adjacent fragment is adjacent to one of the second fragments; and

updating the registry with all identified first adjacent fragments.

16. The computer program product of claim 4, wherein the dedicated environment comprises a virtual machine operated by an operating system, wherein the generating of the one or more dedicated environments using the identified cluster resources comprises:

creating the virtual machine using one or more identified virtual resources;

connecting the identified non-virtual real hardware resource with the variable configuration to the virtual machine; and

installing the operating system on the virtual machine.

17. The computer program product of claim 4, wherein each program is operable for requesting usage of the cluster resource specifying the type and the configuration description, wherein the computer executable code further causes the processor to:

identify the cluster resource responsive to a request of the program for usage of the cluster resource, wherein the identified cluster resource has the specified type, wherein the configuration description of the identified cluster resource comprises the specified configuration description.

18. The computer program product of claim 4, wherein the aborted program of the pair of the programs is the program having a larger number of the cluster resources requested for usage in comparison with another program of the pair.

19. The computer program product of claim 4, wherein each program has a workload to be processed during execution, wherein the aborted program of the pair of the programs is the program having a lower percentage of the executed workload in comparison with another program of the pair.

20. The computer program product of claim 4, wherein the aborted program of the pair of the programs is the program having a shorter duration of execution in comparison with another program of the pair.