Event-based dynamic resource provisioning

• US 8,977,752 B2
• Filed: 04/16/2009
• Issued: 03/10/2015
• Est. Priority Date: 04/16/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Company (International Business Machines Corporation)
• Inventors: Arimilli, Ravi Kumar, Chaudhary, Piyush
• Primary Examiner(s): NGUYEN, MINH CHAU
• Application Number:
  US12/424,893
• Publication Number:
  US 20100269119A1
• Time in Patent Office: 2,154 Days
• Field of Search: 709/226, 709/201, 709/203
• US Class Current: 709/226
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources

Abstract

Disclosed are a method, a system and a computer program product for automatically allocating and de-allocating resources for jobs executed or processed by one or more supercomputer systems. In one or more embodiments, a supercomputing system can process multiple jobs with respective supercomputing resources. A global resource manager can automatically allocate additional resources to a first job and de-allocate resources from a second job. In one or more embodiments, the global resource manager can provide the de-allocated resources to the first job as additional supercomputing resources. In one or more embodiments, the first job can use the additional supercomputing resources to perform data analysis at a higher resolution, and the additional resources can compensate for an amount of time the higher resolution analysis would take using originally allocated supercomputing resources.

Claims

1. A computer-implemented method for operating a supercomputing system, comprising:

processing a first supercomputing job with a first amount of resources of the supercomputing system;

determining that an event occurred while processing a data set of the first supercomputing job, wherein the determining includes automatically determining the event occurred based on analysis of the data set by the processing;

in response to determining that the event occurred;

notifying a resource manager that the event occurred;

determining a first amount of additional resources of the supercomputing system based on a first resolution of data employed in the processing of the data set, a second resolution of data to be employed in the processing of the data set, a size of the data set, and a target completion time for the first supercomputing job;

allocating the first amount of additional resources of the supercomputing system;

distributing at least a portion of the data set to the first additional computing resources;

processing the first supercomputing job at the second resolution of the data set with the first amount of resources of the supercomputing system and the first amount of additional resources of the supercomputing system;

during said processing of the first supercomputing job at the second resolution of the data set with the first amount of resources of the supercomputing system and the first amount of additional resources of the supercomputing system, determining whether the first supercomputing job is processing anomalous data not indicative of the event; and

in response to determining that the first supercomputing job is processing anomalous data not indicative of the event, de-allocating the first amount of additional resources of the supercomputing system and resuming processing of the first supercomputing job at the first resolution with the first amount of resources of the supercomputing system.

2. The computer-implemented method of claim 1, further comprising:

processing a second supercomputing job with a second amount of resources of the supercomputing system; and

de-allocating a portion of the second amount of resources of the supercomputing system of the second supercomputing job.

3. The computer-implemented method of claim 2, further comprising:

processing the second supercomputing job with the second amount of resources of the supercomputing system less the de-allocated portion;

wherein said allocating the first amount of additional resources of the supercomputing system includes allocating the de-allocated portion of the second amount of resources of the supercomputing system.

4. The computer-implemented method of claim 3, further comprising:

de-allocating the first amount of additional resources of the supercomputing system;

processing the first supercomputing job with the first amount of resources of the supercomputing system;

allocating the de-allocated portion of the second amount of resources of the supercomputing system for the second supercomputing job; and

processing the second supercomputing job with the second amount of resources of the supercomputing system.

5. The computer-implemented method of claim 1, wherein the supercomputing system is coupled to a second supercomputing system through a network;

the method further comprising;

allocating a first amount of additional resources of the second supercomputing system; and

processing the first supercomputing job with the first amount of resources of the supercomputing system, the first amount of additional resources of the supercomputing system, and the first amount of additional resources of the second supercomputing system.

6. The method of claim 1, wherein the first resolution corresponds to a lesser analysis of the data set and the second resolution corresponds to a greater analysis of the data set.

7. A supercomputing system, comprising:

a plurality of computer nodes, wherein each of the plurality of computer nodes is coupled to another of the plurality of computer nodes;

data storage coupled to at least a first computer node of the plurality of computer nodes, wherein the data storage includes instructions that when executed on the first computer node provides logic for performing the functions of;

processing a first supercomputing job with a first amount of resources of the supercomputing system;

determining that an event occurred while processing a data set of the first supercomputing job, wherein the determining includes automatically determining the event occurred based on analysis of the data set by the processing;

in response to determining that the event occurred;

providing a notification that the event occurred;

determining a first amount of additional resources of the supercomputing system based on a first resolution of data employed in the processing of the data set, a second resolution of data to be employed in the processing of the data set, a size of the data set, and a target completion time for the first supercomputing job;

allocating the first amount of additional resources of the supercomputing system;

distributing at least a portion of the data set to the first additional computing resources; and

processing the first supercomputing job at the second resolution of the data set with the first amount of resources of the supercomputing system and the first amount of additional resources of the supercomputing system;

during said processing of the first supercomputing job at the second resolution of the data set with the first amount of resources of the supercomputing system and the first amount of additional resources of the supercomputing system not indicative of the event, determining whether the first supercomputing job is processing anomalous data; and

in response to determining that the first supercomputing job is processing anomalous data not indicative of the event, de-allocating the first amount of additional resources of the supercomputing system and resuming processing of the first supercomputing job at the first resolution with the first amount of resources of the supercomputing system.

8. The supercomputing system of claim 7, wherein the data storage further includes instructions that when executed on the first computer node provides logic for performing the functions of:

processing a second supercomputing job with a second amount of resources of the supercomputing system; and

p1 de-allocating a portion of the second amount of resources of the supercomputing system of the second supercomputing job.

9. The supercomputing system of claim 8, wherein the data storage further includes instructions that when executed on the first computer node provides logic for performing the function of:

processing the second supercomputing job with the second amount of resources of the supercomputing system less the de-allocated portion;

wherein said allocating the first amount of additional resources of the supercomputing system includes allocating the de-allocated portion of the second amount of resources of the supercomputing system.

10. The supercomputing system of claim 9, wherein the data storage further includes instructions that when executed on the first computer node provides logic for performing the functions of:

de-allocating the first amount of additional resources of the supercomputing system;

processing the first supercomputing job with the first amount of resources of the supercomputing system;

allocating the de-allocated portion of the second amount of resources of the supercomputing system for the second supercomputing job; and

processing the second supercomputing job with the second amount of resources of the supercomputing system.

11. The supercomputing system of claim 7, wherein the supercomputing system is coupled to a second supercomputing system through a network;

wherein the data storage further includes instructions that when executed on the first computer node provides logic for performing the functions of;

allocating a first amount of additional resources of the second supercomputing system; and

processing the first supercomputing job with the first amount of resources of the supercomputing system, the first amount of additional resources of the supercomputing system, and the first amount of additional resources of the second supercomputing system.

12. The supercomputing system of claim 7, wherein the first resolution corresponds to a lesser analysis of the data set and the second resolution corresponds to a greater analysis of the data set.

13. A computer readable memory medium comprising instructions, which when executed on a processing system of a supercomputing system, cause the supercomputing system to perform:

processing a first supercomputing job with a first amount of resources of the supercomputing system;

determining that an event occurred while processing a data set of the first supercomputing job, wherein the determining includes automatically determining the event occurred based on analysis of the data set by the processing;

in response to determining that the event occurred;

notifying a resource manager that the event occurred;

determining a first amount of additional resources of the supercomputing system based on a first resolution of data employed in the processing of the data set, a second resolution of data to be employed in the processing of the data set, a size of the data set, and a target completion time for the first supercomputing job;

allocating the first amount of additional resources of the supercomputing system;

distributing at least a portion of the data set to the first additional computing resources;

processing the first supercomputing job at the second resolution of the data set with the first amount of resources of the supercomputing system and the first amount of additional resources of the supercomputing system;

during said processing of the first supercomputing job at the second resolution of the data set with the first amount of resources of the supercomputing system and the first amount of additional resources of the supercomputing system not indicative of the event, determining whether the first supercomputing job is processing anomalous data; and

in response to determining that the first supercomputing job is processing anomalous data not indicative of the event, de-allocating the first amount of additional resources of the supercomputing system and resuming processing of the first supercomputing job at the first resolution with the first amount of resources of the supercomputing system.

14. The computer readable memory medium of claim 13, wherein the instructions, when executed on the processing system of the supercomputing system, cause the supercomputing system to perform:

processing the second supercomputing job with the second amount of resources of the supercomputing system less the de-allocated portion;

wherein said allocating the first amount of additional resources of the supercomputing system includes allocating the de-allocated portion of the second amount of resources of the supercomputing system.

15. The computer readable memory medium of claim 14, wherein the instructions, when executed on the processing system of the supercomputing system, cause the supercomputing system to perform:

de-allocating the first amount of additional resources of the supercomputing system;

processing the first supercomputing job with the first amount of resources of the supercomputing system;

allocating the de-allocated portion of the second amount of resources of the supercomputing system for the second supercomputing job; and

processing the second supercomputing job with the second amount of resources of the supercomputing system.

16. The computer readable memory medium of claim 13, wherein the instructions, when executed on the processing system of the supercomputing system, cause the supercomputing system to perform:

processing a second supercomputing job with a second amount of resources of the supercomputing system; and

de-allocating a portion of the second amount of resources of the supercomputing system of the second supercomputing job.

17. The computer readable memory medium of claim 13, wherein the instructions, when executed on the processing system of the supercomputing system, cause the supercomputing system to perform:

allocating a first amount of additional resources of a second supercomputing system coupled to the supercomputing system via a network; and

processing the first supercomputing job with the first amount of resources of the supercomputing system, the first amount of additional resources of the supercomputing system, and the first amount of additional resources of the second supercomputing system.

18. The computer readable memory medium of claim 13, wherein the first resolution corresponds to a lesser analysis of the data set and the second resolution corresponds to a greater analysis of the data set.