Method and apparatus for providing fault-tolerance in parallel-processing systems

• US 20070220298A1
• Filed: 03/20/2006
• Published: 09/20/2007
• Est. Priority Date: 03/20/2006
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Gross, Kenny, Wood, Alan
• Granted Patent: US 7,571,347 B2
• Time in Patent Office: Days
• US Class Current: 714/2
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 2201/81 Threshold

Abstract

A system that provides fault tolerance in a parallel processing system. During operation, the system executes a parallel computing application in parallel across a subset of computing nodes within the parallel processing system. During this process, the system monitors telemetry signals within the parallel processing system. The system analyzes the monitored telemetry signals to determine if the probability that the parallel processing system will fail is increasing. If so, the system increases the frequency at which the parallel computing application is checkpointed, wherein a checkpoint includes the state of the parallel computing application at each computing node within the parallel processing system.

Claims

1. A method for providing fault-tolerance in a parallel-processing system, comprising:

executing a parallel-computing application in parallel across a subset of computing nodes within the parallel-processing system;

monitoring telemetry signals within the parallel-processing system;

analyzing the monitored telemetry signals to determine if the probability that the parallel-processing system will fail is increasing; and

if so, increasing the frequency at which the parallel-computing application is checkpointed, wherein a checkpoint includes the state of the parallel-computing application at each computing node within the parallel-processing system.

2. The method of claim 1, wherein analyzing the monitored telemetry signals to determine if the probability that the parallel-processing system will fail is increasing involves:

using the monitored telemetry signals to generate a health index, which indicates the risk that the parallel-processing system will fail; and

determining if the health index indicates that the probability that the parallel-processing system will fail is increasing.

3. The method of claim 2, wherein increasing the frequency at which the parallel-computing application is checkpointed involves increasing the frequency by a factor which is inversely proportional to the value of the health index.

4. The method of claim 2, wherein prior to executing the parallel-computing application, the method further comprises initializing the health index to 1, which indicates that the parallel-processing system is healthy, wherein the probability that the parallel-processing system will fail increases as the health index approaches 0.

5. The method of claim 1, wherein the method further comprises:

determining if a checkpoint time interval has elapsed; and

if so, performing a checkpointing operation to save the state of the distributed parallel-computing application.

6. The method of claim 1, wherein if the parallel-processing system fails, the method further comprises:

using the checkpoint to restore the state of the parallel-processing system to the state of the last checkpoint; and

resuming execution of the parallel-computing application from the restored state.

7. The method of claim 1, wherein the telemetry signals can include hardware variables and/or software variables.

8. The method of claim 7, wherein the software variables include load metrics, CPU utilization, idle time, memory utilization, disk activity, transaction latencies, and other performance metrics reported by the operating system.

9. The method of claim 7, wherein the hardware variables include temperature and voltage.

10. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for providing fault-tolerance in a parallel-processing system, the method comprising:

executing a parallel-computing application in parallel across a subset of computing nodes within the parallel-processing system;

monitoring telemetry signals within the parallel-processing system;

analyzing the monitored telemetry signals to determine if the probability that the parallel-processing system will fail is increasing; and

if so, increasing the frequency at which the parallel-computing application is checkpointed, wherein a checkpoint includes the state of the parallel-computing application at each computing node within the parallel-processing system.

11. The computer-readable storage medium of claim 10, wherein analyzing the monitored telemetry signals to determine if the probability that the parallel-processing system will fail is increasing involves:

using the monitored telemetry signals to generate a health index, which indicates the risk that the parallel-processing system will fail; and

determining if the health index indicates that the probability that the parallel-processing system will fail is increasing.

12. The computer-readable storage medium of claim 11, wherein increasing the frequency at which the parallel-computing application is checkpointed involves increasing the frequency by a factor which is inversely proportional to the value of the health index.

13. The computer-readable storage medium of claim 11, wherein prior to executing the parallel-computing application, the method further comprises initializing the health index to 1, which indicates that the parallel-processing system is healthy, wherein the probability that the parallel-processing system will fail increases as the health index approaches 0.

14. The computer-readable storage medium of claim 10, wherein the method further comprises:

determining if a checkpoint time interval has elapsed; and

if so, performing a checkpointing operation to save the state of the distributed parallel-computing application.

15. The computer-readable storage medium of claim 10, wherein if the parallel-processing system fails, the method further comprises:

using the checkpoint to restore the state of the parallel-processing system to the state of the last checkpoint; and

resuming execution of the parallel-computing application from the restored state.

16. The computer-readable storage medium of claim 10, wherein the telemetry signals can include hardware variables and/or software variables.

17. The computer-readable storage medium of claim 16, wherein the software variables include load metrics, CPU utilization, idle time, memory utilization, disk activity, transaction latencies, and other performance metrics reported by the operating system.

18. The computer-readable storage medium of claim 16, wherein the hardware variables include temperature and voltage.

19. An apparatus that provides fault-tolerance in a parallel-processing system, comprising:

an execution mechanism configured to execute a parallel-computing application in parallel across a subset of computing nodes within the parallel-processing system;

a health-monitoring mechanism configured to monitor telemetry signals within the parallel-processing system;

a checkpointing mechanism configured to;

analyze the monitored telemetry signals to determine if the probability that the parallel-processing system will fail is increasing; and

if so, to increase the frequency at which the parallel-computing application is checkpointed, wherein a checkpoint includes the state of the parallel-computing application at each computing node within the parallel-processing system.

20. The apparatus of claim 19, wherein while analyzing the monitored telemetry signals to determine if the probability that the parallel-processing system will fail is increasing, the checkpointing mechanism is configured to:

use the monitored telemetry signals to generate a health index, which indicates the risk that the parallel-processing system will fail; and

to determine if the health index indicates that the probability that the parallel-processing system will fail is increasing.