SYSTEM AND METHOD FOR USING FAILURE CASTING TO MANAGE FAILURES IN COMPUTER SYSTEMS

• US 20080244585A1
• Filed: 03/27/2007
• Published: 10/02/2008
• Est. Priority Date: 03/27/2007
• Status: Active Grant

• Current Assignee: Teradata US, Inc. (Teradata Corporation)
• Original Assignee: Aster Data Systems, Inc. (Teradata Corporation)
• Inventors: Bawa, Mayank, Candea, George, Argyros, Anastasios
• Granted Patent: US 8,359,495 B2
• Time in Patent Office: Days
• US Class Current: 718/102
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level

Abstract

A system and method for using failure casting to manage failures in computer system. In accordance with an embodiment, the system uses a failure casting hierarchy to cast failures of one type into failures of another type. In doing this, the system allows incidents, problems, or failures to be cast into a (typically smaller) set of failures, which the system knows how to handle. In accordance with a particular embodiment, failures can be cast into a category that is considered reboot-curable. If a failure is reboot-curable then rebooting the system will likely cure the problem. Examples include hardware failures, and reboot-specific methods that can be applied to disk failures and to failures within clusters of databases. The system can even be used to handle failures that were hitherto unforeseen—failures can be cast into known failures based on the failure symptoms, rather than any underlying cause.

Claims

1. A system for managing failures in a computer system using failure casting comprising:

a system manager that performs actions on a computer system to address failures that occur within the computer system;

a failure casting logic that detects failures as they occur in the computer system; and

a failure casting hierarchy that defines a plurality of the failures that can occur within the computer system, and which is used by the failure casting logic upon detecting the occurrence of a failure to cast the failure from a first failure type to a second failure type, wherein the second failure type is then communicated to the system manager to allow the system manager to treat the failure as if it were the second failure type.

2. The system of claim 1 wherein the actions performed by the system manager can include restarting a component of the computer system that includes the failure.

3. The system of claim 1 wherein the action that is performed by the system manager is that of rebooting the computer system as a whole, to automatically correct the failure.

4. The system of claim 1 wherein the failure casting hierarchy defines at least two sets of failures, including a set of reboot-curable failures, and a set of non-reboot-curable failures, wherein the reboot-curable failures are addressed by the system manager by rebooting the computer system or component thereof that includes the failure.

5. The system of claim 4 wherein the failure casting logic and the failure casting hierarchy are part of a script that detects the occurrence of failures in the computer system and then casts the failure into one of either a reboot-curable failure, or non-reboot-curable failure.

6. The system of claim 5 wherein the script is executed by the computer system when powered on.

7. The system of claim 5 wherein the script can be executed by the computer system at any time during runtime.

8. The system of claim 6 wherein the computer system includes an array of disks and wherein the script is used to address failures within the array of disks at boot time by verifying the health of each disk prior to adding the disk to the array.

9. The system of claim 8 wherein the failure casting hierarchy in the script includes the set of non-reboot curable failures that are checked at boot time, and if the disk, when added to the array, exhibits a failure upon bootup within the set of non-reboot-curable failures, then the disk is not added to the array.

10. The system of claim 1 wherein the computer system comprises

a cluster of nodes;

a global database distributed over the cluster of nodes;

a script at each node that detects the occurrence of failures in the node and casts some of the failures into reboot-curable failures; and

wherein the system manager upon receiving a request to access the global database, and subsequently detecting a failure of a node, causes the failed node to reboot, and then applies the request to the remaining nodes in the cluster.

11. A method for managing failures in a computer system using failure casting comprising the steps of:

detecting the occurrence of failures in the computer system;

referring to a failure casting hierarchy that defines a plurality of failures that can occur within the computer system;

using the failure casting hierarchy to cast the failure from a first failure type to a second failure type;

communicating the second failure type to a system manager; and

performing an action by the system manager on the computer system to address the failure including treating the failure as if it were the second failure type.

12. The method of claim 11 wherein the actions performed by the system manager can include restarting a component of the computer system that includes the failure.

13. The method of claim 11 wherein the action that is performed by the system manager is that of rebooting the computer system as a whole, to automatically correct the failure.

14. The method of claim 11 wherein the failure casting hierarchy defines at least two sets of failures, including a set of reboot-curable failures, and a set of non-reboot-curable failures, wherein the reboot-curable failures are addressed by the system manager by restarting the computer system or component thereof that includes the failure.

15. The method of claim 14 wherein the failure casting logic and the failure casting hierarchy are part of a script that detects the occurrence of failures in the computer system and then casts the failure into one of either a reboot-curable failure, or non-reboot-curable failure.

16. The method of claim 15 wherein the script is executed by the computer system when first powered on.

17. The method of claim 15 wherein the script can be executed by the computer system at any time during runtime.

18. The method of claim 16 wherein the computer system includes an array of disks and wherein the script is used to address failures within the array of disks at boot time by verifying the health of each disk prior to adding the disk to the array.

19. The method of claim 18 wherein the failure casting hierarchy in the script includes the set of non-reboot curable failures that are checked at boot time, and if the disk, when added to the array exhibits a failure upon bootup within the set of non-reboot-curable failures, then the disk is not added to the array.

20. The method of claim 19 wherein the computer system comprises

a cluster of nodes;

a global database distributed over the cluster of nodes;

a script at each node that detects the occurrence of failures in the node and casts some of the failures into reboot-curable failures; and

wherein the system manager upon receiving a request to access the global database, and subsequently detecting a failure of a node, causes the failed node to reboot, and then applies the request to the remaining nodes in the cluster.

21. A system for managing failures in a computer system using failure casting comprising:

a system manager that performs actions on a computer system to address failures that occur within the computer system; and

a script that detects the occurrence of a failure in the computer system and then uses a failure casting hierarchy within the script to cast the failure from a first failure type to a second failure type, wherein the second failure type is then communicated to the system manager to allow the system manager to treat the failure as if it was the second failure type.

22. The system of claim 21 wherein the script is executed by the computer system when first powered on.

23. The system of claim 21 wherein the script can be executed by the computer system at any time during runtime.

24. The system of claim 21 wherein the computer system includes an array of disks and wherein the script is used to address failures within the array of disks at boot time by verifying the health of each disk prior to adding the disk to the array.

25. A method for managing failures in a computer system using failure casting comprising the steps of:

providing a system manager that performs actions on a computer system to address failures that occur within the computer system; and

executing a script that detects the occurrence of a failure in the computer system and then uses a failure casting hierarchy within the script to cast the failure from a first failure type to a second failure type, wherein the second failure type is then communicated to the system manager to allow the system manager to treat the failure as if it was the second failure type.

26. The method of claim 25 wherein the script is executed by the computer system when first powered on.

27. The method of claim 25 wherein the script can be executed by the computer system at any time during runtime.

28. The method of claim 25 wherein the computer system includes an array of disks and wherein the script is used to address failures within the array of disks at boot time by verifying the health of each disk prior to adding the disk to the array.

29. A system readable medium, including instructions stored thereon, which when executed by a system cause the system to perform the steps of:

detecting the occurrence of failures in the system;

using a failure casting hierarchy to cast the failure from a first failure type to a second failure type, wherein the failure casting hierarchy defines a plurality of failures that can occur within the system;

performing an action on the system to address the failure, including treating the failure as if it were the second failure type.

30. The system readable medium of claim 29 wherein the failure casting hierarchy defines at least two sets of failures, including a set of reboot-curable failures, and a set of non-reboot-curable failures, wherein the reboot-curable failures are addressed by restarting the system or component thereof that includes the failure.

31. The system readable medium, of claim 29 wherein the system includes an array of disks and wherein the script is used to address failures within the array of disks at boot time by verifying the health of each disk prior to adding the disk to the array, and wherein if the disk, when added to the array exhibits a failure upon bootup within the set of non-reboot-curable failures, then the disk is not added to the array.

32. The system readable medium, of claim 29 wherein the system comprises

a cluster of nodes;

a global database distributed over the cluster of nodes;

a script at each node that detects the occurrence of failures in the node and casts some of the failures into reboot-curable failures; and

wherein the system upon receiving a request to access the global database, and subsequently detecting a failure of a node, causes the failed node to reboot, and then applies the request to the remaining nodes in the cluster.

33. A system readable medium, including instructions stored thereon, which when executed by a system cause the system to perform the steps of:

executing a script that detects the occurrence of a failure in the computer system; and

using a failure casting hierarchy within the script to cast the failure from a first failure type to a second failure type, wherein the second failure type is then communicated to the system manager to allow the system manager to treat the failure as if it was the second failure type.

34. The system readable medium of claim 33 wherein the script defines at least two sets of failures, including a set of reboot-curable failures, and a set of non-reboot-curable failures, wherein the reboot-curable failures are addressed by restarting the system or component thereof that includes the failure.

35. The system readable medium of claim 33 wherein the system includes an array of disks and wherein the script is used to address failures within the array of disks at boot time by verifying the health of each disk prior to adding the disk to the array, and wherein if the disk, when added to the array exhibits a failure upon bootup within the set of non-reboot-curable failures, then the disk is not added to the array.

36. The system readable medium of claim 33 wherein the system comprises

a cluster of nodes;

a global database distributed over the cluster of nodes;

a script at each node that detects the occurrence of failures in the node and casts some of the failures into reboot-curable failures; and

wherein the system upon receiving a request to access the global database, and subsequently detecting a failure of a node, causes the failed node to reboot, and then applies the request to the remaining nodes in the cluster.