Managing faults in a high availability system

• US 10,346,230 B2
• Filed: 06/28/2017
• Issued: 07/09/2019
• Est. Priority Date: 11/26/2013
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Jayaraman, Arunachalam
• Primary Examiner(s): Ko, Chae M
• Application Number:
  US15/635,684
• Publication Number:
  US 20170300373A1
• Time in Patent Office: 741 Days
• Field of Search: 714 472, 714 473
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3419 by assessing time

Abstract

An approach is provided for managing a failure of a critical high availability (HA) component. Weights are received and assigned to categories of critical HA components in a HA system. A current value indicating a performance of a component included in the identified components is obtained by periodically monitoring the components. A reference value for the performance of the component is received. A deviation between the current value and the reference value is determined. Based on the deviation, the component is determined to have failed. Based in part on the failed component, the categories, and the weights, a health index is determined in real-time. The health index indicates in part how much the component having failed affects a measure of health of the HA system.

Claims

1. A method of managing a failure of a critical high availability (HA) component, the method comprising the steps of:

a computer receiving and assigning weights to categories of a plurality of critical HA components of a HA system;

the computer obtaining a current value indicating a performance of a critical HA component included in the plurality of critical HA components, the current value obtained by periodically monitoring the plurality of critical HA components;

the computer receiving a reference value for the performance of the critical HA component;

the computer determining a deviation between the current value and the reference value;

based on the deviation, the computer determining the critical HA component has failed; and

based in part on the failed critical HA component, the categories, and the weights, the computer determining a health index in real-time, the health index indicating in part how much the critical HA component having failed affects a measure of health of the HA system.

2. The method of claim 1, further comprising the steps of:

the computer determining n counts C₁ . . . C_n, wherein C_i in C₁ . . . C_n is a count of how many of the critical HA components are assigned to an i-th category of the categories assigned to the critical HA components; and

the computer determining n counts FC₁ . . . FC_n, wherein FC_i in FC₁ . . . FC_n is a count of how many of the critical HA components that are assigned in the i-th category have a failed status,

wherein the categories consist of n categories, wherein the weights consist of n weights W₁ . . . W_n, wherein n is an integer greater than one, wherein the step of determining the health index includes the step of the computer determining;

{{{C₁×

W₁)+ . . . +(C_n×

W_n)}−

{(FC₁×

W₁)+ . . . +(FC_n×

W_n)}}/{C₁×

W₁)+ . . . +(C_n×

W_n}}*100%.

3. The method of claim 1, further comprising the steps of:

the computer determining the health index is not equal to 100%;

based on the health index not being equal to 100%, the computer receiving indexes corresponding to respective critical HA components, each index indicating whether the respective critical HA component is an immediately recoverable component;

based on the received index corresponding to the critical HA component that has failed, the computer determining whether the critical HA component that has failed is an immediately recoverable component, in response to a determination that the critical HA component that has failed is the immediately recoverable component, the computer performing branch based decision making to automatically recover the critical HA component, and in response to a determination that the critical HA component that has failed is not the immediately recoverable component, the computer generating a ticket indicating a need for human intervention to recover the critical HA component.

4. The method of claim 3, wherein the step of determining whether the critical HA component that has failed is the immediately recoverable component includes determining whether the critical HA component is a hardware component or a software component, the critical HA component being the hardware component indicating the critical HA component is not the immediately recoverable component, and the critical HA component being the software component indicating the critical HA component is the immediately recoverable component.

5. The method of claim 3, wherein the step of performing the branch based decision making includes the steps of:

the computer determining whether the critical HA component is assigned to a leaf category included in the categories, wherein the categories consist of the leaf category, a stem category, and a root category;

in response to a determination that the critical HA component is assigned to the leaf category, the computer automatically recovering the critical HA component by invoking a self-healing script for the leaf category;

in response to a determination that the critical HA component is not assigned to the leaf category, the computer determining whether the critical HA component is assigned to the stem category;

in response to a determination that the critical HA component is assigned to the stem category, the computer automatically recovering the critical HA component by invoking a self-healing script for the stem category;

in response to a determination that the critical HA component is not assigned to the stem category, the computer determining whether the critical HA component is assigned to the root category;

in response to a determination that the critical HA component is assigned to the root category, the computer automatically recovering the critical HA component by invoking a self-healing script for the root category; and

in response to a determination that the critical HA component is not assigned to the root category, the computer generating a ticket indicating a self-healing of the critical HA component has failed.

6. The method of claim 5, further comprising the steps of:

subsequent to recovering the critical HA component by invoking the self-healing script for the leaf, stem or root category, the computer identifying a peer index of the critical HA component;

based on the peer index, the computer determining a peer of the critical HA component, the peer index indicating the peer and the critical HA component have substantially similar functions in a subsystem;

the computer determining whether a weight assigned to the peer indicates the peer is in the stem, leaf, or root category;

in response to a determination that the weight assigned to the peer indicates the peer is in the stem category,

the computer traversing up a branch from the peer to one or more critical HA components in the leaf category, performing the branch based decision making on the one or more critical HA components to determine if any of the one or more critical HA components have failed, and performing a self-healing to recover any of the one or more critical HA components that have failed; and

the computer traversing down the branch from the peer to a critical HA component in the root category, performing the branch based decision making to determine if the critical HA component in the root category has failed, and performing a self-healing to recover the critical HA component in the root category that has failed;

in response to a determination that the weight assigned to the peer indicates the peer is in the leaf category, the computer traversing down the branch from the peer to a critical HA component in the stem category, performing the branch based decision making to determine if the critical HA component in the stem category has failed, and performing a self-healing to recover the critical HA component in stem category that has failed; and

in response to a determination that the weight assigned to the peer indicates the peer is in the root category, the computer traversing up the branch from the peer to one or more critical HA components in the stem category, performing the branch based decision making to determine if any of the one or more critical HA components in the stem category have failed, and performing a self-healing to recover any of the one or more critical HA components in the stem category that have failed.

7. The method of claim 1, further comprising the step of:

providing at least one support service for at least one of creating, integrating, hosting, maintaining, and deploying computer-readable program code in the computer, the program code being executed by a processor of the computer to implement the steps of receiving and assigning weights to the categories, obtaining the current value indicating the performance of the critical HA component, receiving the reference value for the performance of the critical HA component, determining the deviation between the current value and the reference value, determining the critical HA component has failed, and determining the health index in real-time.

8. The computer of claim 1, one or more processors of the computer, one or more computer-readable memories, one or more computer-readable storage devices, other computer hardware, and program instructions which are stored on the one or more storage devices for execution by the one or more processors via the one or more memories and when executed by the one or more processors with the other computer hardware perform the method of claim 1.

9. A computer program product comprising:

a computer-readable, tangible storage device; and

a computer-readable program code stored in the computer-readable, tangible storage device, the computer-readable program code containing instructions that are executed by a central processing unit (CPU) of a computer system to implement a method of managing a failure of a critical high availability (HA) component, the method comprising the steps of;

the computer system receiving and assigning weights to categories of a plurality of critical HA components of a HA system;

the computer system obtaining a current value indicating a performance of a critical HA component included in the plurality of critical HA components, the current value obtained by periodically monitoring the plurality of critical HA components;

the computer system receiving a reference value for the performance of the critical HA component;

the computer system determining a deviation between the current value and the reference value;

based on the deviation, the computer system determining the critical HA component has failed; and

based in part on the failed critical HA component, the categories, and the weights, the computer system determining a health index in real-time, the health index indicating in part how much the critical HA component having failed affects a measure of health of the HA system.

10. The program product of claim 9, wherein the method further comprises the steps of:

the computer system determining n counts C₁ . . . C_n, wherein C_i in C₁ . . . C_n is a count of how many of the critical HA components are assigned to an i-th category of the categories assigned to the critical HA components; and

the computer system determining n counts FC₁ . . . FC_n, wherein FC_i in FC₁ . . . FC_n is a count of how many of the critical HA components that are assigned in the i-th category have a failed status,

wherein the categories consist of n categories, wherein the weights consist of n weights W₁ . . . W_n, wherein n is an integer greater than one, wherein the step of determining the health index includes the step of the computer system determining;

{{{C₁×

W₁)+ . . . +(C_n×

W_n)}−

{(FC₁×

W₁)+ . . . +(FC_n×

W_n)}}/{C₁×

W₁)+ . . . +(C_n×

W_n}}*100%.

11. The program product of claim 9, wherein the method further comprises the steps of:

the computer system determining the health index is not equal to 100%;

based on the health index not being equal to 100%, the computer system receiving indexes corresponding to respective critical HA components, each index indicating whether the respective critical HA component is an immediately recoverable component;

based on the received index corresponding to the critical HA component that has failed, the computer system determining whether the critical HA component that has failed is an immediately recoverable component, in response to a determination that the critical HA component that has failed is the immediately recoverable component, the computer system performing branch based decision making to automatically recover the critical HA component, and in response to a determination that the critical HA component that has failed is not the immediately recoverable component, the computer system generating a ticket indicating a need for human intervention to recover the critical HA component.

12. The program product of claim 11, wherein the step of determining whether the critical HA component that has failed is the immediately recoverable component includes determining whether the critical HA component is a hardware component or a software component, the critical HA component being the hardware component indicating the critical HA component is not the immediately recoverable component, and the critical HA component being the software component indicating the critical HA component is the immediately recoverable component.

13. The program product of claim 11, wherein the step of performing the branch based decision making includes the steps of:

the computer system determining whether the critical HA component is assigned to a leaf category included in the categories, wherein the categories consist of the leaf category, a stem category, and a root category;

in response to a determination that the critical HA component is assigned to the leaf category, the computer system automatically recovering the critical HA component by invoking a self-healing script for the leaf category;

in response to a determination that the critical HA component is not assigned to the leaf category, the computer system determining whether the critical HA component is assigned to the stem category;

in response to a determination that the critical HA component is assigned to the stem category, the computer system automatically recovering the critical HA component by invoking a self-healing script for the stem category;

in response to a determination that the critical HA component is not assigned to the stem category, the computer system determining whether the critical HA component is assigned to the root category;

in response to a determination that the critical HA component is assigned to the root category, the computer system automatically recovering the critical HA component by invoking a self-healing script for the root category; and

in response to a determination that the critical HA component is not assigned to the root category, the computer system generating a ticket indicating a self-healing of the critical HA component has failed.

14. The program product of claim 13, wherein the method further comprises the steps of:

subsequent to recovering the critical HA component by invoking the self-healing script for the leaf, stem or root category, the computer system identifying a peer index of the critical HA component;

based on the peer index, the computer system determining a peer of the critical HA component, the peer index indicating the peer and the critical HA component have substantially similar functions in a subsystem;

the computer system determining whether a weight assigned to the peer indicates the peer is in the stem, leaf, or root category;

in response to a determination that the weight assigned to the peer indicates the peer is in the stem category,

the computer system traversing up a branch from the peer to one or more critical HA components in the leaf category, performing the branch based decision making on the one or more critical HA components to determine if any of the one or more critical HA components have failed, and performing a self-healing to recover any of the one or more critical HA components that have failed; and

the computer system traversing down the branch from the peer to a critical HA component in the root category, performing the branch based decision making to determine if the critical HA component in the root category has failed, and performing a self-healing to recover the critical HA component in the root category that has failed;

in response to a determination that the weight assigned to the peer indicates the peer is in the leaf category, the computer system traversing down the branch from the peer to a critical HA component in the stem category, performing the branch based decision making to determine if the critical HA component in the stem category has failed, and performing a self-healing to recover the critical HA component in stem category that has failed; and

in response to a determination that the weight assigned to the peer indicates the peer is in the root category, the computer system traversing up the branch from the peer to one or more critical HA components in the stem category, performing the branch based decision making to determine if any of the one or more critical HA components in the stem category have failed, and performing a self-healing to recover any of the one or more critical HA components in the stem category that have failed.