HEALTH MONITORING AND RECOVERY FOR INFRASTRUCTURE DEVICES

• US 20150212901A1
• Filed: 01/27/2014
• Published: 07/30/2015
• Est. Priority Date: 01/27/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: AGGARWAL, CHANDAN, YAQOOB, ASAD, MCKONE, JOSH DAVID, EASON, MATTHEW JEREMIAH, MERCHANT, AKIL M.
• Granted Patent: US 9,430,318 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3096 wherein the means or processing minimize the use of computing system or of computing system component resources, e.g. non-intrusive monitoring which minimizes the probe effect: sniffing, intercepting, indirectly deriving the monitored data from other directly available data

Abstract

Automated health monitoring and recovery is provided for infrastructure devices supporting server devices in a data center. Health analysis operations may be selected to be performed on an infrastructure device based on the capabilities of the infrastructure device and/or how the infrastructure device is being used to support server devices in the data center. If the infrastructure device is unhealthy, an automated recovery operation may be performed. The automated recovery operation may include recovery actions selected based on the capabilities of the infrastructure device, the failure mode of the infrastructure device, and/or how the infrastructure device is being used to support server devices in the data center.

Claims

1. One or more computer storage media storing computer-usable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:

triggering a health monitoring event for an infrastructure device supporting one or more server devices in a data center;

identifying device information for the infrastructure device;

determining an operational context of the infrastructure device in supporting the one or more server devices in the data center;

determining a health monitoring process for the infrastructure device based on the device information for the infrastructure device and the operational context of the infrastructure device in supporting the one or more server devices in the data center;

performing the determined health monitoring process for the infrastructure device to assess the health of the infrastructure device;

determining to perform an automated recovery operation for the infrastructure device based on the health of the infrastructure device;

in response to determining to perform the automated recovery operation for the infrastructure device, determining one or more recovery actions for the automated recovery operation based on the device information for the infrastructure device, the operational context of the infrastructure device in supporting the one or more server devices in the data center, and a failure context of the infrastructure device determined from the health monitoring process for the infrastructure device; and

performing at least a portion of the one or more recovery actions for the infrastructure device.

2. The one or more computer storage media of claim 1, wherein the health monitoring event is triggered based at least in part on determining a pre-determined time has elapsed in accordance with a scheduled basis for monitoring the health of the infrastructure device.

3. The one or more computer storage media of claim 1, wherein the health monitoring event is triggered based at least in part on registering information indicating the infrastructure device may be unhealthy.

4. The one or more computer storage media of claim 1, wherein the device information for the infrastructure device comprises at least one selected from the following:

a device type of the infrastructure device, a manufacture of the infrastructure device, and a model of the infrastructure device.

5. The one or more computer storage media of claim 1, wherein determining the health monitoring process for the infrastructure device comprises:

determining a plurality of available health analysis operations for the infrastructure device; and

selecting one or more of the available health analysis operations for the health monitoring process based on the operational context of the infrastructure device.

6. The one or more computer storage media of claim 1, wherein determining the health monitoring process for the infrastructure device includes determining to delay at least a portion of the health monitoring process for the infrastructure device until the operational context for the infrastructure device changes.

7. The one or more computer storage media of claim 1, wherein determining the need to perform the automated recovery operation based on the health of the infrastructure device includes determining to delay at least a portion of the automated recovery operation until the operational context for the infrastructure device changes.

8. The one or more computer storage media of claim 1, wherein determining the one or more recovery actions for the automated recovery operation comprises:

determining a plurality of available recovery actions for the infrastructure device; and

selecting the one or more recovery actions from the available recovery actions based on the operational context of the infrastructure device.

9. The one or more computer storage media of claim 1, wherein the operations further include:

monitoring operation of at least one of the one or more server devices when the one or more recovery actions are performed;

detecting an adverse change in the operation of at least one of the one or more server devices; and

in response to detecting the adverse change, modifying the one or more recovery actions.

10. The one or more computer storage media of claim 1, wherein the operations further include:

determining that the automated recovery operation is unsuccessful in recovering the infrastructure device; and

prompting human intervention to recovery the infrastructure device.

11. One or more computer storage media storing computer-usable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:

triggering a health monitoring event for an infrastructure device supporting one or more server devices in a data center;

determining a health monitoring process for the infrastructure device based on at least one selected from the following;

device information for the infrastructure device, and an operational context of the infrastructure device in supporting the one or more server devices in the data center; and

performing the determined health monitoring process for the infrastructure device to assess the health of the infrastructure device.

12. The one or more computer storage media of claim 11, wherein the health monitoring event is triggered based at least in part on determining a pre-determined time has elapsed in accordance with a scheduled basis for monitoring the health of the infrastructure device.

13. The one or more computer storage media of claim 11, wherein the health monitoring event is triggered based at least in part on registering information indicating the infrastructure device may be unhealthy.

14. The one or more computer storage media of claim 11, wherein determining the health monitoring process for the infrastructure device comprises:

determining a plurality of available health analysis operations for the infrastructure device; and

selecting one or more of the available health analysis operations for the health monitoring process based on the operational context of the infrastructure device.

15. The one or more computer storage media of claim 11, wherein determining the health monitoring process for the infrastructure device includes determining to delay at least a portion of the health monitoring process for the infrastructure device until the operational context for the infrastructure device changes.

16. A system comprising:

one or more processors; and

one or more computer storage media storing computer-usable instructions that, when used by the one or more processors, cause the one or more processors to;

determine to perform an automated recovery operation for an infrastructure device supporting one or more server devices in a data center;

in response to determining to perform the automated recovery operation for the infrastructure device, determine one or more recovery actions for the automated recovery operation based on at least on selected from the following;

device information for the infrastructure device, an operational context of the infrastructure device in supporting the one or more server devices in the data center, and a failure context of the infrastructure device; and

perform at least a portion of the one or more recovery actions for the infrastructure device.

17. The system of claim 16, wherein determining to perform the automated recovery operation based on the health of the infrastructure device includes determining to delay at least a portion of the automated recovery operation until the operational context for the infrastructure device changes.

18. The system of claim 16, wherein determining the one or more recovery actions for the automated recovery operation comprises:

determining a plurality of available recovery actions for the infrastructure device; and

selecting the one or more recovery actions from the available recovery actions based on the operational context of the infrastructure device.

19. The system of claim 16, wherein the instructions further cause the one or more processors to:

monitor operation of at least one of the one or more server devices when the one or more recovery actions are performed;

detect an adverse change in the operation of at least one of the one or more server devices; and

in response to detecting the adverse change, modify the one or more recovery actions.

20. The system of claim 16, wherein the instructions further cause the one or more processors to:

determine that the automated recovery operation is unsuccessful in recovering the infrastructure device; and

prompt human intervention to recovery the infrastructure device.