Automated Datacenter Network Failure Mitigation

• US 20160020940A1
• Filed: 04/29/2015
• Published: 01/21/2016
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Maltz, David A., Yuan, Lihua, Zhang, Ming, Chen, Chao-Chih, Wu, Xin, Turner, Daniel Joseph
• Granted Patent: US 10,075,327 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/0883 Semiautomatic configuration, e.g. proposals from system
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/147 for predicting network behaviour
• CPC Class Codes: H04L 41/149 for prediction of maintenance
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/0823 Errors, e.g. transmission errors

Abstract

The subject disclosure is directed towards a technology that automatically mitigates datacenter failures, instead of relying on human intervention to diagnose and repair the network. Via a mitigation pipeline, when a network failure is detected, a candidate set of components that are likely to be the cause of the failure is identified, with mitigation actions iteratively targeting each component to attempt to alleviate the problem. The impact to the network is estimated to ensure that the redundancy present in the network will be able to handle the mitigation action without adverse disruption to the network.

Claims

1. A method performed at least in part by at least one processor, comprising:

monitoring a network;

detecting a failure;

determining a component set corresponding to the failure, in which the component set comprises one or more suspected faulty components; and

taking automated action on the component set to mitigate the failure, including when the component set comprises a plurality of components, iterating through one or more of the components applying one or more mitigation actions until the failure is mitigated.

2. The method of claim 1 further comprising, selecting a proposed mitigation action, and estimating an impact on the network before the proposed mitigation action is taken.

3. The method of claim 2 wherein estimating the impact comprises estimating a result of restarting or deactivating a component of the network.

4. The method of claim 3 wherein estimating the impact comprises using at least some historical data.

5. The method of claim 3 wherein estimating the result comprises processing forwarding data to determine at least one hop to a destination.

6. The method of claim 5 further comprising, learning the forwarding data.

7. The method of claim 1 wherein detecting a failure comprises aggregating failure information corresponding to the failure to determine any ongoing mitigation of the failure.

8. The method of claim 1 further comprising providing a plan comprising at least one mitigation action, and wherein taking the action comprises executing the plan.

9. The method of claim 1 wherein taking the action comprises restarting or deactivating at least one component of the component set.

10. The method of claim 1 wherein the failure corresponds to a link-down failure or uneven-split failure, and wherein providing the mitigation plan comprises deactivating or restarting a plurality of link aggregation groups.

11. A system comprising:

a failure detector configured to process network state data to determine a state indicative of a network failure;

a planner configured to determine a mitigation plan for mitigating the network failure, including a plan to iterate through a plurality of suspected faulty components to apply one or more mitigation actions to one or more components until the failure is mitigated, the planner coupled to an impact estimator configured to determine an impact if an action is taken, the planner further configured to adjust the plan based upon the impact; and

a plan executor, the plan executor configured to access the mitigation plan and take one or more actions identified in the plan on a network component set comprising the plurality of suspected faulty components to mitigate the failure.

12. The system of claim 11 wherein the failure detector is coupled to a failure aggregator, the failure aggregator configured to decide whether a detected failure instance corresponds to a new or ongoing failure.

13. The system of claim 11 further comprising a database, the planner coupled to the database to obtain failure data corresponding to the failure and to output the plan to the database.

14. The system of claim 11 wherein the mitigation plan includes a command to restart or deactivate a component, or to deactivate a component if an attempt to restart the component fails to mitigate the failure.

15. The system of claim 11 wherein the component set includes a switch, and wherein the mitigation plan includes a command to restart or deactivate the switch or switch ports, or any combination of commands to restart or deactivate the switch or switch ports

16. The system of claim 11 wherein the failure detector monitors SNMP data and one or more performance counters, or both, to detect a failure state comprising at least one of:

a link layer loop failure, an imbalance triggered overload failure, a frame checksum error, a switch forwarding failure, a lost configuration failure, high CPU utilization, or any other errors on a switch.

17. One or more computer-readable devices having computer-executable instructions, which when executed by at least one computer perform steps, comprising:

a) determining that a network failure corresponding to a component set has occurred;

b) providing a mitigation plan, the mitigation plan comprising one or more mitigation actions that if taken on one or more suspected faulty components of the component set are likely to mitigate the failure;

c) estimating whether a selected action of the mitigation plan, if taken on a component, will adversely impact the network, and if so, discarding that action, and if not, keeping the action for execution;

d) performing the selected action on a suspected faulty component and determining whether the action mitigated the failure, and if so, advancing to step e), and if not, returning to step c) to select another action until the failure is mitigated or no other action in the plan remains to be performed; and

e) recording or updating, or both recording and updating, information regarding the failure, the plan, and one or more mitigation-related actions.

18. The one or more computer-readable devices of claim 17 having further computer-executable instructions comprising, aggregating failure instances to decide whether a detected failure instance corresponds to a new or ongoing failure.

19. The one or more computer-readable devices of claim 17 wherein the failure corresponds to a link-down or uneven-split failure, and wherein providing the mitigation plan comprises deactivating or restarting a plurality of link aggregation groups.

20. The one or more computer-readable devices of claim 17 wherein the failure corresponds to a frame checksum error, and having further computer-executable instructions comprising, using a frame checksum error propagation model to localize one or more faulty links.