Automated datacenter network failure mitigation

• US 10,075,327 B2
• Filed: 04/29/2015
• Issued: 09/11/2018
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Maltz, David A., Yuan, Lihua, Zhang, Ming, Wu, Xin, Turner, Daniel Joseph, Chen, Chao-Chih
• Primary Examiner(s): Jung, Min
• Application Number:
  US14/700,128
• Publication Number:
  US 20160020940A1
• Time in Patent Office: 1,231 Days
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/0883 Semiautomatic configuration, e.g. proposals from system
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/147 for predicting network behaviour
• CPC Class Codes: H04L 41/149 for prediction of maintenance
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/0823 Errors, e.g. transmission errors

Abstract

The subject disclosure is directed towards a technology that automatically mitigates datacenter failures, instead of relying on human intervention to diagnose and repair the network. Via a mitigation pipeline, when a network failure is detected, a candidate set of components that are likely to be the cause of the failure is identified, with mitigation actions iteratively targeting each component to attempt to alleviate the problem. The impact to the network is estimated to ensure that the redundancy present in the network will be able to handle the mitigation action without adverse disruption to the network.

Claims

1. A method performed at least in part by at least one processor, comprising:

monitoring a network;

determining a network component set corresponding to abnormal behavior, in which the network component set comprises a plurality of links; and

taking automated action on the network component set to mitigate the abnormal behavior by;

identifying a plurality of proposed actions;

estimating an impact on the network for executing each of the proposed actions;

estimating a success rate for each of the proposed actions selecting one of more of the proposed actions based at least on the estimated impact and the estimated success rate; and

applying the one or more of the proposed actions based at least on the estimated impact and the estimated success rate.

2. The method of claim 1, further comprising:

selecting two or more of the proposed actions based at least on the estimated impact and the estimated success rate.

3. The method of claim 2, wherein estimating the impact comprises estimating a result of restarting or deactivating a component of the network.

4. The method of claim 3, wherein estimating the impact comprises using at least some historical data.

5. The method of claim 1, wherein the abnormal behavior is a link failure that causes traffic latency, traffic overload, or traffic imbalance between a first link and a second link in the plurality of links.

6. The method of claim 5, further comprising, restarting the first link or the second link as a result of determining the abnormal behavior causes traffic imbalance.

7. The method of claim 5, further comprising restarting a switch corresponding to the first link or the second link as a result of determining the abnormal behavior.

8. The method of claim 1, further comprising providing a plan comprising at least one action, and wherein taking the action comprises executing the plan.

9. The method of claim 1, wherein taking the action comprises restarting or deactivating at least one component of the network component set.

10. The method of claim 1, wherein the abnormal behavior corresponds to a link-down failure or uneven-split failure, and wherein providing the plan comprises deactivating or restarting one of the plurality of links.

11. A system comprising:

a detector configured to process network state data for a network to determine a state indicative of abnormal behavior in a network component set comprising a plurality of links;

a planner configured to determine a plan for mitigating the abnormal behavior by identifying a plurality of proposed actions, the planner coupled to an impact estimator configured to;

estimate an impact on the network for executing each of the proposed actions; and

estimate a success rate for each of the proposed actions; and

a plan executor, the plan executor configured to access the plan and take one or more actions identified in the plan on one of the plurality of links to mitigate the abnormal behavior by applying one or more of the proposed actions based at least on the estimated impact and the estimated success rate.

12. The system of claim 11, wherein the detector is coupled to an aggregator, the aggregator configured to decide whether a detected abnormal behavior corresponds to a new or an ongoing abnormal behavior.

13. The system of claim 11, further comprising a database, the planner coupled to the database to obtain data corresponding to the abnormal behavior and to output the plan to the database.

14. The system of claim 11, wherein the plan includes a command to restart a link.

15. The system of claim 11, wherein the component set further includes a switch, and wherein the plan further includes a command to restart or deactivate the switch.

16. The system of claim 11, wherein the detector monitors SNMP data and one or more performance counters, or both, to detect abnormal behavior comprising:

a, and traffic latency or traffic imbalance.

17. One or more computer-readable devices having computer-executable instructions, which when executed by at least one computer perform operations comprising:

monitoring a network;

determining a network component set corresponding to abnormal behavior, in which the network component set comprises a plurality of links; and

taking automated action on the network component set to mitigate the abnormal behavior by;

identifying a plan comprising a plurality of proposed actions;

estimating an impact on the network for executing each of the proposed actions;

estimating a success rate for each of the proposed actions; and

applying one or more of the proposed actions based at least on the estimated impact and the estimated success rate.

18. The one or more computer-readable devices of claim 17 having further computer-executable instructions comprising, determining whether the abnormal behavior corresponds to a new or an ongoing abnormal behavior.

19. The one or more computer-readable devices of claim 17, wherein the plan comprises deactivating or restarting one of the plurality of links.

20. The one or more computer-readable devices of claim 17, wherein the plan comprises deactivating or restarting a switch.