Identifying troubleshooting options for resolving network failures

• US 10,263,836 B2
• Filed: 03/24/2014
• Issued: 04/16/2019
• Est. Priority Date: 03/24/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Jain, Navendu, Potharaju, Rahul
• Primary Examiner(s): Leibovich, Yair
• Application Number:
  US14/223,995
• Publication Number:
  US 20150271008A1
• Time in Patent Office: 1,849 Days
• Field of Search: 714 41
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/0661 by reconfiguring faulty entities
• CPC Class Codes: H04L 41/0686 Additional information in the notification, e.g. enhancement of specific meta-data
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/0886 Fully automatic configuration
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence

Abstract

Described herein are various technologies pertaining to providing assistance to an operator in a data center with respect to failures in the data center. An alarm is received, and a failing device is identified based upon content of the alarm. Failure conditions of the alarm are mapped to a failure symptom that may be exhibited by the failing device, and troubleshooting options previously employed to mitigate the failure symptom are retrieved from historical data. Labels are respectively assigned to the troubleshooting options, where a label is indicative of a probability that a troubleshooting option to which the label has been assigned will mitigate the failure symptom.

Claims

1. A method comprising:

receiving an alarm generated by a device in a data center, the alarm comprising a failure condition that is indicative of a network failure in the data center;

responsive to receiving the alarm and based upon the alarm, identifying a failing device that causes the network failure;

responsive to identifying the failing device, and based upon historical data retained in a data repository, identifying a failure symptom of the network failure based upon the failure condition and the failing device, wherein the historical data comprises;

at least one failure symptom;

troubleshooting options previously undertaken with respect to the failing device in the data center to mitigate the network failure; and

a failure history table that comprises historical failure frequency of the failing device;

outputting data that is indicative of the historical failure frequency of the failing device responsive to the failing device being identified;

responsive to identifying the failing device, identifying the troubleshooting options;

assigning labels to the troubleshooting options, the labels indicative of probabilities that the troubleshooting options, when undertaken with respect to the failing device, will mitigate the failure symptom;

outputting the plurality of troubleshooting options and their labels;

receiving feedback from an operator that the failure symptom has been mitigated and that a troubleshooting option in the troubleshooting options was employed to mitigate the failure symptom; and

updating the failure history table based upon the feedback.

2. The method of claim 1, wherein the device failure history table further comprises:

failure symptoms previously exhibited by the failing device, the failure symptom is included in the failure symptoms previously exhibited by the failing device; and

second troubleshooting options for the failure symptoms previously exhibited by the failing device.

3. The method of claim 1, wherein the historical data further comprises a second device failure history table for a second device that is on a same platform as the failing device, the second device failure history table comprises failure symptoms previously exhibited by the second device, and wherein the failure symptom is included in the failure symptoms previously exhibited by the second device.

4. The method of claim 1, wherein the historical data further comprises a link failure history table that comprises data pertaining to historic failures of a link in the data center, and wherein the link failure history table comprises the failure symptom.

5. The method of claim 1, further comprising:

receiving a selection of the troubleshooting option from the troubleshooting options; and

responsive to receiving the selection of the troubleshooting option, displaying a debugging step, the debugging step comprising an instruction for resolving the network failure.

6. The method of claim 1, further comprising:

responsive to outputting the plurality of troubleshooting options and their respective labels, transmitting a signal to the failing device, the signal causing the troubleshooting option in the troubleshooting options to be performed without operator intervention.

7. The method of claim 1, further comprising:

responsive to receiving the alarm, correlating the alarm with a second alarm to represent the network failure, wherein the failure symptom is identified in the historical data based upon the alarm being correlated with the second alarm.

8. The method of claim 7, wherein correlating the alarm with the second alarm is based upon at least one of an amount of time between when the alarm and the second alarm were generated, devices that respectively generated the alarm and the second alarm, types of the devices that respectively generated the alarm and the second alarm, or a number of hops between the devices that respectively generated the alarm and the second alarm.

9. The method of claim 1, further comprising computing the probabilities based upon a number of times that the troubleshooting options were indicated as being taken in the historical data and a number of times that the troubleshooting operations were indicated as successfully resolving respective network failures.

18. The method of claim 1, wherein the device is one of a switch, a router, a re-router, a gateway, a hub, or a bridge.

10. A resolution system that facilitates resolving network failures in a data center, the resolution system comprising:

at least one processor; and

memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform acts comprising;

receiving an alarm generated by a device in the data center, wherein the alarm is indicative of a network failure in the data center;

responsive to receiving the alarm, identifying a failure symptom of the network failure based upon a failure condition in the received alarm;

identifying a failing device that causes the network failure, wherein the failing device is identified based upon the alarm;

outputting data that is indicative of historical failure frequency of the failing device relative to historic failure frequencies of other devices in the data center responsive to the failing device being identified; and

outputting troubleshooting options for resolving the network failure based upon the failure symptom, wherein the troubleshooting options have labels assigned thereto that are indicative of confidences that the troubleshooting options, when performed by an operator of the data center, will resolve the network failure.

11. The resolution system of claim 10, the acts further comprising determining the labels based upon operator feedback pertaining to the troubleshooting options, wherein the operator feedback is indicative as to whether or not the troubleshooting options previously resolved the network failure.

12. The resolution system of claim 10, further comprising a data store that comprises historical data, wherein the historical data comprises a failure history table for the failing device, wherein the failure history table comprises the failure symptom, and further wherein the failure symptom in the failure history table is mapped to the failure condition of the alarm.

13. The resolution system of claim 12, the acts further comprising:

receiving feedback from the operator, wherein the feedback indicates whether or not the operator successfully resolved the network failure by using the troubleshooting option; and

updating the failure history table based upon the feedback.

14. The resolution system of claim 10, wherein the device is one of a switch, a router, a re-router, a gateway, a hub, or a bridge.

15. The resolution system of claim 10, the acts further comprising outputting a debugging step that corresponds to a troubleshooting option in the troubleshooting options, wherein the debugging step provides instructions to the operator to resolve the network failure.

19. The resolution system of claim 10, wherein the failure history table comprises the historical failure frequency of the failing device.

20. The resolution system of claim 10, the acts further comprising:

responsive to outputting the troubleshooting options, transmitting a signal to the failing device, the signal causing a troubleshooting option in the troubleshooting options to be performed without operator intervention.

16. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:

receiving an alarm, the alarm comprises a failure condition that is indicative of a network failure in a data center;

responsive to receiving the alarm, identifying a failing device that causes the network failure, the failing device identified based upon the failure condition;

outputting data that is indicative of historical failure frequency of the failing device responsive to the failing device being identified;

responsive to identifying the failing device, identifying a failure symptom in a failure history table based upon the failure condition, wherein the failure history table indicates that the failure symptom was previously exhibited by the failing device, and further wherein the failure history table comprises troubleshooting options previously employed to mitigate the failure symptom;

retrieving the troubleshooting options responsive to identifying the failure symptom; and

outputting the troubleshooting options and labels for the troubleshooting options, the labels indicative of confidences that the troubleshooting options, when employed by an operator, will mitigate the failure symptom, wherein the labels are determined based upon operator feedback pertaining to the troubleshooting options, and further wherein the operator feedback is indicative as to whether or not the troubleshooting options previously resolved the network failure.

17. The computer-readable storage medium of claim 16, the acts further comprising:

receiving feedback from the operator that the failure symptom has been mitigated and that a first troubleshooting option in the troubleshooting options was employed to mitigate the failure symptom; and

updating the failure history table based upon the feedback.