Monitoring and detecting causes of failures of network paths

• US 8,661,295 B1
• Filed: 03/31/2011
• Issued: 02/25/2014
• Est. Priority Date: 03/31/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Khanna, Richendra, Brar, Jagwinder Singh, Marr, Michael David, McKelvie, Samuel J., DeSantis, Peter N., Nowland, Ian P., Klein, Matthew D., Gabrielson, Jacob, Wallace, Edward L., Mason, Justin
• Primary Examiner(s): Riad, Amine
• Application Number:
  US13/077,296
• Time in Patent Office: 1,062 Days
• Field of Search: 714/43
• US Class Current: 714/43
• CPC Class Codes: H04L 41/065 involving logical or physical relationship, e.g. grouping and hierarchies
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

Generally described, systems and methods are provided for monitoring and detecting causes of failures of network paths. The system collects performance information from a plurality of nodes and links in a network, aggregates the collected performance information across paths in the network, processes the aggregated performance information for detecting failures on the paths, analyzes each of the detected failures to determine at least one root cause, and initiates a remedial workflow for the at least one root cause determined. In some aspects, processing the aggregated information may include performing a statistical regression analysis or otherwise solving a set of equations for the performance indications on each of a plurality of paths. In another aspect, the system may also include an interface which makes available for display one or more of the network topology, the collected and aggregated performance information, and indications of the detected failures in the topology.

Claims

1. A computing device comprising:

a processing unit in communication with the memory and the interface, the processing unit operative to;

generate a first network topology comprising a plurality of network paths, wherein each one of the network paths includes at least two nodes and at least one link;

allocate health checks across the plurality of network paths in order to monitor all links in the network;

measure communication attributes across each of the plurality of paths in the network by using the generated first network topology, and by manipulating port numbers to modify paths between two nodes;

process the measured communication attributes for detecting failures on the paths;

analyze each of the detected failures to determine at least one root cause for each of the failures; and

remedy the at least one root cause determined;

a memory that stores at least one of the first network topology, the subsequent network topology, the measured communication attributes, the processed communication attributes, the detected failures, the determined root causes; and

an interface which makes available for display at least one of the first network topology, the subsequent network topology, the measured communication attributes, the processed communication attributes, the detected failures, the determined root causes and the alarm.

2. The computing device of claim 1, wherein the processing unit is further operative to compare the generated first network topology to an expected topology.

3. The computing device of claim 2, wherein the processing unit is further operative to use an external data source to reconcile the generated first network topology to the expected topology.

4. The computing device of claim 3, wherein the external data source comprises at least one of configuration files, technicians'"'"' information, automated switch building, subnet analysis, SNMP query information regarding run-time configuration states of devices, and other monitoring services.

5. The computing device of claim 1, wherein remedying the root cause comprises at least one of generating a subsequent network topology, generating an alarm indicating the root cause, shutting down a faulty node, rerouting packets of data away from the faulty node, disabling ports on the faulty node, cycling power to the faulty node.

6. A computer-implemented method for network path monitoring, the method comprising:

generating a first network topology comprising a plurality of network paths, wherein each one of the network paths includes at least two nodes and at least one link;

allocating health checks across the plurality of network paths in order to monitor all links in the network;

measuring communication attributes across each of the plurality of paths in the network, wherein measuring the communication attributes across each the plurality of paths comprises using the generated first network topology, and manipulating port numbers to modify paths between two nodes;

processing the measured communication attributes for detecting failures on the paths;

analyzing each of the detected failures to determine at least one root cause for each of the failures; and

remedying the at least one root cause determined, wherein remedying comprises at least one of gathering a subsequent network topology, reallocating the health checks, and generating an alarm indicating the root cause.

7. The computer-implemented method of claim 6 further comprising comparing the gathered first network topology to an expected topology.

8. The computer-implemented method of claim 7 further comprising using an external data source to reconcile the gathered first network topology to the expected topology.

9. The computer-implemented method of claim 8, wherein the external data source comprises at least one of configuration files, technicians'"'"' information, automated switch building, subnet analysis, SNMP query information regarding run-time configuration states of devices, and other monitoring services.

10. The computer-implemented method of claim 6, wherein the network topology is gathered for layer 3 of the Open Systems Interconnection (OSI) standard.

11. The computer-implemented method of claim 6, further comprising generating a first network topology for layer 2 of the OSI standard.

12. The computer-implemented method of claim 6 further comprising generating a first topology for at least one of:

power to racks, power to hosts, and power to network devices.

13. The computer-implemented method of claim 6 further comprising generating a first topology for at least one of a physical location within a thermal zone, a room, and a data center.

14. A non-transitory, computer-readable medium having a computer-executable component, the computer-executable component comprising:

a network path monitoring component for;

generating a first network topology comprising a plurality of network paths, wherein each one of the network paths includes at least two nodes and at least one link;

allocating health checks across the plurality of network paths in order to monitor all links in the network;

measuring communication attributes across each of the plurality of paths in the network, wherein measuring the communication attributes across each the plurality of paths comprises using the generated first network topology, and manipulating port numbers to modify paths between two nodes; and

processing the measured communication attributes for detecting failures on the paths.

15. The non-transitory computer-readable medium of claim 14, wherein the network topology is generated for at least one of layer 3 and layer 2 of the Open Systems Interconnection (OSI) standard.

16. The non-transitory computer-readable medium of claim 14, wherein the network path monitoring component is further for:

comparing the generated first network topology to an expected topology;

using an external data source to reconcile the generated first network topology to the expected topology.

17. The non-transitory computer-readable medium of claim 16, wherein the external data source comprises at least one of configuration files, technicians'"'"' information, automated switch building, subnet analysis, SNMP query information regarding run-time configuration states of devices, and other monitoring services.