Network configuration change analysis using machine learning

• US 10,680,889 B2
• Filed: 04/02/2018
• Issued: 06/09/2020
• Est. Priority Date: 04/02/2018
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Vasseur, Jean-Philippe, Kolar, Vinay Kumar, Pandey, Santosh Ghanshyam
• Primary Examiner(s): Taha, Shukri
• Application Number:
  US15/942,665
• Publication Number:
  US 20190306023A1
• Time in Patent Office: 799 Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/082 the condition being updates or upgrades of network functionality
• CPC Class Codes: H04L 41/147 for predicting network behaviour
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence
• CPC Class Codes: H04L 43/50 Testing arrangements

Abstract

In one embodiment, a network assurance service that monitors one or more networks receives data indicative of networking device configuration changes in the one or more networks. The service also receives one or more performance indicators for the one or more networks. The service trains a machine learning model based on the received data indicative of the networking device configuration changes and on the received one or more performance indicators for the one or more networks. The service predicts, using the machine learning model, a change in the one or more performance indicators that would result from a particular networking device configuration change. The service causes the particular networking device configuration change to be made in the network based on the predicted one or more performance indicators.

Claims

1. A method comprising:

receiving, at a network assurance service that monitors one or more networks, data indicative of networking device configuration changes in the one or more networks;

receiving, at the service, one or more performance indicators for the one or more networks;

training, by the service, a machine learning model based on the received data indicative of the networking device configuration changes and on the received one or more performance indicators for the one or more networks;

constructing, by the service, a pre-change distribution of the one or more performance indicators given a state of the one or more networks before a particular networking device configuration change has been made;

predicting, by the service and using the machine learning model, a change in the one or more performance indicators that would result from the particular networking device configuration change;

causing, by the service, the particular networking device configuration change to be made in the network based on the predicted one or more performance indicators;

constructing, by the service, a post-change distribution of the one or more performance indicators given the state of the one or more networks being updated to reflect the particular networking device configuration change being made; and

computing, by the service, a difference between the pre-change distribution of the one or more performance indicators and the post-change distribution of the one or more performance indicators to determine whether a performance degradation occurred due to the particular networking device configuration change.

2. The method as in claim 1, wherein the one or more performance indicators comprise a number of radio failures in the one or more networks.

3. The method as in claim 2, wherein the network device configuration changes comprise software changes made to wireless access points or to controllers for the wireless access points in the one or more networks.

4. The method as in claim 1, wherein the state of the one or more networks comprises a set of networking device configurations at a given point in time based on the data indicative of the networking device configuration changes.

5. The method as in claim 1, further comprising:

detecting, by the service and using the machine learning model, a change in the one or more performance indicators that resulted from making a networking device configuration change to the one or more networks.

6. The method as in claim 5, wherein the change is detected using a Kolmogorov-Smirnov test.

7. The method as in claim 1, wherein causing the particular networking device configuration change to be made in the network based on the predicted one or more performance indicators comprises:

providing, by the service, an indication of the predicted one or more performance indicators to a user interface.

8. The method as in claim 1, wherein the machine learning model comprises a regression model.

9. The method as in claim 1, wherein the particular networking device configuration comprises a frequency band setting or channel setting for a wireless access point.

10. An apparatus, comprising:

one or more network interfaces to communicate with one or more networks;

a processor coupled to the network interfaces and configured to execute one or more processes; and

a memory configured to store a process executable by the processor, the process when executed configured to;

receive data indicative of networking device configuration changes in the one or more networks;

receive one or more performance indicators for the one or more networks;

train a machine learning model based on the received data indicative of the networking device configuration changes and on the received one or more performance indicators for the one or more networks;

construct a pre-change distribution of the one or more performance indicators given a state of the one or more networks before a particular networking device configuration change has been made;

predict, using the machine learning model, a change in the one or more performance indicators that would result from the particular networking device configuration change;

cause the particular networking device configuration change to be made in the network based on the predicted one or more performance indicators;

construct a post-change distribution of the one or more performance indicators given the state of the one or more networks being updated to reflect the particular networking device configuration change being made; and

compute a difference between the pre-change distribution of the one or more performance indicators and the post-change distribution of the one or more performance indicators to determine whether a performance degradation occurred due to the particular networking device configuration change.

11. The apparatus as in claim 10, wherein the one or more performance indicators comprise a number of radio failures in the one or more networks.

12. The apparatus as in claim 11, wherein the network device configuration changes comprise software changes made to wireless access points or to controllers for the wireless access points in the one or more networks.

13. The apparatus as in claim 10, wherein the state of the one or more networks comprises a set of networking device configurations at a given point in time based on the data indicative of the networking device configuration changes.

14. The apparatus as in claim 10, wherein the process when executed is further configured to:

detect, using the machine learning model, a change in the one or more performance indicators that resulted from making a networking device configuration change to the one or more networks.

15. The apparatus as in claim 10, wherein the apparatus causes the particular networking device configuration change to be made in the network based on the predicted one or more performance indicators by:

providing an indication of the predicted one or more performance indicators to a user interface.

16. The apparatus as in claim 10, wherein the machine learning model comprises a regression model.

17. The apparatus as in claim 10, wherein the particular networking device configuration comprises a frequency band setting or channel setting for a wireless access point.

18. A tangible, non-transitory, computer-readable medium storing program instructions that cause a network assurance service that monitors one or more networks to execute a process comprising:

receiving, at the service, data indicative of networking device configuration changes in the one or more networks;

receiving, at the service, one or more performance indicators for the one or more networks;

training, by the service, a machine learning model based on the received data indicative of the networking device configuration changes and on the received one or more performance indicators for the one or more networks;

constructing, by the service, a pre-change distribution of the one or more performance indicators given a state of the one or more networks before a particular networking device configuration change has been made;

predicting, by the service and using the machine learning model, a change in the one or more performance indicators that would result from the particular networking device configuration change;

causing, by the service, the particular networking device configuration change to be made in the network based on the predicted one or more performance indicators;

constructing, by the service, a post-change distribution of the one or more performance indicators given the state of the one or more networks being updated to reflect the particular networking device configuration change being made; and

computing, by the service, a difference between the pre-change distribution of the one or more performance indicators and the post-change distribution of the one or more performance indicators to determine whether a performance degradation occurred due to the particular networking device configuration change.