Dynamically adjusting sample rates based on performance of a machine-learning based model for performing a network assurance function in a network assurance system

• US 10,691,082 B2
• Filed: 12/05/2017
• Issued: 06/23/2020
• Est. Priority Date: 12/05/2017
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Di Pietro, Andrea, Vasseur, Jean-Philippe, Mota, Javier Cruz
• Primary Examiner(s): Murray, Daniel C.
• Application Number:
  US15/831,482
• Publication Number:
  US 20190171169A1
• Time in Patent Office: 931 Days
• CPC Class Codes: G05B 13/04 involving the use of models or simulators
• CPC Class Codes: G05B 13/045 using a perturbation signal
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence
• CPC Class Codes: H04L 43/024 by adaptive sampling
• CPC Class Codes: H04L 43/16 Threshold monitoring

Abstract

In one embodiment, a network assurance service receives data regarding a monitored network. The service analyzes the received data using a machine learning-based model, to perform a network assurance function for the monitored network. The service detects a lowered performance of the machine learning-based model when a performance metric of the machine learning-based model is below a threshold for the performance metric. When it is determined that the lowered performance of the machine-learning based model is correlated with the sample rate of the received data, the service adjusts the sample rate of the data.

Claims

1. A method comprising:

receiving, at a network assurance service, data regarding a monitored network, the received data including data provided to the network assurance service on a push basis;

interleaving, by the network assurance service, additional data regarding the monitored network received by polling one or more network elements in the monitored network on a pull basis with the data provided to the network assurance service on the push basis;

analyzing, by the network assurance service, the additional data received on the pull basis interleaved with the data provided to the network assurance service on the push basis using a machine learning-based model for performing a network assurance function for the monitored network;

detecting, by the network assurance service, a lowered performance of the machine learning-based model when a performance metric of the machine learning-based model is below a threshold for the performance metric;

determining, by the network assurance service, whether the lowered performance of the machine learning-based model is correlated with a sample rate of the received data; and

increasing, by the network assurance service, the sample rate of the received data when it is determined that the lowered performance of the machine-learning based model is correlated with the sample rate of the received data.

2. The method as in claim 1, wherein the detecting of the lowered performance of the machine learning-based model comprises:

identifying, by the network assurance service, a drop in precision, recall, or prediction accuracy of the machine learning-based model.

3. The method as in claim 1, wherein the network assurance function comprises at least one of:

cognitive analytics of the monitored network to identify a device behavior in the monitored network, predictive analytics to predict a future condition of the monitored network, or trending analysis for capacity planning in the monitored network.

4. The method as in claim 1, further comprising:

providing, by the network assurance service and to a user interface, an indication that the lowered performance of the machine learning-based model is correlated with the sample rate of the received data; and

receiving, at the network assurance service and from the user interface, a policy that controls the adjustment of the sample rate of the received data.

5. The method as in claim 4, wherein the policy indicates a particular protocol to be used to adjust the sample rate of the received data.

6. The method as in claim 1, further comprising:

disabling, by the network assurance service, the machine learning-based model until the sample rate of the received data is adjusted.

7. The method as in claim 1, wherein the data regarding the monitored network comprises data reported by a wireless local area network controller.

8. An apparatus comprising:

one or more network interfaces to communicate with a monitored network;

a memory configured to store computer program instructions for performing a process; and

a processor coupled to the one or more network interfaces and configured to execute the computer program instructions, wherein, upon execution of the program instructions, the processor is configured to;

receive data regarding the monitored network, the received data including data provided to the network assurance service on a push basis;

interleaving, by the network assurance service, additional data regarding the monitored network received by polling one or more network elements in the monitored network on a pull basis with the data provided to the network assurance service on the push basis;

analyze the additional data received on the pull basis interleaved with the data provided to the network assurance service on the push basis using a machine learning-based model for performing a network assurance function for the monitored network;

detect a lowered performance of the machine learning-based model when a performance metric of the machine learning-based model is below a threshold for the performance metric;

determine whether the lowered performance of the machine learning-based model is correlated with a sample rate of the received data; and

increase the sample rate of the received data when it is determined that the lowered performance of the machine learning-based model is correlated with the sample rate of the received data.

9. The apparatus as in claim 8, wherein, when the apparatus detects the lowered performance of the machine learning-based model, the processor is configured to:

identify a drop in precision, recall, or prediction accuracy of the machine learning-based model.

10. The apparatus as in claim 8, wherein the network assurance function comprises at least one of:

cognitive analytics of the monitored network to identify a device behavior in the monitored network, predictive analytics to predict a future condition of the monitored network, or trending analysis for capacity planning in the monitored network.

11. The apparatus as in claim 8, wherein the processor is further configured to:

provide, to a user interface, an indication that the lowered performance of the machine learning-based model is correlated with the sample rate of the received data; and

receive, from the user interface, a policy that controls the adjustment of the sample rate of the received data.

12. The apparatus as in claim 11, wherein the policy indicates a particular protocol to be used to adjust the sample rate of the received data.

13. The apparatus as in claim 8, wherein the processor is further configured to:

disable the machine learning-based model until the sample rate of the data is adjusted.

14. The apparatus as in claim 8, wherein the data regarding the monitored network comprises data reported by a wireless local area network controller.

15. A tangible, non-transitory, computer-readable medium storing program instructions that cause a network assurance service to execute a process comprising:

receiving, at the network assurance service, data regarding a monitored network, the received data including data provided to the network assurance service on a push basis;

interleaving, by the network assurance service, additional data regarding the monitored network received by polling one or more network elements in the monitored network on a pull basis with the data provided to the network assurance service on the push basis;

analyzing, by the network assurance service, the additional data received on the pull basis interleaved with the data provided to the network assurance service on the push basis using a machine learning-based model for performing a network assurance function for the monitored network;

detecting, by the network assurance service, a lowered performance of the machine learning-based model when a performance metric of the machine learning-based model is below a threshold for the performance metric;

determining, by the network assurance service, whether the lowered performance of the machine learning-based model is correlated with a sample rate of the received data; and

increasing, by the network assurance service, the sample rate of the received data when it is determined that that the lowered performance of the machine learning-based model is correlated with the sample rate of the received data.

16. The computer-readable medium as in claim 15, wherein the process further comprises:

using a uniform data model to interleave the additional data received on the pull basis with the data provided to the network assurance service on the push basis.