Automated and adaptive generation of test stimuli for a network or system

• US 10,735,271 B2
• Filed: 12/01/2017
• Issued: 08/04/2020
• Est. Priority Date: 12/01/2017
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Byers, Charles Calvin, Clarke, Joseph Michael, Salgueiro, Gonzalo, Hanes, M. David
• Primary Examiner(s): Katsikis, Kostas J
• Application Number:
  US15/829,139
• Publication Number:
  US 20190173761A1
• Time in Patent Office: 977 Days
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/14 Network analysis or design
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 43/50 Testing arrangements

Abstract

Automatic, adaptive stimulus generation includes receiving, at a network device that is associated with a network or system, analytics data that provides an indication of how the network or system is responding to a set of test stimuli introduced into the network or system to facilitate an analysis operation. The network device analyzes the analytics data based on an intended objective for the analysis operation and generates control settings based on the analyzing. The control settings control creation of a subsequent stimulus to be introduced into the network or system during subsequent execution of the analysis operation.

Claims

1. A method comprising:

receiving, at a network device that is associated with a network, first analytics data that provides an indication of how the network is responding to a first stimulus introduced into the network to facilitate a first analysis operation;

analyzing the first analytics data based on an intended objective for the first analysis operation;

generating, based on the analyzing the first analytics data, control settings for controlling creation of a second stimulus to be introduced into the network during execution of a second analysis operation;

causing a device, based on the control settings, to introduce the second stimulus to the network;

receiving, at the network device that is associated with the network, second analytics data that provides an indication of how the network is responding to the second stimulus introduced into the network to facilitate the second analysis operation; and

analyzing the second analytics data based on an intended objective for the second analysis operation.

2. The method of claim 1, further comprising:

transmitting the control settings to the device.

3. The method of claim 2, further comprising:

transmitting the control settings to an analytics system to cause the analytics system to generate the second analytics data based on the second stimulus introduced into to the network.

4. The method of claim 3, wherein the analyzing the first analytics data employs machine learning techniques,

wherein analyzing the second analytics data employs the machine learning techniques based on the intended objective for the second analysis operation or a modified objective determined by the machine learning techniques;

and the method further comprises;

generating, based on the analyzing of the second analytics data, at least one of;

(a) subsequent control settings for controlling creation of the second stimulus to be introduced into the network;

or (b) a new prioritization of the second stimulus to be introduced into the network.

5. The method of claim 1, wherein the first analytics data comprises:

feedback data from an analytics system associated with the network; and

feedback data from an analytics machine learning system associated with the analytics system.

6. The method of claim 5, further comprises:

transmitting the control settings to the analytics machine learning system to update analytics machine learning techniques applied by the analytics machine learning system.

7. The method of claim 1, wherein the receiving the first analytics data, the analyzing the first analytics data, and the generating are completed automatically during at least one of:

(a) creation of the network;

(b) regular operations of the network;

or (c) abnormal operations of the network.

8. A system comprising:

one or more memories; and

one or more processors, wherein the one or more processors are configured to execute;

an analytics subsystem associated with a network and a stimulitics subsystem associated with the network; and

wherein the one or more processors are configured to execute the analytics subsystem and the stimulitics subsystem by;

receiving, at the analytics subsystem from the one or more memories, first analytics data that provides an indication of how the network is responding to a first stimulus introduced into the network to facilitate a first analysis operation;

analyzing, at the analytics subsystem, the first analytics data based on an intended objective for the first analysis operation;

generating, at the stimulitics subsystem based on the analyzing the first analytics data, control settings for controlling creation of a second stimulus to be introduced into the network during execution of a second analysis operation;

receiving, at the analytics subsystem, second analytics data that provides an indication of how the network is responding to the second stimulus introduced into the network to facilitate the second analysis operation; and

analyzing, at the analytics subsystem, the second analytics data based on an intended objective for the second analysis operation.

9. The system of claim 8, wherein the one or-more processors are configured to execute a stimulus generation subsystem associated with the network,

wherein the one or more processors are configured to execute the stimulitics subsystem by:

transmitting the control settings to the stimulus generation subsystem to cause the stimulus generation subsystem to introduce the second stimulus into the network.

10. The system of claim 9, wherein the one or more processors are further configured to execute the stimulitics subsystem by:

transmitting the control settings to the analytics subsystem to cause the analytics subsystem to generate the second analytics data based on the second stimulus introduced into the network.

11. The system of claim 10, wherein the one or more processors are further configured to execute the stimulitics subsystem by:

employing machine learning techniques;

analyzing the second analytics data with the machine learning techniques based on the intended objective for the second analysis operation or a modified objective determined by the machine learning techniques; and

generating, based on the analyzing of the second analytics data, at least one of;

(a) subsequent control settings for controlling creation of the second stimulus to be introduced into the network;

or (b) a new prioritization of the second stimulus to be introduced into the network.

12. The system of claim 8, wherein the one or more processors are further configured to execute an analytics machine learning subsystem associated with the analytics subsystem,

wherein the one or more processors are further configured to execute the stimulitics subsystem by:

receiving the first analytics data from the analytics subsystem and the analytics machine learning subsystem.

13. The system of claim 12, wherein the one or more processors are further configured to execute the stimulitics subsystem by:

transmitting the control settings to the analytics machine learning subsystem to update analytical machine learning techniques applied by the analytics machine learning subsystem.

14. The system of claim 10, wherein the system operates automatically during at least one of:

(a) creation of the network;

(b) regular operations of the network;

or (c) abnormal operations of the network.

15. One or more non-transitory computer readable storage media encoded with software comprising computer executable instructions and when the software is executed operable to:

receive first analytics data that provides an indication of how a network is responding to a first stimulus introduced into the network to facilitate a first analysis operation;

analyze the first analytics data based on an intended objective for the first analysis operation; and

generate, based on the analyzing the first analytics data, control settings for controlling creation of a second stimulus to be introduced into the network during execution of a second analysis operation;

cause a device, based on the control settings, to introduce the second stimulus to the network;

receive second analytics data that provides an indication of how the network is responding to the second stimulus introduced into the network to facilitate a second analysis operation; and

analyze the second analytics data based on an intended objective for the second analysis operation.

16. The one or more non-transitory computer readable storage media of claim 15, further comprising instructions operable to:

transmit the control settings to a device configured to introduce the second stimulus into the network.

17. The one or more non-transitory computer readable storage media of claim 16, further comprising instructions operable to:

transmit the control settings to an analytics system to cause the analytics system to generate the second analytics data based on the second stimulus introduced into to the network.

18. The one or more non-transitory computer readable storage media of claim 17, wherein the instructions to analyze employ machine learning techniques and the instructions further comprising instructions operable to:

analyze the second analytics data with the machine learning techniques based on the intended objective for the second analysis operation or a modified objective determined by the machine learning techniques; and

generate, based on the analyzing of the second analytics data, at least one of;

(a) subsequent control settings for controlling creation of the second stimulus to be introduced into the network;

or (b) a new prioritization of the second stimulus to be introduced into the network.

19. The one or more non-transitory computer readable storage media of claim 15, wherein the first analytics data comprises:

feedback data from an analytics system associated with the network; and

feedback data from an analytics machine learning system associated with the analytics system.

20. The one or more non-transitory computer readable storage media of claim 15, wherein the instructions are executed automatically during at least one of:

(a) creation of the network;

(b) regular operations of the network;

or (c) abnormal operations of the network.