Automated execution and evaluation of network-based training exercises

• US 9,076,342 B2
• Filed: 02/18/2009
• Issued: 07/07/2015
• Est. Priority Date: 02/19/2008
• Status: Active Grant

• Current Assignee: Architecture Technology Corporation
• Original Assignee: Architecture Technology Corporation
• Inventors: Brueckner, Stephen, Adelstein, Frank N., Bar, Haim, Donovan, Matthew
• Primary Examiner(s): Utama, Robert J
• Application Number:
  US12/388,425
• Publication Number:
  US 20090208910A1
• Time in Patent Office: 2,330 Days
• Field of Search: 726/23, 726/26, 726/22, 726/25, 726/46, 726/11, 434/118, 434/335, 705/7, 707/3, 707/10, 713/201, 706/46, 715/736
• US Class Current: 1/1
• CPC Class Codes: G09B 19/003 Repetitive work cycles; Sequence of movements
• CPC Class Codes: G09B 19/0053 Computers, e.g. programming
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131
• CPC Class Codes: H04L 63/029 Firewall traversal, e.g. tunnelling or, creating pinholes
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 63/1475 Passive attacks, e.g. eavesdropping or listening without modification of the traffic monitored

Abstract

This disclosure generally relates to automated execution and evaluation of computer network training exercises, such as in a virtual machine environment. An example environment includes a control and monitoring system, an attack system, and a target system. The control and monitoring system initiates a training scenario to cause the attack system to engage in an attack against the target system. The target system then performs an action in response to the attack. Monitor information associated with the attack against the target system is collected by continuously monitoring the training scenario. The attack system is then capable of sending dynamic response data to the target system, wherein the dynamic response data is generated according to the collected monitor information to adapt the training scenario to the action performed by the target system. The control and monitoring system then generates an automated evaluation based upon the collected monitor information.

Claims

1. A method comprising:

providing a training environment that includes a control and monitoring system, an attack system, and a target system that are each executable by one or more processors and that each comprise one or more virtual machines, and wherein the training environment is configured to monitor and respond to actions specified by a human trainee, the human trainee using the target system and participating in the training environment;

initiating, by the control and monitoring system, a training scenario within the training environment to cause the attack system to engage in a simulated attack against the target system;

in response to the simulated attack against the target system, performing, by the target system, an action that is specified by the human trainee;

updating a state of the target system based upon the action performed by the target system and specified by the human trainee;

collecting, by the control and monitoring system, monitor information associated with the simulated attack against the target system by continuously monitoring the training scenario, wherein collecting the monitor information associated with the training scenario further comprises;

collecting information associated with the action performed by the target system and specified by the human trainee, and

receiving user input from the human trainee indicating a reason for performing the action;

updating a state of the attack system based upon the collected monitor information that is associated with the action performed by the target system and specified by the human trainee;

generating, by the attack system, dynamic response data according to the updated state of the attack system;

sending the dynamic response data from the attack system to the target system to adapt the training scenario to the action performed by the target system and specified by the human trainee; and

generating, by the control and monitoring system, an automated evaluation of a performance of the human trainee, wherein the automated evaluation is based upon the collected monitor information that is associated with the action performed by the target system and specified by the human trainee during the simulated attack, and wherein generating the automated evaluation further comprises analyzing the user input to determine if the reason for performing the action is correct according to the training scenario.

2. The method of claim 1, wherein the attack system and the target system are included within one attack/target system, and wherein the attack against the target system comprises an insider attack.

3. The method of claim 1, wherein collecting the monitor information comprises collecting active feedback data and passive feedback data, wherein the active feedback data comprises one or more of action data, state data, and metric data, and wherein the passive feedback data comprises one or more of notebook data, instant message data, and state knowledge data.

4. The method of claim 1, wherein receiving the user input comprises receiving the user input either by way of an electronic notebook that records information entered by the human trainee or by way of an instant message exchange.

5. The method of claim 1, further comprising:

sending scenario traffic for the training scenario on a first communication channel;

sending out-of-band data for the training scenario on a second communication channel that is distinct from the first communication channel; and

monitoring and controlling the training scenario within the training environment using the out-of-band data, wherein the monitor information comprises the out-of-band data, wherein the out-of-band data does not interfere with the scenario traffic sent on the first communication channel.

6. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause one or more processors to:

provide a training environment that includes a control and monitoring system, an attack system, and a target system that each comprise one or more virtual machines, wherein the training environment is configured to monitor and respond to actions specified by a human trainee, the human trainee using the target system and participating in the training environment;

initiate, by the control and monitoring system, a training scenario within the training environment to cause the attack system to engage in simulated attack against the target system;

in response to the simulated attack against the target system, perform, by the target system, an action that is specified by the human trainee;

update a state of the target system based upon the action performed by the target system and specified by the human trainee;

collect, by the control and monitoring system, monitor information associated with the simulated attack against the target system by continuously monitoring the training scenario, wherein collecting the monitor information associated with the training scenario further comprises;

collecting information associated with the action performed by the target system and specified by the human trainee, and

receiving user input from the human trainee indicating a reason for performing the action;

update a state of the attack system based upon the collected monitor information that is associated with the action performed by the target system and specified by the human trainee;

generate, by the attack system, dynamic response data according to the updated state of the attack system;

send the dynamic response data from the attack system to the target system to adapt the training scenario to the action performed by the target system and specified by the human trainee; and

generate, by the control and monitoring system, an automated evaluation of a performance of the human trainee, wherein the automated evaluation is based upon the collected monitor information that is associated with the action performed by the target system and specified by the human trainee during the simulated attack, and wherein generating the automated evaluation further comprises analyzing the user input to determine if the reason for performing the action is correct according to the training scenario.

7. A system comprising:

one or more processors;

one or more non-transitory computer-readable storage media comprising instructions that are executable by the one or more processors;

an attack system stored on the one or more non-transitory computer-readable storage media and executable by the one or more processors, wherein the attack system comprises one or more virtual machines;

a target system stored on the one or more non-transitory computer-readable storage media and executable by the one or more processors, wherein the target system comprises one or more virtual machines; and

a control and monitoring system stored on the one or more non-transitory computer-readable storage media and executable by the one or more processors, wherein the control and monitoring system comprises one or more virtual machines, the control and monitoring system being configured to initiate, within a training environment, a training scenario that causes the attack system to engage in a simulated attack against the target system, and further configured to collect monitor information associated with the simulated attack by continuously monitoring the training scenario, the training environment being configured to monitor and respond to actions specified by a human trainee, the human trainee using the target system and participating in the training environment,

wherein in response to the simulated attack against the target system, the target system is configured to perform an action that is specified by the human trainee, wherein the target system updates its state based upon the action performed by the target system and specified by the human trainee,

wherein the collected monitor information comprises information associated with the action performed by the target system and specified by the human trainee, and further includes user input from the human trainee indicating a reason for performing the action, wherein the attack system is configured to update a state of the attack system based upon the collected monitor information that is associated with the action performed by the target system and specified by the human trainee, wherein the attack system is configured to generate dynamic response data according to the updated state of the attack system and to send the dynamic response data to the target system to adapt the training scenario to the action performed by the target system and specified by the human trainee, and wherein the control and monitoring system is configured to generate an automated evaluation of a performance of the human trainee, wherein the automated evaluation is based upon the collected monitor information that is associated with the action performed by the target system and specified by the human trainee during the simulated attack, and wherein generating the automated evaluation includes analyzing the user input to determine if the reason for performing the action is correct according to the training scenario.

8. The system of claim 7, wherein the attack system and the target system are included within one attack/target system, and wherein the attack against the target system comprises an insider attack.

9. The system of claim 7, wherein the monitor information comprises active feedback data and passive feedback data, wherein the active feedback data comprises one or more of action data, state data, and metric data, and wherein the passive feedback data comprises one or more of notebook data, instant message data, and state knowledge data.

10. The system of claim 7, wherein the target system is configured to receive the user input by receiving the user input either by way of an electronic notebook that records information entered by the human trainee or by way of an instant message exchange.

11. The system of claim 7, wherein the attack system and the target system are each configured to send scenario traffic for the training scenario on a first communication channel and to send out-of-band data for the training scenario on a second communication channel that is distinct from the first communication channel, wherein the control and monitoring system is further configured to monitor the training scenario within the training environment by using the out-of-band data, wherein the monitor information comprises the out-of-band data, and wherein the out-of-band data does not interfere with the scenario traffic sent on the first communication channel.