Adaptive team training evaluation system and method

• US 10,600,335 B1
• Filed: 09/18/2017
• Issued: 03/24/2020
• Est. Priority Date: 09/18/2017
• Status: Active Grant

• Current Assignee: Architecture Technology Corporation
• Original Assignee: Architecture Technology Corporation
• Inventors: Donovan, Matthew, Kimball, Colleen
• Primary Examiner(s): Fletcher, Jerry-Daryl
• Application Number:
  US15/707,720
• Time in Patent Office: 918 Days
• CPC Class Codes: G09B 19/0053 Computers, e.g. programming
• CPC Class Codes: G09B 5/10 all student stations being capable of presenting the same information simultaneously G09B5/14 takes precedence
• CPC Class Codes: G09B 5/12 different stations being capable of presenting different information simultaneously G09B5/14 takes precedence
• CPC Class Codes: G09B 7/04 characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying a further explanation
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 67/1057 involving pre-assessment of levels of reputation of peers

Abstract

A computer-implemented adaptive group training method a computer accessing a virtual system and initiating a group training exercise for training a trainee group comprising one or more trainees, the group training exercise including one or more challenges to the virtual system, each of the one or more challenges including a pre-defined sequence of one or more injectable events; the computer controlling subsequent execution of the group training exercise comprising injecting the injectable events; and the computer evaluating performance of the trainee group during the subsequent execution of the group training exercise, including analyzing actions taken by the trainee group in response to each of the injections, and attributing one or more of the actions taken to a trainee.

Claims

1. A computer-implemented adaptive group training method comprising:

a computer accessing a virtual system and initiating a group training exercise for training a trainee group comprising one or more trainees, the group training exercise comprising one or more challenges to the virtual system, each of the one or more challenges including a pre-defined sequence of one or more injectable events;

the computer controlling subsequent execution of the group training exercise comprising injecting the injectable events; and

the computer evaluating performance of the trainee group during the subsequent execution of the group training exercise, comprising;

analyzing actions taken by the trainee group in response to each of the injections, and

attributing one or more of the actions taken to a trainee.

2. The method of claim 1, further comprising the computer, automatically and in real-time during execution of the group training device, executing a group training exercise adaptation based on the evaluated performance of the trainee group comprising stopping the pre-defined sequence and starting a second pre-defined sequence of one or more second injectable events.

3. The method of claim 1, wherein:

evaluating performance of the trainee group comprises;

performing a first evaluation in real-time during execution of the executing group training exercise and a second evaluation after completion of the group training exercise, and

determining an action taken is one of correct and not correct based on an injected event; and

wherein controlling subsequent execution of the group training exercise comprises changing a state of the virtual system based on the actions taken.

4. The method of claim 3, wherein attributing the actions taken to a trainee comprises the computer:

accessing agents monitoring a virtual network to identify sources of the actions taken;

accessing native sensors and trainee group sensors to identify specific trainees taking actions, times of taking actions, and content of the actions;

comparing the sources, specific trainees, times and content to the actions taken to determine the origins of the actions taken; and

comparing the content derived from the native sensors to stored, physical characteristics of the trainees.

5. The method of claim 1, wherein the virtual system is instantiated on a plurality of virtual machines hosted on a physical computing platform, the method further comprising the computer executing a group training exercise adaptation based on availability of virtual machine resources.

6. A non-transitory, computer-readable storage medium comprising instructions for implementing an adaptive group training exercise, wherein a processor executes the instructions to:

access a virtual system and initiate a group training exercise for training a trainee group comprising one or more trainees, the group training exercise comprising one or more challenges to the virtual system, each of the one or more challenges comprising a pre-defined sequence including one or more injectable events;

control subsequent execution of the group training exercise comprising injecting the injectable events; and

evaluate performance of the trainee group during the subsequent execution of the group training exercise, comprising;

analyzing actions taken by the trainee group in response to each of the injections, and attributing one or more of the actions taken to a trainee.

7. The non-transitory, computer-readable storage medium of claim 6, wherein the processor executes automatically and in real-time during execution of the group training exercise, a group training exercise adaptation based on the evaluated performance of the trainee group comprising stopping the pre-defined sequence and starting a second pre-defined sequence of one or more second injectable events.

8. The non-transitory, computer-readable storage medium of claim 6, wherein the processor evaluates performance of the trainee group by:

performing a first evaluation in real-time during execution of the executing group training exercise and a second evaluation after completion of the group training exercise;

determining an action taken is one of correct and not correct based on an injected event; and

wherein controlling subsequent execution of the group training exercise comprises changing a state of the virtual system based on the actions taken.

9. The non-transitory, computer-readable storage medium of claim 6, wherein the processor attributes the actions taken to a trainee by:

accessing agents monitoring a virtual network to identify sources of the actions taken;

accessing native sensors and trainee group sensors to identify specific trainees taking actions, times of taking actions, and content of the actions;

comparing the sources, specific trainees, times and content to the actions taken to determine the origins of the actions taken; and

comparing the content derived from the native sensors to stored, physical characteristics of the trainees.

10. The non-transitory, computer-readable storage medium of claim 6, wherein the virtual system is instantiated on a plurality of virtual machines hosted on a physical computing platform, wherein the processor executes a group training exercise adaptation based on availability of virtual machine resources.

11. A computer-implemented group training method for a trainee group, the trainee group including two or more trainees, the method comprising:

a computer initiating execution of a group training exercise, the group training exercise comprising one or more challenges to a virtual system, each of the one or more challenges including a pre-defined sequence of one or more injectable events;

the computer controls, controlling subsequent execution of the group training exercise, and evaluates performance of the trainee group during the subsequent execution of the group training exercise, comprising;

directing injections of the one or more injectable events, and monitoring actions taken by the trainee group in response to each of the injections;

the computer attributing one or more of the actions taken to one of the two or more trainees of the trainee group; and

the computer executing, automatically and in real-time during execution of the group training exercise, a group training exercise adaptation based on the evaluated performance of the trainee group comprising stopping the pre-defined sequence and starting a second pre-defined sequence of one or more second injectable events.

12. The method of claim 11, wherein the virtual system is a virtual cyber range comprising a virtual network and one or more virtual network components, the virtual network and each of the virtual network components instantiated on a virtual machine, and wherein the one or more challenges comprise a simulated cyber warfare attack.

13. The method of claim 11, wherein evaluating execution of the group training exercise comprises receiving and evaluating each of the actions taken.

14. The method of claim 13, wherein evaluating each of the actions taken comprise:

identifying a class of each of the actions taken;

identifying each of the actions taken as one of a correct action, an incorrect action, and an undefined action; and

assigning a positive score for each correct action and a negative score for each incorrect action.

15. The method of claim 13, wherein the computer evaluating performance of the trainee group comprises the computer assessing trainee group performance after each injectable event.

16. The method of claim 15, wherein the computer assessing trainee group performance, comprises:

for each correct action, determining a timing of the corresponding action taken;

applying a weight to the correct action;

computing an event score as a sum of the product of the weight and the positive score for each correct action and the negative score for each incorrect action; and

computing a cumulative event score comprising summing events scores for each injected event.

17. The method of claim 16, wherein the computer executing the group training exercise adaptation based on the cumulative event score comprises:

comparing the cumulative event score to a first threshold value and selecting a first adaptation when the cumulative event score is less than the first threshold value.

18. The method of claim 17, wherein the computer executing the group training exercise adaptation based on the cumulative event score further comprises:

comparing the cumulative event score to a second threshold value and selecting a second adaptation when the cumulative event score is greater than the second threshold value.

19. The method of claim 11, wherein the computer attributing the actions taken comprises:

the computer receives a group training exercise document, the document indicative of an action taken, the document generated during execution of the group training exercise;

the computer identifies and analyzes content data and metadata in the document; and

the computer associates the document with a specific trainee based on analysis of the content data and metadata.

20. The method of claim 19, further comprising:

based on the content data and the metadata, the computer identifying a specific virtual machine originating the document;

the computer acquiring native sensor data associated with the specific virtual machine; and

the computer comparing the native sensor data with reference data for each of the two or more trainees, wherein the native sensor data comprises trainee image data and trainee voice data.