Real-time monitoring of network-based training exercises

• US 10,083,624 B2
• Filed: 07/28/2015
• Issued: 09/25/2018
• Est. Priority Date: 07/28/2015
• Status: Active Grant

• Current Assignee: Architecture Technology Corporation
• Original Assignee: Architecture Technology Corporation
• Inventors: Brueckner, Stephen K., Donovan, Matthew P.
• Primary Examiner(s): EGLOFF, PETER RICHARD
• Application Number:
  US14/811,403
• Publication Number:
  US 20170032694A1
• Time in Patent Office: 1,155 Days
• CPC Class Codes: G09B 19/0053 Computers, e.g. programming
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip

Abstract

An example method includes outputting a graphical dashboard that includes one or more learning objective nodes and one or more skill nodes, selecting one or more software agents that are associated with the one or more skill nodes, providing, to at least one host computing system, an indication of the one or more software agents that are configured to collect parameter data from the at least one host computing system while a trainee performs actions, receiving the parameter data collected by the one or more software agents during execution, determining, based on the parameter data, that the one or more skills represented by the one or more skill nodes have been demonstrated by the trainee, and updating the one or more skill nodes to graphically indicate that one or more represented skills have been demonstrated.

Claims

1. A method comprising:

outputting, by a management server and for display, a graphical dashboard associated with a training exercise, wherein the graphical dashboard includes one or more learning objective nodes that represent one or more learning objectives to be accomplished by a trainee using at least one host computing system during the training exercise, wherein the graphical dashboard further includes one or more skill nodes that represent one or more skills to be demonstrated by the trainee during the training exercise, wherein the one or more skills support the one or more learning objectives, and wherein the one or more skill nodes graphically indicate that the one or more skills have not yet been demonstrated by the trainee;

determining, by the management server, one or more metrics that are usable to determine whether the one or more skills are demonstrated by the trainee based on one or more evaluation criteria;

identifying, by the management server, parameter data to be collected by the at least one host computing system during the training exercise to calculate the one or more metrics;

selecting, by the management server and based on the one or more skills to be demonstrated by the trainee, one or more software agents that are associated with the one or more skill nodes and that are capable of collecting the parameter data;

after selecting the one or more software agents, providing, by the management server and to the at least one host computing system, an indication of the one or more software agents that are executed on the at least one host computing system during the training exercise;

providing, by the management server and to the at least one host computing system, one or more metric parameters that are used to configure the one or more software agents to collect the parameter data from the at least one host computing system while the trainee performs actions during the training exercise, wherein the one or more metric parameters include at least an operational mode parameter indicating whether the one or more software agents will terminate after collecting the parameter data or will instead stay resident on and monitor the at least one host computing system;

receiving, by the management server, the parameter data collected by the one or more software agents during execution on the at least one host computing system;

calculating, by the management server and based on the parameter data collected by the one or more software agents, the one or more metrics usable to determine whether the one or more skills are demonstrated by the trainee based on the one or more evaluation criteria;

determining, by the management server and based on the calculated one or more metrics, that the one or more skills have been demonstrated by the trainee during the training exercise; and

responsive to determining that the one or more skills have been demonstrated, updating, by the management server and for display, the one or more skill nodes representing the one or more skills to graphically indicate that the one or more skills have been demonstrated by the trainee during the training exercise.

2. The method of claim 1, wherein providing the indication of the one or more software agents comprises sending, from the management server and to the at least one host computing system, the one or more software agents for execution by the at least one host computing system.

3. The method of claim 1, wherein the one or more metrics comprise at least one of a number of attacks detected during the training exercise, a time taken to recover from an attack during the training exercise, a number of attacks successfully identified during the training exercise, or a downtime of an attacked service compared to an attack duration during the training exercise.

4. The method of claim 1, wherein the at least one host computing system comprises at least one of a server, a domain controller, or a workstation.

5. The method of claim 1,

wherein the graphical dashboard graphically indicates a hierarchical relationship between the one or more skill nodes and the one or more learning objective nodes, and

wherein before receiving the parameter data collected by the one or more software agents during execution, the one or more learning objective nodes graphically indicate that the one or more learning objectives have not yet been accomplished by the trainee.

6. The method of claim 5, wherein responsive to determining that the one or more skills have been demonstrated, the method further comprises:

determining, by the management server and based on the hierarchical relationship between the one or more skill nodes and the one or more learning objective nodes, that the one or more learning objectives represented by the one or more learning objective nodes have been accomplished by the trainee during the training exercise; and

responsive to determining that the one or more learning objectives have been accomplished, updating, by the management server and for display, the one or more learning objective nodes to graphically indicate that the one or more learning objectives have been accomplished by the trainee during the training exercise.

7. The method of claim 6,

wherein the one or more skills comprise a plurality of skills,

wherein the one or more skill nodes comprise a plurality of skill nodes, and

wherein the hierarchical relationship in the graphical dashboard between the one or more skill nodes and the one or more learning objective nodes comprises a logical relationship between the plurality of skill nodes and a particular learning objective node of the one or more learning objective nodes, the particular learning objective node representing a particular learning objective of the one or more learning objectives.

8. The method of claim 7,

wherein the logical relationship between the plurality of skill nodes and the particular learning objective node indicates that at least one skill represented by the plurality of skills nodes is to be demonstrated in order for the particular learning objective to be accomplished,

wherein determining that the one or more skills have been demonstrated comprises determining that the at least one skill has been demonstrated,

wherein updating the one or more skill nodes comprises updating at least one skill node representing the at least one skill to graphically indicate that the at least skill has been demonstrated,

wherein determining that the one or more learning objectives have been accomplished comprises determining, based on the logical relationship between the plurality of skill nodes and the particular learning objective node, that the particular learning objective has been accomplished by the trainee during the training exercise, and

wherein responsive to determining that the particular learning objective has been accomplished, updating the one or more learning objective nodes comprises updating, for display, the particular learning objective node to graphically indicate that the particular learning objective has been accomplished by the trainee during the training exercise.

9. The method of claim 7,

wherein the logical relationship between the plurality of skill nodes and the particular learning objective node indicates that each of the plurality of skills is to be demonstrated in order for the particular learning objective to be accomplished,

wherein determining that the one or more skills have been demonstrated comprises determining that each of the plurality of skills has been demonstrated,

wherein updating the one or more skill nodes comprises updating the plurality of skill nodes representing the plurality of skills to graphically indicate that each of the plurality of skills has been demonstrated,

wherein determining that the one or more learning objectives have been accomplished comprises determining, based on the logical relationship between the plurality of skill nodes and the particular learning objective node, that the particular learning objective has been accomplished by the trainee during the training exercise, and

wherein responsive to determining that the particular learning objective has been accomplished, updating the one or more learning objective nodes comprises updating, for display, the particular learning objective node to graphically indicate that the particular learning objective has been accomplished by the trainee during the training exercise.

10. The method of claim 1,

wherein the training exercise comprises a team exercise for a first team and a second team,

wherein the trainee is a member of the first team,

wherein the one or more learning objective nodes represent the one or more learning objectives to be accomplished by the first team during the training exercise,

wherein the one or more skill nodes represent the one or more skills to be demonstrated by the first team during the training exercise,

wherein before receiving the parameter data collected by the one or more software agents during execution, the one or more skill nodes graphically indicate that the one or more skills have not yet been demonstrated by the first team,

wherein determining that the one or more skills represented by the one or more skill nodes have been demonstrated comprises determining, by the management server and based on the parameter data, that the one or more skills have been demonstrated by the first team during the training exercise, and

wherein, responsive to determining that the one or more skills have been demonstrated, updating the one or more skill nodes comprises updating, by the management server and for display, the one or more skill nodes to graphically indicate that the one or more skills have been demonstrated by the first team during the training exercise.

11. The method of claim 10,

wherein the trainee is assigned a particular role on the first team, and

wherein the trainee is assigned to a task of demonstrating the one or more skills during the training exercise based on the particular role.

12. The method of claim 10, further comprising:

generating, by the management server, an exercise summary report of the training exercise, wherein the exercise summary report indicates whether the one or more skills have been demonstrated by the first team during the training exercise and whether the one or more learning objectives have been accomplished by the first team during the training exercise.

13. The method of claim 1, wherein the parameter data indicates at least one event that is associated with at least one action performed by the trainee during the training exercise.

14. The method of claim 1, further comprising:

generating, by the management server, an exercise summary report of the training exercise, wherein the exercise summary report includes whether the one or more skills have been demonstrated by the trainee during the training exercise and whether the one or more learning objectives have been accomplished by the trainee during the training exercise.

15. A computer-readable storage medium comprising instructions that, when executed, cause one or more processors to perform operations comprising:

outputting, for display, a graphical dashboard associated with a training exercise, wherein the graphical dashboard includes one or more learning objective nodes that represent one or more learning objectives to be accomplished by a trainee using at least one host computing system during the training exercise, wherein the graphical dashboard further includes one or more skill nodes that represent one or more skills to be demonstrated by the trainee during the training exercise, wherein the one or more skills support the one or more learning objectives, and wherein the one or more skill nodes graphically indicate that the one or more skills have not yet been demonstrated by the trainee;

determining one or more metrics that are usable to determine whether the one or more skills are demonstrated by the trainee based on one or more evaluation criteria;

identifying parameter data to be collected by the at least one host computing system during the training exercise to calculate the one or more metrics;

selecting, based on the one or more skills to be demonstrated by the trainee, one or more software agents that are associated with the one or more skill nodes and that are capable of collecting the parameter data;

after selecting the one or more software agents, providing, to the at least one host computing system, an indication of the one or more software agents that are executed on the at least one host computing system during the training exercise;

providing, to the at least one host computing system, one or more metric parameters that are used to configure the one or more software agents to collect the parameter data from the at least one host computing system while the trainee performs actions during the training exercise, wherein the one or more metric parameters include at least an operational mode parameter indicating whether the one or more software agents will terminate after collecting the parameter data or will instead stay resident on and monitor the at least one host computing system;

receiving the parameter data collected by the one or more software agents during execution on the at least one host computing system;

calculating, based on the parameter data collected by the one or more software agents, the one or more metrics usable to determine whether the one or more skills are demonstrated by the trainee based on the one or more evaluation criteria;

determining, based on the calculated one or more metrics, that the one or more skills have been demonstrated by the trainee during the training exercise; and

responsive to determining that the one or more skills have been demonstrated, updating, for display, the one or more skill nodes representing the one or more skills to graphically indicate that the one or more skills have been demonstrated by the trainee during the training exercise.

16. The computer-readable storage medium of claim 15,

wherein the graphical dashboard graphically indicates a hierarchical relationship between the one or more skill nodes and the one or more learning objective nodes, and

wherein before receiving the parameter data collected by the one or more software agents during execution, the one or more learning objective nodes graphically indicate that the one or more learning objectives have not yet been accomplished by the trainee.

17. The computer-readable storage medium of claim 16, wherein responsive to determining that the one or more skills have been demonstrated, the operations further comprise:

determining, based on the hierarchical relationship between the one or more skill nodes and the one or more learning objective nodes, that the one or more learning objectives represented by the one or more learning objective nodes have been accomplished by the trainee during the training exercise; and

responsive to determining that the one or more learning objectives have been accomplished, updating, for display, the one or more learning objective nodes to graphically indicate that the one or more learning objectives have been accomplished by the trainee during the training exercise.

18. The computer-readable storage medium of claim 17,

wherein the one or more skills comprise a plurality of skills,

wherein the one or more skill nodes comprise a plurality of skill nodes, and

wherein the hierarchical relationship in the graphical dashboard between the one or more skill nodes and the one or more learning objective nodes comprises a logical relationship between the plurality of skill nodes and a particular learning objective node of the one or more learning objective nodes, the particular learning objective node representing a particular learning objective of the one or more learning objectives.

19. A system comprising:

one or more processors; and

one or more computer-readable storage media storing instructions that, when executed, cause the one or more processors to;

output, for display, a graphical dashboard associated with a training exercise, wherein the graphical dashboard includes one or more learning objective nodes that represent one or more learning objectives to be accomplished by a trainee using at least one host computing system during the training exercise, wherein the graphical dashboard further includes one or more skill nodes that represent one or more skills to be demonstrated by the trainee during the training exercise, wherein the one or more skills support the one or more learning objectives, and wherein the one or more skill nodes graphically indicate that the one or more skills have not yet been demonstrated by the trainee;

determine one or more metrics that are usable to determine whether the one or more skills are demonstrated by the trainee based on one or more evaluation criteria;

identify parameter data to be collected by the at least one host computing system during the training exercise to calculate the one or more metrics;

select, based on the one or more skills to be demonstrated by the trainee, one or more software agents that are associated with the one or more skill nodes and that are capable of collecting the parameter data;

after selecting the one or more software agents, provide, to the at least one host computing system, an indication of the one or more software agents that are executed on the at least one host computing system during the training exercise;

provide, to the at least one host computing system, one or more metric parameters that are used to configure the one or more software agents to collect the parameter data from the at least one host computing system while the trainee performs actions during the training exercise, wherein the one or more metric parameters include at least an operational mode parameter indicating whether the one or more software agents will terminate after collecting the parameter data or will instead stay resident on and monitor the at least one host computing system;

receive the parameter data collected by the one or more software agents during execution on the at least one host computing system;

calculate, based on the parameter data collected by the one or more software agents, the one or more metrics usable to determine whether the one or more skills are demonstrated by the trainee based on the one or more evaluation criteria;

determine, based on the calculated one or more metrics, that the one or more skills have been demonstrated by the trainee during the training exercise; and

responsive to determining that the one or more skills have been demonstrated, update, for display, the one or more skill nodes representing the one or more skills to graphically indicate that the one or more skills have been demonstrated by the trainee during the training exercise.

20. The system of claim 19,

wherein the graphical dashboard graphically indicates a hierarchical relationship between the one or more skill nodes and the one or more learning objective nodes, and

wherein before the instructions cause the one or more processors to receive the parameter data collected by the one or more software agents during execution on the at least one host computing system, the one or more learning objective nodes graphically indicate that the one or more learning objectives have not yet been accomplished by the trainee.

21. The system of claim 20, wherein the instructions further cause the one or more processors to, responsive to determining that the one or more skills have been demonstrated:

determine, based on the hierarchical relationship between the one or more skill nodes and the one or more learning objective nodes, that the one or more learning objectives represented by the one or more learning objective nodes have been accomplished by the trainee during the training exercise; and

responsive to determining that the one or more learning objectives have been accomplished, update, for display, the one or more learning objective nodes to graphically indicate that the one or more learning objectives have been accomplished by the trainee during the training exercise.

22. The system of claim 21,

wherein the one or more skills comprise a plurality of skills,

wherein the one or more skill nodes comprise a plurality of skill nodes, and

wherein the hierarchical relationship in the graphical dashboard between the one or more skill nodes and the one or more learning objective nodes comprises a logical relationship between the plurality of skill nodes and a particular learning objective node of the one or more learning objective nodes, the particular learning objective node representing a particular learning objective of the one or more learning objectives.

23. The system of claim 19, wherein the instructions further cause the one or more processors to:

generate an exercise summary report of the training exercise, wherein the exercise summary report includes whether the one or more skills have been demonstrated by the trainee during the training exercise and whether the one or more learning objectives have been accomplished by the trainee during the training exercise.