Systems and methods for implementing and scoring computer network defense exercises

• US 9,325,728 B1
• Filed: 07/22/2013
• Issued: 04/26/2016
• Est. Priority Date: 01/27/2005
• Status: Expired due to Fees

• Current Assignee: Leidos, Inc. (Leidos Holdings, Inc.)
• Original Assignee: Leidos, Inc. (Leidos Holdings, Inc.)
• Inventors: Spencer, Myoki Elizabeth, Kennedy, Scott, Ayers, Carleton Rose II, Banks, Susan, Allison, Ian Carroll, Diaz, Michael Anthony
• Primary Examiner(s): Le, Chau
• Assistant Examiner(s): Chaudhry, Muhammad
• Application Number:
  US13/947,149
• Time in Patent Office: 1,009 Days
• US Class Current: 1/1
• CPC Class Codes: H04L 63/1433 Vulnerability analysis

Abstract

Processes for the implementation and scoring of computer and network system defense exercises include base and modified scoring in accordance with various parameters and across multiple types of participants including attackers, defenders, forensic analysts, judges and referees. Effects-based scoring includes scoring for ascertaining status of an external client system by a participant in an independent client system.

Claims

1. A process for scoring a client network system defense training exercise implemented over a services-oriented architecture comprising:

receiving by at least one server machine from the client network system data related to a defense training exercise at an end of one or more of predetermined scoring cycles or upon request, the data including one or more of a current list of client network system vulnerabilities, details of client system vulnerability fixes, details of client system vulnerability identifications and details of client system vulnerability exploitations, wherein the details include a unique identifier for each of a vulnerability fixer, a vulnerability identifier and a vulnerability exploiter and a time value for each of time to fix, time to identify and time to exploit;

applying a set of scoring rules by a scoring engine of the at least one server machine to the details to determine one or more base scores at the end of one or more of predetermined scoring cycles or upon request from a second server machine for each participant in the defense training exercise, wherein the participant is an individual user or a group of users;

determining an actual score for each participant using the one or more base scores at the end of the defense training exercise;

receiving by at least one server machine from the client network system at the end of one or more of predetermined scoring cycles or upon request, data related to the operation and maintenance of one or more critical services of the client network system, including details related to percentage up time of the one or more critical services; and

applying the set of scoring rules by the scoring engine of the at least one server machine to the details related to percentage up time of the one or more critical services and, where applicable, including the result in the calculation of the one or more objective scores for each participant.

2. The process according to claim 1, wherein the set of scoring rules is different depending on a role of the participant.

3. The process according to claim 2, wherein the roles of the participant are selected from the group consisting of:

defender of the client network system, attacker of the client network system, forensic analyst of the client network system, judges of the one or more defense training exercises and referees of the one or more defense training exercises.

4. The process according to claim 3, wherein the one or more defense training exercises is a computer network defense (CND) game and includes at least one defender of the client network system, at least one attacker of the client network system, and at least one judge;

wherein defenders are scored on ability to defend one or more computers from attacks thereon by the attackers during each of one or more predetermined scoring cycles as judged by the judges;

the attackers are scored on their ability to compromise one or more computers during each of one or more predetermined scoring cycles as judged by the judges; and

the judges are scored based on one or more of accuracy of scoring other participants and accuracy of one or more task injects to the CND game in response to status.

5. The process of claim 4, wherein the one or more task injects include one or more of:

development of a security policy document, instantiation of a new host, deploying a new critical service and introduction of additional vulnerabilities to systems or targets after a defense training exercise has started.

6. The process according to claim 3, wherein the one or more defense training exercises is a forensics game and includes one or more forensic analysts of the client network system;

wherein the forensic analysts are scored on ability to collect, investigate, analyze and report on data resulting from actions of an attacker, such data including at least one of the following;

identification of method of intrusion, identification of IP address of the attacker, identification of evidence of misuse left by the attacker, ascertaining timeline of hostile activity, identification of MD5 hash value of malware, and identification of username of an account that was attacked.

7. The process according to claim 3, wherein the one or more defense training exercises is a computer network attack (CNA) game and includes multiple individual attackers;

wherein the multiple individual attackers are scored on ability to analyze and exploit authorized targets in the client network system as evidenced by at least one of insertion of an identifying flag in one or more target locations and reporting of an artifact found.

8. The process of claim 7, wherein artifacts found include one or more of:

a hash value stored as credit card data in a database;

a hash value stored in ContactName in the SNMP MIB;

a hash value in a user file; and

a hash value as a command in the administrative page for a web application.

9. The process according to claim 3, wherein the one or more defense training exercises is a capture the flag (CTF) game and includes multiple individual attackers and multiple individual defenders acting in both attacker and defender capacities as the CTF game progresses;

wherein the multiple individual attackers and multiple individual defenders are scored based one or more of ability to gain control of one or more centrally available targets and management of appointed targets while simultaneously attacking the targets run by others.

10. The process according to claim 9, wherein additional targets are introduced after the start of the CTF game.

11. The process according to claim 1, wherein the one or more base objective scores are comprised of at least three of the components selected from the following:

percentage of weighted vulnerabilities fixed per host;

percentage of weighted availability of critical services per host;

percentage dependent on meeting predetermined service level agreement requirements for weighted availability of critical services per host; and

percentage of completed injects per host,

wherein the sum of the at least three percentages must equal 100 percent.

12. The process according to claim 4, wherein the base score and actual score are determined as follows:

Base score={per host weighting}*({Percent of Base Score dependent on vulnerabilities}*Σ

{Vulnerabilities Fixed}*their weighting/(Σ

{all Vulnerabilities}*their weighting))+({Percent of Base Score dependent on availability of critical services}*Σ

{Critical Services ok}*their weighting/(Σ

{all Critical Services}*their weighting))+({Percent of Base Score dependent on meeting SLA}*{True/False was the SLA met?})+({Percent of Base Score dependent on completed injects}*{value of completed injects})

Actual Score=Base Score*{The value of Weighting based on current time}+Positive actions−

Negative Actions, wherein SLA is Service Level Agreement.

13. The process according to claim 6, wherein the base score and actual score are determined as follows:

Base score={per host weighting}*(Σ

{Artifacts Found}*their weightings/(Σ

{all Artifacts}*their weightings))

Actual Score=Base Score*{The value of Weighting based on current time}+Positive/Negative judge scoring,

wherein judge scoring is an evaluation of the accuracy of one or more reports from the forensic analyst.

14. The process according to claim 9, wherein the base score and actual score are determined as follows:

Base score={per host weighting}*({Percent of Base Score dependent on vulnerabilities}*Σ

{Vulnerabilities Fixed}*their weighting/(Σ

{all Vulnerabilities}*their weighting))+({Percent of Base Score dependent on availability of critical services}*Σ

{Critical Services ok}*their weighting/(Σ

{all Critical Services}*their weighting))+({Percent of Base Score dependent on meeting SLA}*{True/False was the SLA met?})*(True/False Is the flag placed for this user/team?)

Actual Score=Base Score*{The value of Weighting based on current time}+Positive actions−

Negative Actions, wherein SLA is Service Level Agreement.

15. The process according to claim 10, wherein the base score and actual score are determined as follows:

Base score={per host weighting}*({Percent of Base Score dependent on vulnerabilities}*Σ

{Vulnerabilities Fixed}*their weighting/(Σ

{all Vulnerabilities}*their weighting))+({Percent of Base Score dependent on availability of critical services}*Σ

{Critical Services ok}*their weighting/(Σ

{all Critical Services}*their weighting))+({Percent of Base Score dependent on meeting SLA}*{True/False was the SLA met?}))*(True/False Is the flag placed for this user/team?)

Actual Score=Base Score*{The value of Weighting based on current time}+Positive actions−

Negative Actions, wherein SLA is Service Level Agreement.