Automated machine learning scheme for software exploit prediction

• US 10,264,009 B2
• Filed: 07/26/2016
• Issued: 04/16/2019
• Est. Priority Date: 07/26/2016
• Status: Active Grant

• Current Assignee: BOOZ Allen Hamilton Holding Corporation
• Original Assignee: BOOZ Allen Hamilton Holding Corporation
• Inventors: Smyth, Eric, Sant-Miller, Aaron, Field, Kevin
• Primary Examiner(s): Gee, Jason K
• Assistant Examiner(s): Cattunal, Dereena T
• Application Number:
  US15/219,713
• Publication Number:
  US 20180034842A1
• Time in Patent Office: 994 Days
• Field of Search: 726 25
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 63/1433 Vulnerability analysis

Abstract

A predictive engine for analyzing existing vulnerability information to determine the likelihood of a vulnerability being exploited by malicious actors against a particular computer or network of computers. The predictive engine relies on multiple data sources providing historical vulnerability information, a plurality of predictive models, and periodic retraining of the prediction ensemble utilizing predictive models. Modeling schemes may also be used when retraining the predictive models forming the prediction ensemble.

Claims

1. A method for providing a threat level based on a prediction of a likelihood that a current vulnerability is employed in a customer computer network, the method comprising:

retrieving a prediction ensemble previously used by a prediction engine the prediction ensemble defining a combination of a plurality of prediction models and adjustments to be used by the prediction engine

retraining the prediction ensemble previously used by the prediction engine the retraining evaluating a combined data set of historical vulnerability information and a modeling scheme (130) to generate a revised prediction ensemble based on a plurality of prediction models;

retrieving current vulnerability information describing a current vulnerability from the plurality of data sources;

processing the current vulnerability information describing the current vulnerability for use by the prediction engine;

generating, using the prediction engine, a prediction based on the revised prediction ensemble and the processed current vulnerability information, the prediction identifying the likelihood that the current vulnerability is employed in a customer computer network;

providing to an administrator of the customer computer network the threat level based on the prediction of the likelihood the current vulnerability is employed in the customer computer network, the threat level being based on predetermined thresholds defined by the administrator of the customer computer network;

translating vulnerability descriptions from the historical vulnerability information to defined values;

performing textual analysis to extract additional information from the historical vulnerability information;

correlating the defined values, the additional information, and untranslated source information, and

generating the combined data set based on the correlated defined values, the additional information, and the untranslated source information of the historical vulnerability information.

2. The method of claim 1, comprising:

retrieving the historical vulnerability information from a plurality of data sources; and

indexing the historical vulnerability information.

3. The method of claim 2, wherein the processing of the current vulnerability information describing the current vulnerability for use by the prediction engine comprises:

indexing the current vulnerability information;

translating vulnerability descriptions from the current vulnerability information into defined values;

performing textual analysis to extract additional information;

correlating the defined values, the additional information, and untranslated source information; and

generating the processed current vulnerability information based on the correlated defined values, additional information, and untranslated source information of the current vulnerability information.

4. The method of claim 1, wherein generating the revised prediction ensemble based on the plurality of prediction models comprises:

receiving a modeling scheme containing adjustments to an existing prediction ensemble; and

training the plurality of prediction models using a training set created from at least a portion of the combined data set and the modeling scheme to revise the prediction ensemble.

5. The method of claim 4, wherein the training of the plurality of predictive models using the training set created from the portion of the combined data set and the modeling scheme comprises:

splitting the combined data set into the training set and a testing set

running the plurality of predictive models against the training set to generate training predictions;

running the plurality of predictive models against the testing set to generate testing predictions; and

revising the prediction ensemble based on the generated training predictions and the generated testing predictions.

6. The method of claim 1, wherein the generation, using the prediction engine, of the prediction based on the revised prediction ensemble, and the processed current vulnerability information comprises:

updating the prediction engine to use the revised prediction ensemble; and

executing the prediction engine with the revised prediction ensemble (155) of the plurality of predictive models against the processed current vulnerability information to obtain the prediction.

7. The method of claim 3, wherein the historical vulnerability information from the plurality of data sources includes a common identifier for each vulnerability;

wherein the generated combined data set is correlated using the common identifier; and

wherein the processed current vulnerability information includes a common identifier for the current vulnerability.

8. The method of claim 1, wherein the retraining of the prediction engine occurs periodically to consider new historical vulnerability information retrieved from the plurality of data sources.

9. The method of claim 1, wherein the generated prediction is further based on a configuration of the customer computer network.

10. A system for providing a threat level based on a prediction of a likelihood that a current vulnerability is employed in a customer computer network, the system comprising:

a processor configured to generate and store in a data repository a combined data set of historical vulnerability information obtained from a plurality of data sources, wherein the processor is configured to translate vulnerability descriptions from the historical vulnerability information into defined values, perform textual analysis to extract additional information from the historical vulnerability information, correlate the defined values, the additional information, and untranslated source information, and generate the combined data set based on the correlated defined values, the additional information, and the untranslated source information;

a retraining server configured to retrain a prediction ensemble previously used for execution by a prediction engine, the retraining evaluating the combined data set and a received modeling scheme to generate a revised prediction ensemble based on a plurality of prediction models;

wherein the prediction ensemble defines a combination of the plurality of prediction models and adjustments to be used by the prediction engine;

the data repository retrieving, from a plurality of data sources, current vulnerability information describing a current vulnerability;

the data repository processing the current vulnerability information describing the current vulnerability for use by the prediction engine; and

a prediction engine execution server configured to generate a prediction using the prediction engine, the prediction being based on the revised prediction ensemble and the processed current vulnerability information, the prediction providing the likelihood that the current vulnerability is employed in a customer computer network;

wherein the prediction engine execution server provides to an administrator of the customer computer network the threat level based on the prediction of the likelihood the current vulnerability is employed in the customer computer network, the threat level being based on predetermined thresholds defined by the administrator of the customer computer network.

11. The system of claim 10, wherein the processor is configured to:

retrieve the historical vulnerability information from the plurality of data sources; and

index the historical vulnerability information,

wherein the data repository transmits at least a portion of the combined data set to the retraining server.

12. The system of claim 11, wherein the data repository processes the current vulnerability information describing the current vulnerability for use by the prediction engine by:

retrieving the current vulnerability information from the plurality of data sources

indexing the current vulnerability information;

translating vulnerability descriptions from the current vulnerability information into defined values;

performing textual analysis to extract additional information from the historical vulnerability information;

correlating the defined values, the additional information, and untranslated source information; and

generating the processed data set based on the correlated defined values, additional information, and untranslated source information;

wherein the data repository transmits at least the processed current vulnerability information to the retraining server.

13. The system of claim 10, wherein the retraining server generates the revised prediction ensemble based on the plurality of prediction models by:

receiving a modeling scheme from a modeling server containing adjustments to an existing prediction ensemble stored in the retraining server;

training the plurality of prediction models at the retraining server using a training set created from at least a portion of the combined data set received from the data repository and the received modeling scheme to revise the stored prediction ensemble.

14. The system of claim 13, wherein the retraining server trains the plurality of predictive models using the training set created from the portion of the combined data set received from the data repository and the received modeling scheme by:

splitting the combined data set received from the data repository into the training set and a testing set

running the plurality of predictive models at the retraining server against the training set to generate training predictions;

running the plurality of predictive models at the retraining server against the testing set to generate testing predictions; and

revising the existing prediction ensemble stored on the retraining server based on the generated training predictions and the generated testing predictions.

15. The system of claim 10, wherein the prediction engine execution server, using the prediction engine executing at the prediction engine execution server, generates the prediction based on the revised prediction ensemble transmitted by the retraining server, and the processed current vulnerability information transmitted by the data repository by:

updating the prediction engine using the received revised prediction ensemble and

executing the prediction engine with the received revised prediction ensemble of the plurality of predictive models against the processed current vulnerability information to obtain the prediction.

16. The system of claim 12, wherein the historical vulnerability information from the plurality of data sources includes a common identifier for each vulnerability;

wherein the generated combined data set is correlated using the common identifier; and

wherein the processed current vulnerability information includes a common identifier for the current vulnerability.

17. The system of claim 10, wherein the retraining server retrains the prediction engine periodically to consider new historical vulnerability information retrieved from the plurality of data sources.

18. The system of claim 10, wherein the generated prediction is further based on a configuration of the customer computer network.