System and method for high speed threat intelligence management using unsupervised machine learning and prioritization algorithms

• US 10,713,586 B2
• Filed: 07/24/2015
• Issued: 07/14/2020
• Est. Priority Date: 07/24/2015
• Status: Active Grant

• Current Assignee: Certis Cisco Security Pte Limited (Temasek Holdings Pte Ltd. (Investment Management))
• Original Assignee: Certis Cisco Security Pte Limited (Temasek Holdings Pte Ltd. (Investment Management))
• Inventors: Lim, Keng Leng Albert
• Primary Examiner(s): Gonzales, Vincent
• Assistant Examiner(s): Figueroa, Kevin W
• Application Number:
  US14/891,621
• Publication Number:
  US 20170228658A1
• Time in Patent Office: 1,817 Days
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06F 18/24147 Distances to closest patterns, e.g. nearest neighbour classification
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

This document discloses a system and method for consolidating threat intelligence data for a computer and its related networks. Massive volumes of raw threat intelligence data are collected from a plurality of sources and are partitioned into a common format for cluster analysis whereby the clustering of the data is done using unsupervised machine learning algorithms. The resulting organized threat intelligence data subsequently undergoes a weighted asset based threat severity level correlation process. All the intermediary network vulnerabilities of a particular computer network are utilized as the critical consolidation parameters of this process. The final processed intelligence data gathered through this high speed automated process is then formatted into predefined formats prior to transmission to third parties.

Claims

1. A method of consolidating threat intelligence data for a computer network, the method to be performed by a computer system comprising:

collecting threat intelligence data from a plurality of sources and normalizing the collected threat intelligence data into a uniform data format;

grouping normalized threat intelligence data into clusters using unsupervised machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data;

categorizing clusters that are severe to the computer network;

comparing the clusters categorized as severe with a security posture of the computer network to determine clusters that are of interest to the computer system; and

formatting the clusters determined to be of interest to the computer system to a predefined format of the computer network; and

wherein the categorizing the clusters that are severe to the computer network comprises;

retrieving a list of computer assets associated with the computer network;

identifying clusters that affect a computing feature of the computer assets; and

classifying identified clusters that affect a computing feature of the computer asset as severe to the computer network.

2. The method according to claim 1 further comprising retrieving severity weightage values accorded to each of the computer assets associated with the computer network;

summing the retrieved severity weightage values; and

allocating the summed severity weightage value to the computer network.

3. The method according to claim 1 wherein the computing feature comprises an operating system or a network protocol of a computer asset.

4. The method according to claim 1, wherein before the comparing the clusters categorized as severe with a security posture of the computer network to determine clusters of interest to the computer system, the method further comprises:

generating the security posture of the computer network.

5. The method according to claim 4 wherein the generating the security posture of the computer network comprises:

creating an object model representing the computer network, wherein the object model includes computer security information of computer assets contained within the computer network; and

executing an analysis program operative to run vulnerability testing of each of the computer assets in the computer network using the object model, wherein the results of the vulnerability testing are used to determine the security posture of the computer network.

6. The method according to claim 5, wherein the vulnerability testing of each of the computer assets in the computer network using the object model comprises tests pertaining to system level and topology vulnerabilities of the computer network, and node level vulnerabilities of the computer assets.

7. The method according to claim 1 wherein the grouping normalised threat intelligence data into clusters machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data further comprises:

validating the clusters using threat intelligence data in each cluster.

8. The method according to claim 7 wherein the validating the clusters comprises:

assigning weightage values to each record contained in the clusters, wherein a record originating from an open source is assigned a lower weightage value as compared to a weightage value assigned to a record originating from a commercial source;

summing the weightage values of records contained in each cluster; and

validating clusters that have summed weightage values that exceed a predefined threshold.

9. The method according to claim 1 further comprising:

using the formatted clusters to update the security posture of the computer network.

10. The method according to claim 1 wherein the attribute of the threat intelligence data comprises a computer security threat or an Internet Protocol (IP) address.

11. A system for consolidating threat intelligence data for a computer network comprising:

a processing unit; and

a non-transitory media readable by the processing unit, the media storing instructions that when executed by the processing unit, cause the processing unit to;

collect threat intelligence data from a plurality of sources and normalize the collected threat intelligence data into a uniform data format;

group normalized threat intelligence data into clusters using unsupervised machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data;

categorize dusters that are severe to the computer network;

compare the clusters categorized as severe with a security posture of the computer network to determine clusters that are of interest to the computer system; and

format the clusters determined to be of interest to the computer system to a predefined format of the computer network; and

wherein the instructions to categorize the clusters that are severe to the computer network comprises;

instructions for directing the processing unit to;

retrieve a list of computer assets associated with the computer network;

identify dusters that affect a computing feature of the computer assets; and

classify identified clusters that affect a computing feature of the computer asset as severe to the computer network.

12. The system according to claim 11 further comprising:

instructions for directing the processing unit to;

retrieve severity weightage values accorded to each of the computer assets associated with the computer network;

sum the retrieved severity weightage values; and

allocate the summed severity weightage value to the computer network.

13. The system according to claim 11 wherein the computing feature comprises an operating system or a network protocol of a computer asset.

14. The system according to claim 11, wherein before the instructions to compare the clusters categorized as severe with a security posture of the computer network to determine clusters of interest to the computer system, the system further comprises:

instructions for directing the processing unit to;

generate the security posture of the computer network.

15. The system according to claim 14 wherein the instructions to generate the security posture of the computer network comprises:

instructions for directing the processing unit to;

create an object model representing the computer network, wherein the object model includes computer security information of computer assets contained within the computer network; and

execute an analysis program operative to run vulnerability testing of each of the computer assets in the computer network using the object model, wherein the results of the vulnerability testing are used to determine the security posture of the computer network.

16. The system according to claim 15, wherein the vulnerability testing of each of the computer assets in the computer network using the object model comprises tests pertaining to system level and topology vulnerabilities of the computer network, and node level vulnerabilities of the computer assets.

17. The system according to claim 11 wherein the instructions to group normalised threat intelligence data into clusters machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data further comprises:

instructions for directing the processing unit to;

validate the clusters using threat intelligence data in each cluster.

18. The system according to claim 17 wherein the instructions to validate the clusters comprises:

instructions for directing the processing unit to;

assign weightage values to each record contained in the clusters, wherein a record originating from an open source is assigned a lower weightage value as compared to a weightage value assigned to a record originating from a commercial source;

sum the weightage values of records contained in each cluster; and

validate clusters that have summed weightage values that exceed a predefined threshold.

19. The system according to claim 11 further comprising:

instructions for directing the processing unit to;

use the formatted clusters to update the security posture of the computer network.

20. The system according to claim 11 wherein the attribute of the threat intelligence data comprises a computer security threat or an Internet Protocol (IP) address.

21. A system for consolidating threat intelligence data for a computer network comprising:

circuitry configured to collect threat intelligence data from a plurality of sources and normalize the collected threat intelligence data into a uniform data format;

circuitry configured to group normalized threat intelligence data into clusters using unsupervised machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data;

circuitry configured to categorize clusters that are severe to the computer network;

circuitry configured to compare the clusters categorized as severe with a security posture of the computer network to determine clusters that are of interest to the computer system; and

circuitry configured to format the clusters determined to be of interest to the computer system to a predefined format of the computer network; and

wherein the circuitry configured to categorize the clusters that are severe to the computer network comprises;

circuitry configured to retrieve a list of computer assets associated with the computer network;

circuitry configured to identify clusters that affect a computing feature of the computer assets; and

circuitry configured to classify identified clusters that affect a computing feature of the computer asset as severe to the computer network.

22. The system according to claim 21 further comprising:

circuitry configured to retrieve severity weightage values accorded to each of the computer assets associated with the computer network;

circuitry configured to sum the retrieved severity weightage values; and

circuitry configured to allocate the summed severity weightage value to the computer network.

23. The system according to claim 21 wherein the computing feature comprises an operating system or a network protocol of a computer asset.

24. The system according to claim 21, wherein before the circuitry configured to compare the clusters categorized as severe with a security posture of the computer network to determine clusters of interest to the computer system, the system further comprises:

circuitry configured to generate the security posture of the computer network.

25. The system according to claim 24 wherein the circuitry configured to generate the security posture of the computer network comprises:

circuitry configured to create an object model representing the computer network, wherein the object model includes computer security information of computer assets contained within the computer network; and

circuitry configured to execute an analysis program operative to run vulnerability testing of each of the computer assets in the computer network using the object model, wherein the results of the vulnerability testing are used to determine the security posture of the computer network.

26. The system according to claim 25, wherein the vulnerability testing of each of the computer assets in the computer network using the object model comprises tests pertaining to system level and topology vulnerabilities of the computer network, and node level vulnerabilities of the computer assets.

27. The system according to claim 21 wherein the circuitry configured to group normalized threat intelligence data into clusters machine learning algorithms, wherein each cluster comprises a group of data that represents an attribute of the threat intelligence data further comprises:

circuitry configured to validate the clusters using threat intelligence data in each cluster.

28. The system according to claim 27 wherein the circuitry configured to validate the clusters comprises:

circuitry configured to assign weightage values to each record contained in the clusters, wherein a record originating from an open source is assigned a lower weightage value as compared to a weightage value assigned to a record originating from a commercial source;

circuitry configured to sum the weightage values of records contained in each cluster; and

circuitry configured to validate clusters that have summed weightage values that exceed a predefined threshold.

29. The system according to claim 21 further comprising:

circuitry configured to use the formatted clusters to update the security posture of the computer network.

30. The system according to claim 21 wherein the attribute of the threat intelligence data comprises a computer security threat or an Internet Protocol (IP) address.