Clustering approach for detecting DDoS botnets on the cloud from IPFix data

• US 10,129,295 B2
• Filed: 08/31/2016
• Issued: 11/13/2018
• Est. Priority Date: 08/31/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Karin, Omer, Ronen, Royi, Neuvirth, Hani, Vilnai, Roey
• Primary Examiner(s): Armouche, Hadi S
• Assistant Examiner(s): Wade, Shaqueal D
• Application Number:
  US15/253,586
• Publication Number:
  US 20180063188A1
• Time in Patent Office: 804 Days
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 2463/144 Detection or countermeasures against botnets
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 63/1466 Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks

Abstract

Use machine learning to train a classifier to classify entities to increase confidence with respect to an entity being part of a distributed denial of service attack. The method includes training a classifier to use a first classification method, to identify probabilities that entities from a set of entities are performing denial of service attacks. The method further includes identifying a subset of entities meeting a threshold probability of performing a denial of service attack. The method further includes using a second classification method, identifying similarity of entities in the subset of entities. The method further includes based on the similarity, classifying individual entities.

Claims

1. A system configured to train and use a classifier to classify entities to determine whether the entities are part of a distributed denial of service (DDoS) attack, the system comprising:

one or more hardware processors; and

one or more computer-readable storage devices having stored thereon instructions that are executable by the one or more hardware processors to configure the system to perform at least the following;

train a classifier to use a first classification method to identify probabilities that entities are performing denial of service attacks, the training comprising applying a captured dataset including data flow protocol information associated with known DDoS attacks;

using the trained classifier, identify a subset of entities from a set of candidate entities that meet or exceed a threshold probability of performing a denial of service attack;

using a second classification method, identify similarity of entities in the identified subset of entities; and

based on the similarity, classify individual entities of the subset of entities as belonging to one or more similarity subgroups, each similarity subgroup comprising entities having a probability of participating in a same DDoS.

2. The system of claim 1, wherein the second classification method clusters similar entities into similarity clusters.

3. The system of claim 2, wherein the one or more computer-readable storage devices further have stored thereon instructions that are executable by the one or more hardware processors to configure the computer system to identify a cluster as a set of compromised entities.

4. The system of claim 1, wherein classifying individual entities comprises identifying entities as performing denial of service.

5. The system of claim 1, wherein the one or more computer-readable storage devices further have stored thereon instructions that are executable by the one or more hardware processors to configure the computer system to identify entities in a particular botnet based on similarity.

6. The system of claim 1, wherein the one or more computer-readable storage devices further have stored thereon instructions that are executable by the one or more processors to configure the computer system to identify entities infected by the same means based on similarity.

7. The system of claim 6, wherein the same means comprises the same malicious software.

8. The system of claim 6, wherein the same means comprises the same command and control.

9. The system of claim 1, wherein using the second classification method, to identify similarity of entities in the subset of entities comprises using the L-method.

10. The system of claim 1, wherein using the second classification method, to identify similarity of entities in the subset of entities comprises using hierarchal clustering.

11. The system of claim 1, wherein the one or more computer-readable storage devices further have stored thereon instructions that are executable by the one or more hardware processors to configure the computer system to correlate entity activity and wherein using hierarchal clustering is based on correlated entity activity.

12. The system of claim 1, wherein the one or more computer-readable storage devices further have stored thereon instructions that are executable by the one or more hardware processors to configure the computer system to use available external data to identify a particular botnet.

13. A computer implemented method for training a classifier for classifying entities to determine whether the entities are part of a distributed denial of service (DDoS) attack, the method comprising:

training a classifier to use a first classification method to identify probabilities that entities are performing denial of service attacks, the training comprising applying a captured dataset including data flow protocol information associated with known DDoS attacks;

using the trained classifier, identifying a subset of entities from a set of candidate entities that meet or exceed a threshold probability of performing a denial of service attack;

using a second classification method, identifying similarity of entities in the subset of identified entities; and

based on the similarity, classifying individual entities of the subset of entities as belonging to one or more similarity subgroups, each similarity subgroup comprising entities having a probability of participating in a same DDoS.

14. The method of claim 13, wherein the second classification method clusters similar entities into similarity clusters.

15. The method of claim 14, further comprising identifying a cluster as a set of compromised entities.

16. The method of claim 13, wherein classifying individual entities comprises identifying entities as performing denial of service.

17. The method of claim 13, further comprising identifying entities in a particular botnet based on similarity.

18. The method of claim 13, further comprising identifying entities infected by the same means based on similarity.

19. The method of claim 18, wherein the same means comprises the same malicious software.

20. A computer system configured to use a trained classifier to classify entities to determine whether the entities are part of a distributed denial of service (DDoS) attack, the system comprising:

a botnet classifier coupled to a plurality of computing entities, the botnet classifier comprising one or more computer processors, wherein the botnet classifier is configured to;

capture data flow protocol information from the entities in the plurality of entities;

provide the captured data flow protocol information from the entities to a trained classifier, the trained classifier having been trained by applying previously captured data including data flow protocol information associated with known DDoS attacks;

the trained classifier implementing a first classification method to identify probabilities that entities are performing denial of service attacks based on the captured data flow protocol information;

identify a subset of entities from a set of candidate entities that meet or exceed a threshold probability of performing a denial of service attack;

use a second classification method, identify similarity of entities in the identified subset of entities; and

based on the similarity, classify individual entities of the subset of entities as belonging to one or more similarity subgroups, each similarity subgroup comprising entities having a probability of participating in a same DDoS.