DETECTION AND CATEGORIZATION OF MALICIOUS URLS

• US 20120158626A1
• Filed: 12/15/2010
• Published: 06/21/2012
• Est. Priority Date: 12/15/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhu, Bin Benjamin, Choi, Hyunsang, Lee, Heejo
• Granted Patent: US 8,521,667 B2
• Time in Patent Office: Days
• US Class Current: 706/13
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00

Abstract

This document describes techniques for using features extracted from a URL to detect a malicious URL and categorize the malicious URL as one of a phishing URL, a spamming URL, a malware URL or a multi-type attack URL. The techniques employ one or more machine learning algorithms to train classification models using a set of training data which includes a known set of benign URLs and a known set of malicious URLs. The classification models are then employed to detect and/or categorize a malicious URL.

Claims

1. A method comprising:

receiving a uniform resource locator (URL);

extracting features associated with the URL, the features including at least one link popularity feature of the URL;

employing, via one or more processors, a binary classification model to determine that the URL is a malicious URL based at least in part on the extracted features; and

categorizing the malicious URL as one of a benign URL, a spam URL, a phishing URL, a malware URL, or a multi-type attack URL.

2. The method as recited in claim 1, wherein the categorizing employs a multi-label classification model to label the malicious URL in accordance with the categorization, the method further comprising:

processing a set of training URLs;

employing a binary classification machine learning algorithm to construct a detection decision criteria used by the binary classification model based at least in part on information extracted from the set of training URLs; and

employing a multi-label classification machine learning algorithm to construct a labeling decision criteria used by the multi-label classification model based at least in part on information extracted from the set of training URLs.

3. The method as recited in claim 2, wherein:

the binary classification machine learning algorithm is a support vector machine (SVM); and

the multi-label classification machine learning algorithm is selected from RAkEL or ML-kNN.

4. The method as recited in claim 1, wherein at least one feature extracted is a lexical feature selected from a group comprising:

a domain token count,

a path token count,

a longest domain token length,

a longest path token length,

an average domain token length,

an average path token length, and

a brand name presence.

5. The method as recited in claim 1, wherein at least one feature extracted is a lexical feature that determines a second level domain hit ratio for the received URL, wherein the second level domain hit ratio is based on how many times a second level domain of the received URL matches a corresponding second level domain of a known malicious URL compared to how many times the second level domain of the received URL matches a corresponding second level domain of a known benign URL.

6. The method as recited in claim 1, wherein the at least one link popularity feature is selected from a group comprising:

a maximum domain link ratio,

a phishing link ratio,

a spamming link ratio, and

a malware link ratio.

7. The method as recited in claim 1, wherein the at least one link popularity feature is a distinct domain link ratio that compares a number of unique domains that link to the received URL to a total number of incoming links that link to the received URL.

8. The method as recited in claim 1, wherein at least one feature extracted is a web page feature selected from a group comprising:

a hyper-text markup language (HTML) tag count,

an iframe count,

a zero size iframe count,

a line count,

a hyperlink count,

a script count, and

a script function count.

9. The method as recited in claim 1, wherein at least one feature extracted is a network feature selected from a group comprising:

a redirection count,

a download packet content length,

actual downloaded bytes,

a domain lookup time, and

a download speed.

10. The method as recited in claim 1, wherein at least one feature extracted is a domain name system (DNS) feature selected from a group comprising:

a resolved internet protocol (IP) count,

a name server count,

a name server IP count,

a malicious autonomous system number (ASN) ratio of resolved IPs, and

a malicious ASN ratio of name server IPs.

11. The method as recited in claim 1, wherein at least one feature extracted is a domain name system (DNS) fluxiness feature which indicates a change in domain name properties between DNS lookups.

12. The method as recited in claim 1, further comprising:

extracting at least one lexical feature, at least one web page content feature, at least one network feature, at least one domain name system (DNS) feature, and at least one DNS fluxiness feature; and

combining the at least one link popularity feature, the at least one lexical feature, the at least one web page content feature, the at least one network feature, the at least one domain name system (DNS) feature, and the at least one DNS fluxiness feature to arrive at a set of features used to determine that the URL is a malicious URL.

13. A system comprising:

one or more processors; and

a memory, coupled to the one or more processors, storing executable instructions including;

a link popularity feature extraction module to extract link popularity features; and

a binary classification model that uses the link popularity features to classify a received uniform resource locator (URL) as a malicious URL or a benign URL.

14. The system of claim 13, wherein the memory further includes a multi-label classification model that uses the link popularity features to classify the received URL as one of a phishing URL, a spamming URL, a malware URL, or a multi-type attack URL

15. The system of claim 13, wherein the memory further includes:

a lexicology feature extraction module that extracts lexical features associated with the received URL;

a web page feature extraction module that extracts web page content features associated with the received URL;

a network feature extraction module that extracts network features associated with the received URL; and

a domain name system (DNS) feature extraction module that extracts DNS features and DNS fluxiness features associated with the received URLs.

16. A method comprising:

receiving a uniform resource locator (URL) from a web browser or a search engine;

extracting one or more link popularity features associated with the URL;

employing one or more classification models to determine whether the URL is a malicious URL based on the one or more link popularity features; and

providing, via a computing device, a notification when the URL is a malicious URL.

17. The method as recited in claim 16, further comprising:

extracting one or more lexical features, one or more web page content features, one or more network features, one or more domain name system (DNS) features, and one or more DNS fluxiness features; and

further employing the one or more classification models to determine whether the URL is a malicious URL based on the one or more lexical features, the one or more web page content features, the one or more network features, the one or more domain name system (DNS) features, and the one or more DNS fluxiness features.

18. The method as recited in claim 16, further comprising:

in an event the URL is a malicious URL, employing the one or more classification models to label the malicious URL as one of a spam URL, a phishing URL, a malware URL, or a multi-type attack URL.

19. The method as recited in claim 16, further comprising;

adapting the one or more classification models using one or more learning algorithms when the URL is verified as a benign URL, or verified as a malicious URL of a known type.

20. The method as recited in claim 16, further comprising constructing the one or more classification models using one or more learning algorithms and a set of known malicious URLs and known benign URLs.