Detecting malicious HTTP redirections using user browsing activity trees

• US 9,912,680 B2
• Filed: 12/02/2016
• Issued: 03/06/2018
• Est. Priority Date: 12/24/2012
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: Narus, Inc. (Gen Digital Inc.)
• Inventors: Torres, Ruben, Mekky, Hesham, Zhang, Zhi-Li, Saha, Sabyasachi, Nucci, Antonio
• Primary Examiner(s): Tran, Tri
• Application Number:
  US15/368,349
• Publication Number:
  US 20170085583A1
• Time in Patent Office: 459 Days
• CPC Class Codes: G06F 21/50 Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2221/2119 Authenticating web pages, e.g. with suspicious links
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/168 above the transport layer
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A method for detecting malicious HTTP redirections. The method includes obtaining, based on a single client IP address, HTTP flows triggered by visiting a website, extracting a sequence of URLs where a downstream URL is extracted from a child HTTP request that is triggered by a parent HTTP request containing an immediate upstream URL, analyzing the URL sequence to generate a statistical feature, and classifying, based on the statistical feature, the HTTP flows as containing at least one malicious HTTP redirection triggered by visiting the website.

Claims

1. A method for detecting malicious HTTP redirections in a network, comprising:

obtaining, from the network and based on a single client IP address, one or more HTTP flows triggered by a client device visiting a website, wherein the one or more HTTP flows comprises a first sequence of universal resource locators (URLs);

constructing a per-user tree using the one more HTTP flows, the per-user tree including nodes corresponding to URLs, including the first sequence of URLs, wherein the per-user tree includes an edge from a parent node to a child node if a request for a URL corresponding to the child node is triggered from the URL corresponding to the parent node, wherein each edge of the per-user tree corresponds to;

1) a URL type assigned to the URL corresponding to the child node and

     2) a time that elapses between HTTP requests in the parent node and child node, wherein the per-user tree includes multiple paths, the multiple paths corresponding to both benign requests and malicious paths;

extracting, from the first sequence, a second sequence of URLs comprising an upstream URL and a downstream URL adjacent to each other in the second sequence;

updating the per-user tree to include paths corresponding to the extracted second sequence of URLs;

analyzing, by a processor of a computer system, the second sequence of URLs to generate a statistical feature of URLs based at least on the upstream URL and the downstream URL, the statistical feature being stored in a statistical feature vector; and

classifying, based on the statistical feature of URLs, the one or more HTTP flows as comprising at least one malicious HTTP redirection triggered by visiting the website, wherein classifying includes updating the per-user tree to reflect that the path on the per-user tree corresponding to the at least one malicious HTTP redirection is malicious

wherein a security operation is initiated with respect to the detected at least one malicious HTTP redirections comprising;

to isolate, deactivate, or neutralize the at least one malicious HTTP redirection.

2. The method of claim 1, further comprising:

obtaining a training HTTP flow set corresponding to training visits to websites;

extracting, from the training HTTP flow set and using a first pre-determined algorithm, a plurality of training sequences of URLs;

selecting, from the plurality of training sequences of URLs, a first portion identified as malicious sequences and a second portion identified as benign sequences;

analyzing (231), using a second pre-determined algorithm, the first portion and the second portion to generate a first plurality of statistical features associated with the malicious sequences and a second plurality of statistical features associated with the benign sequences, respectively; and

determining (233), using a supervised machine learning algorithm, at least one parameter of a classifier based on the first plurality of statistical features and the second plurality of statistical features, wherein the first sequence is classified by using the classifier to analyze the statistical feature of the second sequence.

3. The method of claim 1, wherein any consecutive HTTP flows, if present in the one or more HTTP flows, are separated by less than a pre-determined silence period, and wherein all of the one or more HTTP flows are separated from any other HTTP flow having the single client address by at least the pre-determined silence period.

4. The method of claim 1, wherein the classifying comprises:

determining that the second sequence is malicious based on the statistical feature; and

identifying, in response to the determining, the at least one malicious HTTP redirection based on the second sequence.

5. The method of claim 1, further comprising:

generating, in response to at least the classifying, an alert of a malicious HTTP redirection attack.

6. The method of claim 5, further comprising:

analyzing the first sequence and the second sequence to identify the at least one malicious HTTP redirection, and including, in the alert, information regarding the malicious HTTP redirection.

7. The method of claim 1, wherein the statistical feature of URLs comprises at least one selected from a group consisting of a first tally of different domains in the second sequence of URLs, a second tally of HTTP redirections in the second sequence of URLs, a third tally of different domain HTTP redirections in the second sequence of URLs, a fourth tally of consecutive HTTP redirections in the second sequence of URLs, a fifth tally of consecutive different domain HTTP redirections in the second sequence of URLs, a sixth tally of consecutive short inter-URL durations in the second sequence of URLs, a length of the second sequence of URLs, and a statistical parameter of inter-URL duration distribution of the second sequence of URLs.

8. A system for detecting malicious HTTP redirections in a network, comprising:

a computer processor coupled to memory with;

a flow parser configured to obtain, from the network and based on a single client IP address, one or more HTTP flows triggered by a client device visiting a website, wherein the one or more HTTP flows comprises a first sequence of universal resource locators (URLs);

a per-user tree constructor executing on the computer processor and configured to construct a per-user tree using the one more HTTP flows, the per-user tree including nodes corresponding to URLs, including the first sequence of URLs, wherein the per-user tree includes an edge from a parent node to a child node if a request for a URL corresponding to the child node is triggered from the URL corresponding to the parent node, the per-user tree constructor also configured to update the per-user tree based upon extracted sequences of URLs, wherein each edge of the per-user tree corresponds to;

     1) a URL type assigned to the URL corresponding to the child node and

     2) a time that elapses between HTTP requests in the parent node and child node, wherein the per-user tree includes multiple paths, the multiple paths corresponding to both benign requests and malicious paths;

an universal resource locator (URL) sequence extractor executing on the computer processor and configured to extract, from the first sequence, a second sequence of URLs comprising an upstream URL and a downstream URL adjacent to each other in the second sequence;

a feature extractor executing on the computer processor and configured to analyze the second sequence of URLs to generate a statistical feature of URLs based at least on the upstream URL and the downstream URL, the statistical feature being stored in a statistical feature vector;

a classifier executing on the computer processor and configured to classify, based on the statistical feature of URLs, the one or more HTTP flows as comprising at least one malicious HTTP redirection triggered by visiting the website, wherein classifying includes updating the per-user tree to reflect that the path on the per-user tree corresponding to the at least one malicious HTTP redirection is malicious; and

a repository configured to store the first sequence of URLs, the statistical feature vector, and the second sequence of URLs

wherein a security operation is initiated with respect to the detected at least one malicious HTTP redirection comprising;

to isolate, deactivate, or neutralize the at least one malicious HTTP redirection.

9. The system of claim 8, wherein the flow parser is further configured to obtain a training HTTP flow set corresponding to training visits to websites;

wherein the URL sequence extractor is further configured to extract, from the training HTTP flow set and using a first pre-determined algorithm, a plurality of training sequences of URLs;

wherein the plurality of training sequences of URLs comprises a first portion identified as malicious sequences and a second portion identified as benign sequences;

wherein the feature extractor is further configured to analyze, using a second pre-determined algorithm, the first portion and the second portion to generate a first plurality of statistical features associated with the malicious sequences and a second plurality of statistical features associated with the benign sequences, respectively; and

wherein the system further comprises a supervised machine learning module executing on the computer processor and configured to determine, using a supervised machine learning algorithm, at least one parameter of the classifier based on the first plurality of statistical features and the second plurality of statistical features, and wherein the first sequence is classified by using the classifier in response to the determining.

10. The system of claim 8, wherein any consecutive HTTP flows, if present in the one or more HTTP flows, are separated by less than a pre-determined silence period, and wherein all of the one or more HTTP flows are separated from any other HTTP flow having the single client address by at least the pre-determined silence period.

11. The system of claim 8, wherein the classifying comprises:

determining that the second sequence is malicious based on the statistical feature; and

identifying, in response to the determining, the at least one malicious HTTP redirection based on the second sequence.

12. The system of claim 8, the classifier further configured to:

generate, in response to at least the classifying, an alert of a malicious HTTP redirection attack.

13. The system of claim 11, the classifier further configured to:

analyze the first sequence and the second sequence to identify the at least one malicious HTTP redirection, and include, in the alert, information regarding the malicious HTTP redirection.

14. A non-transitory computer readable medium embodying instructions for detecting malicious HTTP redirections in a network, the instructions when executed by a processor comprising functionality for:

obtaining, from the network and based on a single client IP address, one or more HTTP flows triggered by a client device visiting a website, wherein the one or more HTTP flows comprises a first sequence of universal resource locators (URLs);

constructing a per-user tree using the one more HTTP flows, the per-user tree including nodes corresponding to URLs, including the first sequence of URLs, wherein the per-user tree includes an edge from a parent node to a child node if a request for a URL corresponding to the child node is triggered from the URL corresponding to the parent node, wherein each edge of the per-user tree corresponds to;

     1) a URL type assigned to the URL corresponding to the child node and

     2) a time that elapses between HTTP requests in the parent node and child node, wherein the per-user tree includes multiple paths, the multiple paths corresponding to both benign requests and malicious paths;

extracting, from the first sequence, a second sequence of URLs comprising an upstream URL and a downstream URL adjacent to each other in the second sequence;

updating the per-user tree to include paths corresponding to the extracted second sequence of URLs;

analyzing, by a processor of a computer system, the second sequence of URLs to generate a statistical feature of URLs based at least on the upstream URL and the downstream URL, the statistical feature being stored in a statistical feature vector; and

classifying, based on the statistical feature of URLs, the one or more HTTP flows as comprising at least one malicious HTTP redirection triggered by visiting the website, wherein classifying includes updating the per-user tree to reflect that the path on the per-user tree corresponding to the at least one malicious HTTP redirection is malicious wherein a security operation is initiated with respect to the detected at least one malicious HTTP redirection comprising;

to isolate, deactivate, or neutralize the at least one malicious HTTP redirection.

15. The non-transitory computer readable medium of claim 14, further comprising:

obtaining a training HTTP flow set corresponding to training visits to websites;

extracting, from the training HTTP flow set and using a first pre-determined algorithm, a plurality of training sequences of URLs;

selecting, from the plurality of training sequences of URLs, a first portion identified as malicious sequences and a second portion identified as benign sequences;

analyzing, using a second pre-determined algorithm, the first portion and the second portion to generate a first plurality of statistical features associated with the malicious sequences and a second plurality of statistical features associated with the benign sequences, respectively; and

determining, using a supervised machine learning algorithm, at least one parameter of a classifier based on the first plurality of statistical features and the second plurality of statistical features, wherein the first sequence is classified by using the classifier to analyze the statistical feature of the second sequence.

16. The non-transitory computer readable medium of claim 14, wherein any consecutive HTTP flows, if present in the one or more HTTP flows, are separated by less than a pre-determined silence period, and wherein all of the one or more HTTP flows are separated from any other HTTP flow having the single client address by at least the pre-determined silence period.

17. The non-transitory computer readable medium of claim 14, wherein the classifying comprises:

determining that the second sequence is malicious based on the statistical feature; and

identifying, in response to the determining, the at least one malicious HTTP redirection based on the second sequence.

18. The non-transitory computer readable medium of claim 14, further comprising:

generating, in response to at least the classifying, an alert of a malicious HTTP redirection attack.

19. The non-transitory computer readable medium of claim 18, further comprising:

analyzing the first sequence and the second sequence to identify the at least one malicious HTTP redirection, and including, in the alert, information regarding the malicious HTTP redirection.

20. The non-transitory computer readable medium of claim 14, wherein the statistical feature of URLs comprises at least one selected from a group consisting of a first tally of different domains in the second sequence of URLs, a second tally of HTTP redirections in the second sequence of URLs, a third tally of different domain HTTP redirections in the second sequence of URLs, a fourth tally of consecutive HTTP redirections in the second sequence of URLs, a fifth tally of consecutive different domain HTTP redirections in the second sequence of URLs, a sixth tally of consecutive short inter-URL durations in the second sequence of URLs, a length of the second sequence of URLs, and a statistical parameter of inter-URL duration distribution of the second sequence of URLs.