Method and system for detecting malicious web addresses

• US 10,567,407 B2
• Filed: 09/25/2015
• Issued: 02/18/2020
• Est. Priority Date: 04/30/2015
• Status: Active Grant

• Current Assignee: YunTian Co., Ltd.
• Original Assignee: Iyuntian Co., Ltd.
• Inventors: Tang, Chengguang, Yang, Nian, Geng, Zhifeng
• Primary Examiner(s): Lwin, Maung T
• Assistant Examiner(s): Debnath, Suman
• Application Number:
  US15/504,276
• Publication Number:
  US 20180041530A1
• Time in Patent Office: 1,607 Days
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06F 2221/2103 Challenge-response
• CPC Class Codes: G06F 2221/2119 Authenticating web pages, e.g. with suspicious links
• CPC Class Codes: H04L 63/107 wherein the security policies are location-dependent, e.g. entities privileges depend on current location or allowing specific operations only from locally connected terminals
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis
• CPC Class Codes: H04L 63/1483 service impersonation, e.g. phishing, pharming or web spoofing detection of rogue wireless access points H04W12/12
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal

Abstract

The present application provides a method and system for detecting malicious web addresses. The method includes: receiving a uniform resource locator (URL) reported by a user; acquiring a HyperText Transfer Protocol (HTTP) request chain associated with the URL, wherein the HTTP request chain is a sequential linked list including information about multiple HTTP request-response interactions during an access to the URL; and analyzing the HTTP request chain to determine whether the URL is a malicious web address. The technical solution of the present application can provide an accurate result of malicious web address detection, can detect various newly emerging malicious web addresses, and are user-friendly. The user only needs to upload the URL and does not need to provide any other information.

Claims

1. A method for detecting a malicious web address, comprising:

receiving a uniform resource locator (URL) reported by a terminal device of a user;

acquiring a HyperText Transfer Protocol (HTTP) request chain associated with the URL, the HTTP request chain being a sequential linked list comprising information about multiple HTTP request-response interactions during an access to the URL; and

analyzing the HTTP request chain to determine whether the URL is a malicious web address,

wherein the acquiring an HTTP request chain comprises acquiring the HTTP request chain by using an active crawler server nearest to the user in a plurality of active crawler servers distributed in different geographical locations and configured to acquire the HTTP request chain associated with the URL, wherein a geographical location of the used active crawler server nearest to the user is a nearest geographical location to a geographical location of the terminal device of the user among the geographical locations of the plurality of active crawler servers,

wherein the acquiring the HTTP request chain further comprises;

determining a geographical location and network environment information of the user;

wherein the determining the geographical location and network environment information of the user comprises;

determining the geographical location and network operator information of the user based on an Internet Protocol (IP) address of the URL reported by the user; and

determining the network environment information of the user based on the network operator information.

2. The method according to claim 1, wherein the acquiring the HTTP request chain comprises:

determining the geographical location and network environment information of the user;

scheduling the URL to an active crawler server having a network bandwidth closest to the user in the plurality of active crawler servers; and

downloading from the active crawler server, a web page content associated with the URL to obtain the HTTP request chain.

3. The method according to claim 2, wherein the downloading a web page content associated with the URL to obtain the HTTP request chain comprises:

grabbing a web page content after jumping, and saving an intermediate result.

4. The method according to claim 3, wherein the grabbing a web page content after jumping comprises at least one of the following:

rendering a Hypertext Markup Language Document Object Model (HTML DOM) tree by using a layout engine of a browser, to grab a web page content that jumps using an inline frame (iframe) tag in an HTML document;

executing a JavaScript code by using a JavaScript engine, to grab a web page content that jumps using the JavaScript code; and

executing Flash by using a Flash player plugin, to grab a web page content that jumps using Flash.

5. The method according to claim 1, wherein the analyzing the HTTP request chain to determine whether the URL is a malicious web address comprises:

extracting a characteristic in one of following dimensions from the HTTP request chain;

upstream and downstream information, a server dimension, a web page programming language dimension, a time dimension, and own descriptive information of a web page; and

determining whether the URL is a normal web address or a suspicious malicious web address based on the extracted characteristic and by using a machine learning-based and built classification model.

6. The method according to claim 5, wherein

the upstream and downstream information comprises at least one of following information:

a number of 302 jumps, a percentage of 404 pages, whether a child URL comprises an advertising alliance link, whether a child URL comprises a malicious sub-link, and whether a child URL comprises a small website statistics tool;

the server dimension comprises at least one of following information;

whether a server has a foreign Internet Protocol (IP) address, whether a server is Windows IIS, whether a content distribution network (CDN) technology is used, whether a server is a kangle server, whether a server is a netbox server, whether a server is a nginx server, whether a server is an apache server, and whether multimedia video is used;

the web page programming language dimension comprises at least one of following information;

whether a web page is compiled using the Active Server Page (ASP) language, and whether a web page is compiled using the Hypertext Preprocessor (PHP) language;

the time dimension comprises at least one of following information;

whether time is hot time, and whether time is weekend; and

the web page description information comprises at least one of following information;

a web page size, time for loading a single URL, whether a website is on record, whether a web page is encrypted, and whether a web page has a free subdomain name.

7. The method according to claim 5, further comprising:

rendering a web page content associated with the URL into a picture and extracting a web page text content by using an optical character recognition (OCR) technology, in response to determining the URL being a suspicious malicious web address;

making a topic judgment on the web page text content by using a latent semantic model; and

determining whether the URL is a malicious web address based on the result of the topic judgment.

8. The method according to claim 7, further comprising:

performing false alarm removal processing on a result of the topic judgment.

9. The method according to claim 8, wherein the false alarm removal processing comprises at least one of:

determining, according to a white list, whether the determination of the URL as a malicious web address is a false alarm;

determining, by querying access information related to the URL, whether the determination of the URL as a malicious web address is a false alarm;

determining, by Internet content provider (ICP) filing information of the URL, whether the determination of the URL as a malicious web address is a false alarm; and

determining, by querying qualification data related to the URL, whether the determination of the URL as a malicious web address is a false alarm.

10. A system for detecting a malicious web address, comprising a crawler subsystem and a detection subsystem,

the crawler subsystem comprising a crawler scheduling server and a plurality of active crawler servers distributed in different geographical locations, the crawler scheduling server being configured to receive a uniform resource locator (URL) reported by a terminal device of a user, and schedule the one or more active crawler servers;

and the active crawler servers being configured to acquire, as scheduled by the crawler scheduling server, a HyperText Transfer Protocol (HTTP) request chain associated with the URL, the HTTP request chain being a sequential linked list comprising information about multiple HTTP request-response interactions during an access to the URL; and

the detection subsystem comprises an analysis unit configured to analyze the HTTP request chain to determine whether the URL is a malicious web address,

wherein the crawler scheduling server is configured to schedule the reported URL to the active crawler server nearest to the user in the plurality of active crawler servers, wherein a geographical location of the active crawler server nearest to the user is a nearest geographical location to a geographical location of the terminal device of the user among the geographical locations of the plurality of active crawler servers,

wherein the acquiring the HTTP request chain further comprises;

determining a geographical location and network environment information of the user;

wherein the determining the geographical location and network environment information of the user comprises;

determining the geographical location and network operator information of the user based on an Internet Protocol (IP) address of the URL reported by the user; and

determining the network environment information of the user based on the network operator information.

11. The system according to claim 10, wherein the acquiring, by the active crawler server, an HTTP request chain comprises:

downloading, by the active crawler server, a web page content associated with the URL to obtain the HTTP request chain.

12. The system according to claim 11, wherein the active crawler server is configured to grab a web page content after jumping and save an intermediate result by performing at least one of:

rendering a Hypertext Markup Language Document Object Model (HTML DOM) tree by using a layout engine of a browser, to grab a web page content that jumps by using an inline frame (iframe) tag in an HTML document;

executing a JavaScript code by using a JavaScript engine, to grab a web page content that jumps by using the JavaScript code; and

executing Flash by using a Flash player plugin, to grab a web page content that jumps using Flash.

13. The system according to claim 10, wherein the analysis unit comprises:

a characteristic extraction subunit, configured to extract a characteristic in one of following dimensions from the HTTP request chain;

upstream and downstream information, a server dimension, a web page programming language dimension, a time dimension, and own descriptive information of a web page; and

a classification subunit, configured to determine whether the URL is a normal web address or a suspicious malicious web address based on the extracted characteristic and by using a machine learning-based and built classification model.

14. The system according to claim 13, wherein

the upstream and downstream information comprises at least one of following information:

a number of 302 jumps, a percentage of 404 pages, whether a child URL comprises an advertising alliance link, whether a child URL comprises a malicious sub-link, and whether a child URL comprises a small website statistics tool;

the server dimension comprises at least one of following information;

whether a server has a foreign Internet Protocol (IP) address, whether a server is Windows IIS, whether a content distribution network (CDN) technology is used, whether a server is a kangle server, whether a server is a netbox server, whether a server is a nginx server, whether a server is an apache server, and whether multimedia video is used;

the web page programming language dimension comprises at least one of following information;

whether a web page is compiled using the Active Server Page ASP language, and whether a web page is compiled using the Hypertext Preprocessor (PHP) language;

the time dimension comprises at least one of the following information;

whether the time is a hot time, and whether the time is weekend; and

the web page description information comprises at least one of the following information;

a web page size, a time for loading a single URL, whether a website has been put on records, whether a web page has been encrypted, and whether a web page has a free subdomain name.

15. The system according to claim 13, wherein the detection subsystem further comprises:

an image recognition unit, configured to;

for the URL determined by the classification subunit as a suspicious malicious web address, extract a web page text content from the web page content associated with the URL and rendered into a picture, by using an optical character recognition (OCR) technology; and

a semantic parsing unit, configured to make a topic judgment on the web page text content by using a latent semantic model, to determine whether the URL is a malicious web address.

16. The system according to claim 15, wherein the detection subsystem further comprises:

a false alarm removal unit, configured to perform false alarm removal processing on a result of the topic judgment.

17. A device, comprising:

one or more processors;

a memory; and

one or more programs stored in the memory, the one or more programs being used by the one or more processors to;

receive a uniform resource locator (URL) reported by a terminal device of a user;

acquire a HyperText Transfer Protocol (HTTP) request chain associated with the URL, wherein the HTTP request chain is a sequential linked list comprising information about multiple HTTP request-response interactions during an access to the URL; and

analyze the HTTP request chain to determine whether the URL is a malicious web address,

wherein the acquiring an HTTP request chain comprises acquiring the HTTP request chain by using an active crawler server nearest to the user in a plurality of active crawler servers distributed in different geographical locations and configured to acquire the HTTP request chain associated with the URL, wherein a geographical location of the used active crawler server nearest to the user is a nearest geographical location to a geographical location of the terminal device of the user among the geographical locations of the plurality of active crawler servers,

wherein the acquiring the HTTP request chain further comprises;

determining a geographical location and network environment information of the user;

wherein the determining the geographical location and network environment information of the user comprises;

determining the geographical location and network operator information of the user based on an Internet Protocol (IP) address of the URL reported by the user; and

determining the network environment information of the user based on the network operator information.

18. A non-volatile computer readable storage medium, storing one or more programs, the one or more programs, when executed by a device, causing the device to:

receive a uniform resource locator (URL) reported by a terminal device of a user;

acquire a HyperText Transfer Protocol (HTTP) request chain associated with the URL, the HTTP request chain being a sequential linked list comprising information about multiple HTTP request-response interactions during an access to the URL; and

analyze the HTTP request chain to determine whether the URL is a malicious web address,

wherein the acquiring an HTTP request chain comprises acquiring the HTTP request chain by using an active crawler server nearest to the user in a plurality of active crawler servers distributed in different geographical locations and configured to acquire the HTTP request chain associated with the URL, wherein a geographical location of the used active crawler server nearest to the user is a nearest geographical location to a geographical location of the terminal device of the user among the geographical locations of the plurality of active crawler servers,

wherein the acquiring the HTTP request chain further comprises;

determining a geographical location and network environment information of the user;

wherein the determining the geographical location and network environment information of the user comprises;

determining the geographical location and network operator information of the user based on an Internet Protocol (IP) address of the URL reported by the user; and

determining the network environment information of the user based on the network operator information.