Using hash signatures of DOM objects to identify website similarity

• US 9,386,037 B1
• Filed: 11/11/2015
• Issued: 07/05/2016
• Est. Priority Date: 09/16/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: RiskIQ, Inc. (Microsoft Corporation)
• Inventors: Hunt, Adam, Pon, David, Edgeworth, Jonas, Linn, Joseph, Kiernan, Chris, Adams, Ben, Manousos, Elias
• Primary Examiner(s): Lemma, Samson
• Application Number:
  US14/938,814
• Time in Patent Office: 237 Days
• Field of Search: 713/176, 713/181
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06F 21/128 involving web programs, i.e. using technology especially used in internet, generally interacting with a web browser, e.g. hypertext markup language [HTML], applets, java
• CPC Class Codes: G06F 2221/2119 Authenticating web pages, e.g. with suspicious links
• CPC Class Codes: H04L 63/0876 based on the identity of the terminal or configuration, e.g. MAC address, hardware or software configuration or device fingerprint
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis
• CPC Class Codes: H04L 63/1483 service impersonation, e.g. phishing, pharming or web spoofing detection of rogue wireless access points H04W12/12
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 9/3247 involving digital signatures

Abstract

Embodiments are directed to using a hash signature of a rendered DOM object of a website to find similar content and behavior on other websites. Embodiments break a DOM into a large number of data portions (i.e., “shingles”), apply a hashing algorithm to the shingles, select a predetermined number of hashes from the hashed shingles according to a selection criteria to create a hash signature, and compare the hash signature to that of a reference page to determine similarity of website DOM object content. Embodiments can be used to identify phishing websites, defaced websites, spam websites, significant changes in the content of a webpage, copyright infringement, and any other suitable purposes related to the similarity between website DOM object content.

Claims

1. A method for determining a similarity between two websites, the method comprising, at a computer system:

receiving website information from a web server corresponding to a website;

rendering a document object model (DOM) object of the website using the website information;

separating content within the DOM object into a plurality of data portions, each of the plurality of data portions having a fixed length;

generating, by a hardware processor of the computer system, a hash signature of the DOM object by;

applying a predetermined number of hashing functions to each of the plurality of data portions, wherein the predetermined number of hashing functions are generated using a common seed value, and wherein applying the predetermined number of hashing functions results in a predetermined number of values for each of the plurality of data portions; and

selecting, using a selection policy, a predetermined number of hashed data portions of the plurality of hashed data portions, wherein the predetermined number of hashed data portions are selected to create a hash signature of the DOM object;

comparing the hash signature of the DOM object to a known hash signature of a DOM object associated with a known website having a first classification, wherein comparing the hash signature of the DOM object to the known hash signature of the DOM object associated with the known website includes comparing each of the plurality of hashed data portions to a plurality of known hashed data portions of the known hash signature;

calculating a similarity measurement between the hash signature of the DOM object and the known hash signature of the DOM object associated with the known website;

comparing the similarity measurement to a threshold; and

determining that the website has the first classification based on the similarity measurement exceeding the threshold.

2. The method of claim 1, wherein each of the predetermined number of hashing functions is generated using at least one input variable.

3. The method of claim 1, wherein the selection policy includes one of a minimum value of the plurality of hashed data portions, a maximum value of the plurality of hashed data portions, or a median value of the plurality of hashed data portions.

4. The method of claim 1, wherein the fixed length is determined based on a number of characters in the DOM object.

5. The method of claim 1, wherein selecting the predetermined number of hashed data portions using the selection policy to create the hash signature of the DOM object further comprises:

applying the selection policy to each of the predetermined number of values for each of the plurality of data portions to select a single value for each of the plurality of data portions.

6. The method of claim 1, wherein the similarity measurement includes a magnitude of hashed data portions that are shared between the hash signature of the DOM object and the known hash signature of the DOM object associated with the known website divided by a magnitude of hashed data portions including the plurality of hashed data portions and the plurality of known hashed data portions of the known hash signature.

7. The method of claim 6, wherein the magnitude of the hashed data portions is equal to the predetermined number of hashing functions.

8. The method of claim 1, wherein comparing the hash signature of the DOM object to the known hash signature of the DOM object associated with the known website further comprises:

determining a classification for comparing a similarity of the website to known websites;

identifying a plurality of known hash signatures, each of the plurality of known hash signatures corresponding to a different one of DOM objects, wherein each of the DOM objects corresponds to a different one of the known websites associated with the classification, and wherein the plurality of known hash signatures is identified by searching a database of hash signatures for each of the plurality of known hash signatures classified as being associated with the classification;

calculating a similarity measurement of the hash signature of the DOM object for the classification;

comparing the similarity measurement of the hash signature of the DOM object to a similarity threshold associated with the classification; and

reporting the website as being associated with the classification based on the determining that the similarity measurement is over the similarity threshold associated with the classification.

9. The method of claim 8, further comprising:

calculating a static hash of the DOM object by applying a static hash function to the DOM object;

comparing the static hash of the DOM object to a known static hash for each of the plurality of known hash signatures to identify an exact match;

determining an exact match is present in the known static hash for each of the plurality of known static hash signatures; and

classifying the website as being a duplicate of the known websites associated with the exact match for each of the plurality of known static hash signatures.

10. The method of claim 8, wherein calculating a similarity measurement of the hash signature of the DOM object for the classification further comprises:

calculating a similarity measurement between the hash signature of the DOM object and each of the plurality of the known hash signatures identified by searching the database, wherein each of the plurality of the known hash signatures are classified as confirmed for the classification or classified as dismissed for the classification;

determining a predetermined number of most similar known hash signatures;

weighting a similarity of the predetermined number of the most similar known hash signatures according to a distribution of similarity measurements of the predetermined number of the most similar known hash signatures; and

calculating a weighted similarity of the DOM object by dividing a sum of similarity weights of a number of the most similar known hash signatures that are classified as confirmed by a sum of similarity weights of a total number of the most similar known hash signatures.

11. The method of claim 8, wherein the classification determined for comparing the similarity of the website to the known websites includes a designation as a phishing website.

12. The method of claim 8, wherein the classification determined for comparing the similarity of the website to the known websites includes being associated with copyrighted content.

13. The method of claim 8, wherein the classification determined for comparing the similarity of the website to the known websites includes being associated with unauthorized content.

14. A computer comprising:

a processor; and

a computer product coupled to the processor, the computer product comprising code, executable by the processor, to provide a computer program configured to perform a method of determining a similarity between two websites, the method comprising;

receiving website information from a web server corresponding to a website;

rendering a document object model (DOM) object of the website using the website information;

separating content within the DOM object into a plurality of data portions, each of the plurality of data portions having a fixed length;

generating a hash signature of the DOM object by;

applying a predetermining number of hashing functions to each of the plurality of data portions, wherein the predetermined number of hashing functions are generated using a common seed value, and wherein applying the predetermined number of hashing functions results in a predetermined number of values for each of the plurality of data portions; and

selecting, using a selection policy, a predetermined number of hashed data portions of the plurality of hashed data portions, wherein the predetermined number of hashed data portions are selected to create a hash signature of the DOM object;

comparing the hash signature of the DOM object to a known hash signature of a DOM object associated with a known website having a first classification, wherein comparing the hash signature of the DOM object to the known hash signature of the DOM object associated with the known website includes comparing each of the plurality of hashed data portions to a plurality of known hashed data portions of the known hash signature;

calculating a similarity measurement between the hash signature of the DOM object and the known hash signature of the DOM object associated with the known website;

comparing the similarity measurement to a threshold; and

determining that the website has the first classification based on the similarity measurement exceeding the threshold.

15. The computer of claim 14, wherein each of the predetermined number of hashing functions is generated using at least one input.

16. The computer of claim 14, wherein selecting the predetermined number of hashed data portions using the selection policy to create the hash signature of the DOM object further comprises:

applying the selection policy to each of the predetermined number of values for each of the plurality of data portions to select a single value for each of the plurality of data portions.

17. The computer of claim 14, wherein the similarity measurement includes a magnitude of hashed data portions that are shared between the hash signature of the DOM object and the known hash signature of the DOM object associated with the known website divided by a magnitude of hashed data portions including the plurality of hashed data portions and the plurality of known hashed data portions of the known hash signature.

18. A system comprising:

a web server computer configured to serve website information associated with a website; and

a similarity analysis computer communicatively coupled to the web server computer through a network connection, the similarity analysis computer configured to;

receive website information from the web server computer corresponding to the website;

render a document object model (DOM) object of the website using the website information;

separate content within the DOM object into a plurality of data portions, each of the plurality of data portions having a fixed length;

generate a hash signature of the DOM object by;

apply a predetermined number of hashing functions to each of the plurality of data portions, wherein the predetermined number of hashing functions are generated using a common seed value, and wherein applying the predetermined number of hashing functions results in a predetermined number of values for each of the plurality of data portions; and

select, using a selection policy, a predetermined number of hashed data portions of the plurality of hashed data portions, wherein the predetermined number of hashed data portions are selected to create a hash signature of the DOM object;

compare the hash signature of the DOM object to a known hash signature of a DOM object associated with a known website having a first classification, wherein comparing the hash signature of the DOM object to the known hash signature of the DOM object associated with the known website includes comparing each of the plurality of hashed data portions to a plurality of known hashed data portions of the known hash signature;

calculate a similarity measurement between the hash signature of the DOM object and the known hash signature of the DOM object associated with the known website;

compare the similarity measurement to a threshold; and

determine that the website has the first classification based on the similarity measurement exceeding the threshold.

19. The system of claim 18, wherein each of the predetermined number of hashing functions is generated using at least one input variable.

20. The system of claim 18, wherein selecting the predetermined number of hashed data portions using the selection policy to create the hash signature of the DOM object further comprises:

applying the selection policy to each of the predetermined number of values for each of the plurality of data portions to select a single value for each of the plurality of data portions.