SEARCH SPAM ANALYSIS AND DETECTION

• US 20110087648A1
• Filed: 12/13/2010
• Published: 04/14/2011
• Est. Priority Date: 05/31/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Yi-Min, Ma, Ming
• Granted Patent: US 8,972,401 B2
• Time in Patent Office: Days
• US Class Current: 707/709
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: G06F 2221/2119 Authenticating web pages, e.g. with suspicious links
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 63/0227 Filtering policies mail message filtering H04L51/212
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection

Abstract

Defeating click-through cloaking includes retrieving a search results page to set a browser variable, inserting a link to a page into the search results page and clicking through to the page using the inserted link. Investigating cloaking includes providing script associated with a suspected spam URL, modifying the script to de-obfuscate the script and executing the modified script to reveal cloaking logic associated with the script.

Claims

1. A method comprising:

under control of at least one processor configured with executable instructions,

retrieving a search results page to set a browser variable;

inserting a link to a suspect uniform resource locator (URL) into the search results page; and

clicking through to the suspect URL using the inserted link to determine that a host of the suspect URL serves spam content to users that click through to the suspect URL from search results.

2. The method according to claim 1, wherein the browser variable indicates, to the host of the suspect URL, a referring page that links to the suspect URL.

3. The method according to claim 2, wherein the browser variable indicates, to the host of the suspect URL, that the referring page is the search results page and that a specific search engine is a source of the referring page.

4. The method according to claim 2, wherein the browser variable comprises a document.referrer variable set in a web browser as a result of using the web browser to retrieve the search results page from a search engine.

5. The method according to claim 1, further comprising performing the inserting and clicking through when the host of the suspect URL uses click-through cloaking in which a non-spam page is served to a web crawler that accesses the suspect URL during crawling and a spam page is served to a web browser that clicks through to the suspect URL from search results.

6. The method according to claim 1, wherein clicking through to determine that the host of the suspect URL serves spam content to users that click through to the suspect URL from search results comprises:

accessing the suspect URL with a web crawler to obtain a first page;

comparing the first page with a second page returned by clicking through to the suspect URL using the link inserted into the search results page; and

identifying the suspect URL for potential blacklisting when content of the first page is different from content of the second page.

7. The method according to claim 1, further comprising:

intercepting redirection to one or more redirection URLs; and

recording the one or more redirection URLs.

8. The method according to claim 7, further comprising:

receiving a group of the redirection URLs;

analyzing similarities between the redirection URLs in the group to identify related URLs; and

identifying, based on the analyzing, one or more doorway URLs that share a redirection domain.

9. The method according to claim 8, further comprising:

verifying that an identified redirection domain is responsible for serving spam content; and

performing backward propagation to detect other related spam URLs associated with the redirection domain.

10. The method according to claim 1, further comprising identifying the suspect URL as a suspect by performing operations including:

obtaining script associated with the suspect URL;

modifying the script to de-obfuscate the script; and

executing the modified script to determine whether the script checks the browser variable against a name of one or more search engines.

11. A method the method comprising:

under control of at least one processor configured with executable instructions,

accessing a uniform resource locator (URL) with a browser;

obtaining script associated with the URL;

modifying the script to de-obfuscate the script;

executing the modified script to determine that the script checks a browser variable against a name of one or more search engines; and

identifying the URL as a suspect URL.

12. The method according to claim 11 further comprising executing the modified script to determine whether the script checks that a domain name corresponding to the suspect URL is included in a referrer string.

13. The method according to claim 11, wherein the browser variable comprises a document.referrer variable that indicates, to a host of the suspect URL, that a specific search engine is a source of a referring page that links to the suspect URL.

14. The method according to claim 11, wherein a host of the suspect URL uses click-through cloaking in which a non-spam page is served to a web crawler that accesses the suspect URL during crawling and a spam page is served to the browser that clicks through to the suspect URL from search results.

15. The method according to claim 11, further comprising determining whether a host of the suspect URL serves spam content by:

accessing the suspect URL with a web crawler to obtain a first page;

comparing the first page with a second page returned by clicking through to the suspect URL using a link inserted into a search results page; and

identifying the suspect URL for potential blacklisting when content of the first page is different from content of the second page.

16. The method according to claim 11, further comprising:

intercepting redirection to one or more redirection URLs; and

recording the one or more redirection URLs.

17. The method according to claim 16, further comprising:

receiving a group of the redirection URLs;

analyzing similarities between the redirection URLs in the group to identify related URLs; and

identifying, based on the analyzing, one or more doorway URLs that share a redirection domain.

18. The method according to claim 17, further comprising:

verifying that an identified redirection domain is responsible for serving spam content; and

performing backward propagation to detect other related spam URLs associated with the redirection domain.

19. A computing device comprising:

one or more processors coupled to computer storage media; and

a spam detection component stored on the computer storage media and executed on the one or more processors to identify sources of spam web pages, the spam detection component configured to perform operations including;

deriving, based on a name of a suspect uniform resource locator (URL), one or more keywords;

submitting the one or more keywords to a search engine;

receiving, by a browser, as a result of submitting the one or more keywords to the search engine, a search results page that sets a browser variable in the browser;

inserting a link to the suspect URL into the search results page; and

clicking through to the suspect URL using the inserted link to determine whether a host of the suspect URL serves spam web pages.

20. The computing device according to claim 19, wherein determining whether the host of the suspect URL serves spam web pages comprises:

accessing the suspect URL without clicking through to the suspect URL from a search results page to obtain a first web page;

comparing the first web page with a second page returned by clicking through to the suspect URL using the link inserted into the search results page; and

identifying the suspect URL for potential blacklisting when content of the first page is different from content of the second page.