Using content analysis to detect spam web pages

• US 7,962,510 B2
• Filed: 02/11/2005
• Issued: 06/14/2011
• Est. Priority Date: 02/11/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Najork, Marc Alexander, Ntoulas, Alexandros, Fetterly, Dennis Craig, Manasse, Mark Steven
• Primary Examiner(s): Breene; John
• Assistant Examiner(s): Bullock; Joshua
• Application Number:
  US11/056,788
• Publication Number:
  US 20060184500A1
• Time in Patent Office: 2,314 Days
• Field of Search: 707/102, 707/2, 707/6, 707/709, 707/711, 707/771, 707/708, 709/206
• US Class Current: 707/771
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Evaluating content includes receiving content, analyzing the content for web spam using a content-based identification technique, and classifying the content according to the analysis. An index of analyzed contents may be created. A system for evaluating content includes a storage device configured to store data and a processor configured to analyze content for web spam using content-based identification techniques.

Claims

1. A method comprising:

receiving content by crawling a web page;

analyzing the content for web spam using a content-based identification technique,

wherein the content-based identification technique comprises at least one of;

determining a fraction of visible content to total content on the web page;

or

determining a ratio of compressed visible content to uncompressed visible content on the web page; and

classifying the content according to said analysis.

2. The method of claim 1, further comprising:

creating an entry of the web page in an index associated with the classified content, wherein the entry comprises an identifier of the web page and an indication of whether the web page is web spam; and

using the index to filter web spam from a result set computed for a received query.

3. The method of claim 2, wherein using the index comprises comparing the index against the result set computed for the received query.

4. The method of claim 3, further comprising blocking a result from the result set based on the index associated with the analyzed and classified content.

5. The method of claim 3, further comprising adjusting a ranking of a result from the result set based on the index associated with the analyzed and classified content.

6. The method of claim 1, further comprising crawling a set of web pages to obtain the content, wherein the content comprises an item from each of the web pages in the set of web pages being crawled.

15. The method of claim 1, wherein analyzing the content for web spam using a content-based identification technique comprises scanning the content of the web page for a plurality of additional metrics associated with the content, wherein each of the additional metrics is correlated with a probability that the web page is web spam.

17. The method of claim 1, wherein the content-based identification technique further comprises determining an average word length from amongst a plurality of words contained in the web page, and determining if the average word length falls outside an expected range of values.

18. The method of claim 17, wherein the expected range of values comprises a low end average word length value and a high end average word length value, the low end average word length value defined by a first number of characters, and the high end average word length value defined by a second number of characters, the second number of characters being larger than the first number of characters.

7. A system for identifying web spam, the system comprising:

a storage device configured to store an index; and

a processor configured to;

receive content from a crawled web page;

analyze the content using a content-based identification technique to determine whether web spam is present, wherein the content-based identification technique comprises at least one of;

determining a fraction of visible content to total content on the web page;

or

determining a ratio of compressed visible content to uncompressed visible content on the web page; and

classify the content according to said analysis.

8. The system of claim 7, wherein the content-based identification technique further comprises using an additional metric.

9. The system of claim 8, wherein said processor further:

compares the classified content to a threshold to determine if the content is web spam; and

creates an entry of the web page in the index, the entry comprising an identifier of the web page and an indication whether the web page is web spam.

10. The system of claim 9, wherein the threshold is predetermined or determined based on analyzed results.

11. The system of claim 7, wherein the index stored in said storage device is used to filter web spam from a result set computed for a received query.

12. The system of claim 11, wherein the processor further:

compares the index against the result set computed for the received query; and

blocks a result from the result set based on the index.

16. The system of claim 7, wherein, during the analysis of the content for web spam, the processor scans the content of the crawled web page for a plurality of additional metrics associated with the content, wherein each of the additional metrics is correlated with a probability that the web page is web spam.

19. The system of claim 7, wherein the content-based identification technique further comprises determining an average word length from amongst a plurality of words contained in the web page, and determining if the average word length falls outside an expected range of values.

13. A computer-readable storage medium comprising computer-executable instructions stored thereon, wherein the computer-executable instructions, when executed by a computer, cause performance of acts, the acts comprising:

receiving a set of web pages in response to a query;

analyzing content of the set of web pages for web spam by using a content-based identification technique comprising at least one of;

determining a fraction of visible content to total content on the web page;

or

determining a ratio of compressed visible content to uncompressed visible content on the web page; and

classifying the content according to said analysis.

14. The computer-readable storage medium of claim 13, further comprising computer-executable instructions for creating an index for analyzing a future query.

20. The computer-readable storage medium of claim 13, wherein the content-based identification technique further comprises determining an average word length from amongst a plurality of words contained in the web page, and determining if the average word length falls outside an expected range of values.