Using content analysis to detect spam web pages

• US 20060184500A1
• Filed: 02/11/2005
• Published: 08/17/2006
• Est. Priority Date: 02/11/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Najork, Marc Alexander, Ntoulas, Alexandros, Fetterly, Dennis Craig, Manasse, Mark Steven
• Granted Patent: US 7,962,510 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Evaluating content includes receiving content, analyzing the content for web spam using a content-based identification technique, and classifying the content according to the analysis. An index of analyzed contents may be created. A system for evaluating content includes a storage device configured to store data and a processor configured to analyze content using content-based identification techniques to determine whether web spam is present.

Claims

1. A method for identifying web spam, comprising:

receiving content;

analyzing the content for web spam using a content-based identification technique; and

classifying the content according to said analysis.

2. The method of claim 1, wherein the content-based identification technique comprises using at least one metric comprising a threshold.

3. The method of claim 2, wherein said analysis comprises comparing the content to the threshold to determine if the content is web spam.

4. The method of claim 3, wherein the threshold is predetermined.

5. The method of claim 3, wherein the threshold is determined based on analyzed results.

6. The method of claim 1, wherein the content comprises a web page.

7. The method of claim 1, further comprising creating an index based on the analyzed and classified content.

8. The method of claim 7, further comprising comparing the classified content against a result set computed for a received query.

9. The method of claim 8, further comprising blocking a result based on the classified content.

10. The method of claim 8, further comprising adjusting a ranking of a result based on the classified content.

11. The method of claim 1, further comprising crawling a set of web pages to obtain the content.

12. A system for identifying web spam, comprising:

a storage device configured to store data; and

a processor configured to analyze content using a content-based identification technique to determine whether web spam is present.

13. The system of claim 12, wherein the content-based identification technique comprises at least one metric.

14. The system of claim 13, wherein said processor compares the content to a threshold to determine if the content is web spam.

15. The system of claim 14, wherein the threshold is predetermined or determined based on analyzed results.

16. The system of claim 12, wherein an index of contents is stored in said storage device.

17. The system of claim 12, wherein the content comprises a web page.

18. The system of claim 12, wherein the processor classifies the content and indexes the content for future queries.

19. A computer program for identifying web spam, the computer program product being embodied in a computer readable medium and comprising computer instructions for:

receiving content;

analyzing the content for web spam using a content-based identification technique; and

classifying the content according to said analysis.

20. The computer program of claim 19, further comprising computer instructions for creating an index for analyzing a future query.