Content evaluation

• US 20060069667A1
• Filed: 09/30/2004
• Published: 03/30/2006
• Est. Priority Date: 09/30/2004
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Najork, Marc Alexander, Fetterly, Dennis Craig, Manasse, Mark Steven
• Application Number:
  US10/956,228
• Publication Number:
  US 20060069667A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

Evaluating content is described, including generating a data set using an attribute associated with the content, evaluating the data set using a statistical distribution to identify a class of statistical outliers, and analyzing a web page to determine whether it is part of the class of statistical outliers. A system includes a memory configured to store data, and a processor configured to generate a data set using an attribute associated with the content, evaluate the data set using a statistical distribution to identify a class of statistical outliers, and analyze a web page to determine whether it is part of the class of statistical outliers. Another technique includes crawling a set of web pages, evaluating the set of web pages to compute a statistical distribution, flagging an outlier page in the statistical distribution as web spam, and creating an index of the web pages and the outlier page for answering a query.

Claims

1. A method for evaluating content, comprising:

generating a data set using an attribute associated with the content;

evaluating the data set using a statistical distribution to identify a class of statistical outliers; and

analyzing a web page to determine whether it is part of the class of statistical outliers.

2. The method recited in claim 1, wherein the attribute is an address.

3. The method recited in claim 1, wherein the attribute is an address property.

4. The method as recited in claim 1, wherein the attribute is a uniform resource locator property.

5. The method as recited in claim 1, wherein the attribute is a hostname resolution characteristic.

6. The method as recited in claim 5, wherein the hostname resolution characteristic represents a number of names assigned to an address.

7. The method as recited in claim 5, wherein the hostname resolution characteristic is a host-machine ratio.

8. The method as recited in claim 1, wherein the attribute is a link structure.

9. The method as recited in claim 1, wherein the attribute is syntactic content.

10. The method as recited in claim 1, wherein the attribute is content evolution.

11. The method as recited in claim 1, wherein the attribute is a cluster of similar web pages.

12. The method recited in claim 1, wherein the data set is generated prior to selecting a sample population.

13. The method recited in claim 1, wherein analyzing a web page further comprises determining whether web spam is present.

14. The method recited in claim 13, wherein determining whether web spam is present further comprises:

evaluating a plurality of web pages; and

determining the length of a host name associated with each of the web pages.

15. The method recited in claim 13, wherein determining whether web spam is present further comprises:

evaluating the web page, wherein a host name associated with the web page resolves to an address; and

determining whether other web pages resolve other host names to the address.

16. The method recited in claim 13, wherein determining whether web spam is present further comprises evaluating the web page to determine a host-machine ratio.

17. The method recited in claim 16, wherein the host machine ratio is determined by dividing a number of distinct host names contained in the web page by a number of distinct addresses associated with the number of distinct host names.

18. The method recited in claim 1, wherein evaluating the data set further comprises using the statistical distribution to identify an in-degree value that is included in the class of statistical outliers.

19. The method recited in claim 1, wherein analyzing the web page further comprises;

determining an in-degree value of the web page; and

determining whether the in-degree value of the web page is included in the class of statistical outliers.

20. The method recited in claim 1, wherein evaluating the data set further comprises using the statistical distribution to identify an out-degree value that is included in the class of statistical outliers.

21. The method recited in claim 1, wherein analyzing the web page further comprises:

determining an out-degree value of the web page; and

determining whether the out-degree value of the web page is included in the class of statistical outliers.

22. The method recited in claim 1, wherein analyzing the web page further comprises determining whether the web page has a near-zero variance in word count.

23. The method recited in claim 1, wherein analyzing the web page further comprises determining whether the web page has a near-zero variance in size.

24. The method recited in claim 1, wherein analyzing the web page further comprises determining an average number of matching features relative to a number of successive downloads from an address over a period of time.

25. The method recited in claim 1, wherein analyzing the web page further comprises determining the size of clusters of substantially identical web pages.

26. The method recited in claim 1, wherein the class of statistical outliers identifies undesirable content.

27. A method for evaluating content, comprising:

crawling a set of web pages;

evaluating the set of web pages to compute a statistical distribution;

flagging an outlier page in the statistical distribution as web spam; and

creating an index of the web pages and the outlier page for answering a query.

28. A system for evaluating content, comprising:

a memory configured to store data; and

a processor configured to generate a data set using an attribute associated with the content, evaluate the data set using a statistical distribution to identify a class of statistical outliers, and analyze a web page to determine whether it is part of the class of statistical outliers.

29. A computer program product for evaluating content, the computer program product being embodied in a computer readable medium and comprising computer instructions for:

generating a data set using an attribute associated with the content;

evaluating the data set using a statistical distribution to identify a class of statistical outliers; and

analyzing a web page to determine whether it is part of the class of statistical outliers.