Detecting spam documents in a phrase based information retrieval system

• US 20060294155A1
• Filed: 06/28/2006
• Published: 12/28/2006
• Est. Priority Date: 07/26/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Patterson, Anna Lynn
• Granted Patent: US 7,603,345 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99937 Sorting

Abstract

An information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are identified that predict the presence of other phrases in documents. Documents are the indexed according to their included phrases. A spam document is identified based on the number of related phrases included in a document.

Claims

1. A computer implemented method for identifying spam documents in an information retrieval system, the method comprising:

maintaining a list of phrases, each phrase associated with a list of related phrases;

determining a number of related phrases expected to be present in a document for any phrase on the list of phrases;

determining for a document, and for at least one phrase in the document, an actual number of related phrases present in the document; and

identifying the document as a spam document by comparing the actual number of related phrases present in the document with the expected number of related phrases.

2. The method of claim 1, wherein determining a number of related phrases expected to be present in a document for any phrase on the list of phrases further comprises:

traversing an index of documents;

for each document, determining a set of phrases in the document from the list of phrases, and for each phrase in the document, determining a number of related phrases also in the document;

determining the expected number of related phrases, as a medium of the determined number of related phrases across the traversed documents.

3. The method of claim 1, wherein identifying the document as a spam document, further comprises:

responsive to the actual number of related phrases present in the document for at least one phrase significantly exceeding the expected number of related phrases, identifying the document as a spam document.

4. The method of claim 1, wherein identifying the document as a spam document, further comprises:

responsive to the actual number of related phrases present in the document for at least one phrase exceeding the expected number of related phrases by at least a multiple of a standard deviation of the expected number of related phrases, identifying the document as a spam document.

5. The method of claim 1, wherein identifying the document as a spam document, further comprises:

responsive to the actual number of related phrases present in the document for at least one phrase exceeding the expected number of related phrases by at least a multiple of the expected number of related phrases, identifying the document as a spam document.

6. The method of claim 1, wherein identifying the document as a spam document, further comprises:

identifying the document as a spam document where, for each of a minimum plurality of phrases in the document, the actual number of related phrases present in the document significantly exceeds the expected number of related phrases.

7. The method of claim 1, wherein identifying the document as a spam document, further comprises:

identifying the document as a spam document where the actual number of related phrases present in the document for at least one phrase exceeds predetermined maximum expected number of related phrases.

8. The method of claim 1, wherein identifying the document as a spam document, further comprises:

determining for a document, a set of most significant phrases present in the document;

for each of the most significant related phrases, determining an actual number of related phrases present in the document; and

responsive to the actual number of related phrases significantly exceeds the expected number of related phrases, identifying the document as a spam document with respect to that significant phrase.

9. The method of claim 1, further comprises:

responsive to identifying the document as a spam document, adding the document to a list of spam documents.

10. The method of claim 9, further comprising:

receiving a search query;

retrieving a set of documents relevant to the search query, each document having a relevance score;

for each document in the set of documents, determining whether the document has been identified as a spam document; and

responsive to a document being identified as a spam document, down weighting the relevance score of the document;

organizing the set of documents by their relevance scores.

11. The method of claim 8, further comprising:

adding the document to a list of spam document associated with the most significant phrase; and

for each related phrase of the most significant phrase, adding the document to a list of spam documents associated with the related phrase.