System and method for spam filtering using insignificant shingles

• US 9,391,936 B2
• Filed: 02/17/2015
• Issued: 07/12/2016
• Est. Priority Date: 06/06/2013
• Status: Active Grant

• Current Assignee: Kaspersky Lab ZAO
• Original Assignee: Lab A.O. Kaspersky
• Inventors: Zagorsky, Sergey G., Loseva, Darya V., Skvortsov, Vladimir A.
• Primary Examiner(s): Christensen, Scott B
• Application Number:
  US14/623,977
• Publication Number:
  US 20150163183A1
• Time in Patent Office: 511 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 51/52 for supporting social networking services

Abstract

Disclosed system and methods for detecting spam using shingles. An example system identifies in a received message one or more insignificant text portions based on a text pattern database storing defined insignificant text patterns not containing spam; removes at least a portion of the one or more identified insignificant text portions from the message to generate an abridged and canonized message; generates a set of shingles from the abridged and canonized message; identifies in the set of shingles one or more shingles based on a shingles database storing defined insignificant shingles that occur only in messages not containing spam; removes one or more identified shingles from the set of shingles to generate a reduced set of shingles upon detecting the one or more identified shingles matching at least one of the defined insignificant shingles; and determines whether the received message contains spam based on the reduced set of shingles.

Claims

1. A computer-implemented method for detecting spam in a message, the method comprising:

identifying in a received message one or more insignificant text portions based on a text pattern database storing defined insignificant text patterns not containing spam;

removing at least a portion of the one or more identified insignificant text portions from the message to generate an abridged and canonized message;

generating a set of shingles from the abridged and canonized message;

identifying in the set of shingles one or more shingles based on a shingles database storing defined insignificant shingles that occur only in messages not containing spam;

removing one or more identified shingles from the set of shingles to generate a reduced set of shingles upon detecting the one or more identified shingles matching at least one of the defined insignificant shingles; and

determining whether the received message contains spam based on the reduced set of shingles.

2. The method of claim 1, wherein the defined insignificant text patterns comprise a plurality of words, phrases, symbols that occur only in messages not containing spam.

3. The method of claim 2, further comprising:

collecting from known users messages not containing spam;

identifying in the collected messages insignificant text patterns and shingles; and

storing the identified insignificant text patterns and insignificant shingles in each respective text pattern database and shingles database.

4. The method of claim 1, wherein removing at least the portion of the one or more identified insignificant text portions from the message to generate the abridged message is performed upon detecting the one or more identified insignificant text portions matching at least one of the defined insignificant text patterns.

5. The method of claim 1, wherein each defined insignificant text pattern comprises a text pattern, text identification information and a usage frequency, and each defined insignificant shingle comprises a hash, a shingle pattern, a text portion corresponding to the shingle pattern, and a usage frequency.

6. The method of claim 1, wherein determining whether the received message contains spam comprises performing spam filtering of the reduced set of shingles using a Bayes method.

7. The method of claim 1, wherein determining whether the received message contains spam comprises performing spam filtering of the reduced set of shingles using a cluster filtering method.

8. A computer system for detecting spam, the system comprising:

a processor configured to;

identify in a received message one or more insignificant text portions based on a text pattern database storing defined insignificant text patterns not containing spam;

remove at least a portion of the one or more identified insignificant text portions from the message to generate an abridged and canonized message;

generate a set of shingles from the abridged and canonized message;

identify in the set of shingles one or more shingles based on a shingles database storing defined insignificant shingles that occur only in messages not containing spam;

remove one or more identified shingles from the set of shingles to generate a reduced set of shingles upon detecting the one or more identified shingles matching at least one of the defined insignificant shingles; and

determine whether the received message contains spam based on the reduced set of shingles.

9. The system of claim 8, wherein the defined insignificant text patterns comprise a plurality of words, phrases, symbols that occur only in messages not containing spam.

10. The system of claim 9, wherein the processor is further configured to:

collect from known users messages not containing spam;

identify in the collected messages insignificant text patterns and shingles; and

store the identified insignificant text patterns and insignificant shingles in each respective text pattern database and shingles database.

11. The system of claim 8, wherein removing at least the portion of the one or more identified insignificant text portions from the message to generate the abridged message is performed upon detecting the one or more identified insignificant text portions matching at least one of the defined insignificant text patterns.

12. The system of claim 8, wherein each defined insignificant text pattern comprises a text pattern, text identification information and a usage frequency, and each defined insignificant shingle comprises a hash, a shingle pattern, a text portion corresponding to the shingle pattern, and a usage frequency.

13. The system of claim 8, wherein determining whether the received message contains spam comprises performing spam filtering of the reduced set of shingles using a Bayes method.

14. The system of claim 8, wherein determining whether the received message contains spam comprises performing spam filtering of the reduced set of shingles using a cluster filtering method.

15. A computer program product stored on a non-transitory computer-readable storage medium, the computer program product comprising computer-executable instructions for detecting spam, including instructions for:

identifying in a received message one or more insignificant text portions based on a text pattern database storing defined insignificant text patterns not containing spam;

removing at least a portion of the one or more identified insignificant text portions from the message to generate an abridged and canonized message;

generating a set of shingles from the abridged and canonized message;

identifying in the set of shingles one or more shingles based on a shingles database storing defined insignificant shingles that occur only in messages not containing spam;

removing one or more identified shingles from the set of shingles to generate a reduced set of shingles upon detecting the one or more identified shingles matching at least one of the defined insignificant shingles; and

determining whether the received message contains spam based on the reduced set of shingles.

16. The product of claim 15, wherein the defined insignificant text patterns comprise a plurality of words, phrases, symbols that occur only in messages not containing spam.

17. The product of claim 16, further comprising instructions for:

collecting from known users messages not containing spam;

identifying in the collected messages insignificant text patterns and shingles; and

storing the identified insignificant text patterns and insignificant shingles in each respective text pattern database and shingles database.

18. The product of claim 15, wherein removing at least the portion of the one or more identified insignificant text portions from the message to generate the abridged message is performed upon detecting the one or more identified insignificant text portions matching at least one of the defined insignificant text patterns.

19. The product of claim 15, wherein each defined insignificant text pattern comprise text pattern, text identification information and a usage frequency, and each defined insignificant shingle comprises a hash, a shingle pattern, a text portion corresponding to the shingle pattern, and a usage frequency.

20. The product of claim 15, wherein the instructions for determining whether the received message contains spam comprise instructions for performing spam filtering of the reduced set of shingles using at least one of:

a Bayes method or a duster filtering method.