Email analysis using fuzzy matching of text

• US 8,515,894 B2
• Filed: 12/30/2009
• Issued: 08/20/2013
• Est. Priority Date: 03/09/2004
• Status: Expired due to Fees

• Current Assignee: Gozoom.com, Inc.
• Original Assignee: Gozoom.com, Inc.
• Inventors: Yu, Tonny
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Tran, Mai T
• Application Number:
  US12/650,431
• Publication Number:
  US 20100106677A1
• Time in Patent Office: 1,329 Days
• Field of Search: 706/52, 706 45- 47, 706/62, 709/206
• US Class Current: 706/52
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

Translation of text or messages provides a message that is more reliably or efficiently analyzed for purposes as, for example, to detect spam in email messages. One translation process takes into account statistics of erroneous and intentional misspellings. Another process identifies and removes characters or character codes that do not generate visible symbols in a message displayed to a user. Another process detects symbols such as periods, commas, dashes, etc., interspersed in text such that the symbols do not unduly interfere with, or prevent, a user from perceiving a spam message. Another process can detect use of foreign language symbols and terms. Still other processes and techniques are presented to counter obfuscating spammer tactics and to provide for efficient and accurate analysis of message content. Groups of similar content items (e.g., words, phrases, images, ASCII text, etc.) are correlated and analysis can proceed after substitution of items in the group with other items in the group so that a more accurate detection of “sameness” of content can be achieved. Dictionaries are used for spam or ham words or phrases. Other features are described.

Claims

1. A method to determine a probability that an email message is spam, the method comprising:

(a) receiving an email message;

(b) identifying one or more words and/or phrases of the email message that are likely being obfuscated;

(c) identifying one or more obfuscation techniques that are being used to obfuscate the one or more words and/or phrases that are identified as likely being obfuscated; and

(d) determining a probability that the email message is spam in dependence on at least both of the following

which particular one or more words and/or phrases are identified as likely being obfuscated, wherein identifying some particular words and/or phrases as likely being obfuscated increases the probability that the email message is spam more than identifying other particular words and/or phrases as likely being obfuscated, and

which particular one or more obfuscation techniques are identified as being used to obfuscate the one or more words and/or phrases that are identified as likely being obfuscated, wherein identifying use of some obfuscation techniques increases the probability that the email message is spam more than identifying use of other obfuscation techniques;

wherein one or more of steps (b), (c) and (d) are performed using one or more processors.

2. The method of claim 1, wherein step (d) includes determining the probability that the email message is spam also in dependence on a quantity of words and/or phrases identified as likely being obfuscated, wherein the higher the quantity of words and/or phrases of the email message identified as likely being obfuscated the higher the probability that the email message is spam.

3. The method of claim 1, further comprising:

deobfuscating each word or phrase of the text that is identified as likely being obfuscated, to produce deobfuscated text; and

analyzing the deobfuscated text by comparing the deobfuscated text to text of one or more other messages known to be spam; and

wherein step (d) includes determining the probability that the email message is spam also in dependence on results of the analyzing the deobfuscated text, wherein the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to be spam the higher the probability the email message is spam.

4. The method of claim 3, wherein:

the analyzing the deobfuscated text also includes comparing the deobfuscated text to text of one or more other messages known to not be spam; and

wherein the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to not be spam the lower the probability the email message is spam.

5. The method of claim 1, further comprising:

deobfuscating each word or phrase of the text that is identified as likely being obfuscated, to produce deobfuscated text; and

analyzing the deobfuscated text by comparing the deobfuscated text to text of one or more other messages known to not be spam; and

wherein step (d) includes determining the probability that the email message is spam also in dependence on results of the analyzing the deobfuscated text, wherein the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to not be spam the lower the probability the email message is spam.

6. The method of claim 1, further comprising:

determining whether the words and/or phrases that are likely being obfuscated are known to be used in spam; and

wherein determining that words and/or phrases known to be used in spam are likely being obfuscated increases the probability being determined at step (d) that the email message is spam more than determining that words and/or phrases not known to be used in spam are likely being obfuscated.

7. The method of claim 1, wherein determining that words and/or phrases known to not be used in spam are likely being obfuscated reduces the probability being determined at step (d) that the email message is spam.

8. A system to determine a probability that an email message is spam, the system comprising:

one or more processors;

machine-readable storage medium including instructions that are executable by the one or more processors;

wherein the instructions include

instructions to receive an email message;

instructions to identify one or more words and/or phrases of the email message that are likely being obfuscated; and

instructions to identify one or more obfuscation techniques that are being used to obfuscate the one or more words and/or phrases that are identified as likely being obfuscated; and

instructions to determine a probability that the email message is spam in dependence on at least both of the following

which particular one or more words and/or phrases are identified as likely being obfuscated, wherein identifying some particular words and/or phrases as likely being obfuscated increases the probability that the email message is spam more than identifying other particular words and/or phrases as likely being obfuscated, and

which particular one or more obfuscation techniques are being used to obfuscate the one or more words and/or phrases that are identified as likely being obfuscated, wherein detecting use of some obfuscation techniques increases the probability that the email message is spam more than detecting use of other obfuscation techniques.

9. The system of claim 8, wherein the instructions to determine a probability that the email message is spam also include instructions to determine the probability that the email message is spam also in dependence on a quantity of words and/or phrases identified as likely being obfuscated, wherein the higher the quantity of words and/or phrases of the email message identified as likely being obfuscated the higher the probability that the email message is spam.

10. The system of claim 8, wherein the instructions also include:

instructions to deobfuscate each word or phrase of the text that is identified as likely being obfuscated, to produce deobfuscated text; and

instructions to compare the deobfuscated text to text of one or more other messages known to be spam;

wherein the instructions to determine a probability that the email message is spam also include instructions to determine the probability that the email message is spam also in dependence on results of the comparison of the deobfuscated text to text of one or more other messages known to be spam.

11. The system of claim 10, wherein the instructions also include:

instructions to compare the deobfuscated text to text of one or more other messages known to not be spam;

wherein the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to not be spam the lower the probability the email message is spam.

12. The system of claim 8, wherein the instructions also include:

instructions to deobfuscate each word or phrase of the text that is identified as likely being obfuscated, to produce deobfuscated text; and

instructions to compare the deobfuscated text to text of one or more other messages known to not be spam;

wherein the determining the probability that the email message is spam is also in dependence on results of the comparison of the deobfuscated text to text of one or more other messages known to not be spam;

wherein the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to not be spam the lower the probability the email message is spam.

13. The system of claim 8, wherein identifying that words and/or phrases known to be used in spam are likely being obfuscated increases the probability that the email message is spam more than identifying that other words and/or phrases not known to be used in spam are likely being obfuscated.

14. The system of claim 8, wherein identifying that words and/or phrases known to not be used in spam are likely being obfuscated reduces the probability that the email message is spam.

15. A non-transitory machine-readable storage medium including instructions executable by one or more processors to determine a probability that an email message is spam, the non-transitory machine-readable storage medium comprising:

instructions to receive an email message;

instruction to identify one or more words and/or phrases of the email message that are likely being obfuscated; and

instructions to determine a probability that the email message is spam in dependence on at least both of the following which particular one or more words and/or phrases are identified as likely being obfuscated, wherein identifying some particular words and/or phrases as likely being obfuscated increases the probability that the email message is spam more than identifying other particular words and/or phrases as likely being obfuscated, and which particular one or more obfuscation techniques are being used to obfuscate the one or more words and/or phrases that are identified as likely being obfuscated, wherein detecting use of some obfuscation techniques increases the probability that the email message is spam more than detecting use of other obfuscation techniques.

16. The non-transitory machine-readable storage medium of claim 15, wherein the instructions to determine a probability that the email message is spam also include instructions to determine the probability that the email message is spam also in dependence on a quantity of words and/or phrases identified as likely being obfuscated, wherein the higher the quantity of words and/or phrases of the email message identified as likely being obfuscated the higher the probability that the email message is spam.

17. The non-transitory machine-readable storage medium of claim 15, wherein the instructions also include:

instructions to deobfuscate each word or phrase of the text that is identified as likely being obfuscated, to produce deobfuscated text; and

instructions to compare the deobfuscated text to text of one or more other messages known to be spam;

wherein the instructions to determine a probability that the email message is spam also include instructions to determine the probability that the email message is spam also in dependence on results of the comparison of the deobfuscated text to text of one or more other messages known to be spam.

18. The non-transitory machine-readable storage medium of claim 17, wherein the instructions also include:

instructions to compare the deobfuscated text to text of one or more other messages known to not be spam;

wherein the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to not be spam the lower the probability the email message is spam.

19. The non-transitory machine-readable storage medium of claim 15, wherein the instructions also include:

instructions to deobfuscate each word or phrase of the text that is identified as likely being obfuscated, to produce deobfuscated text; and

instructions to compare the deobfuscated text to text of one or more other messages known to not be spam;

wherein the determining the probability that the email message is spam is also in dependence on results of the comparison of the deobfuscated text to text of one or more other messages known to not be spam;

wherein the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to not be spam the lower the probability the email message is spam.

20. The non-transitory machine-readable storage medium of claim 15, wherein identifying that words and/or phrases known to be used in spam are likely being obfuscated increases the probability that the email message is spam more than identifying that other words and/or phrases not known to be used in spam are likely being obfuscated.

21. The non-transitory machine-readable storage medium of claim 15, wherein identifying that words and/or phrases known to not be used in spam are likely being obfuscated reduces the probability that the email message is spam.

22. A system to determine a probability that an email message is spam, the system comprising:

means for identifying one or more words and/or phrases of an email message that are likely being obfuscated; and

means for identifying one or more obfuscation techniques that are being used to obfuscate the one or more words and/or phrases that are identified as likely being obfuscated; and

means for determining a probability that the email message is spam in dependence on at least both of the following

which particular one or more words and/or phrases are identified as likely being obfuscated, wherein identifying some particular words and/or phrases as likely being obfuscated increases the probability that the email message is spam more than identifying other particular words and/or phrases as likely being obfuscated, and

which particular one or more obfuscation techniques are being used to obfuscate the one or more words and/or phrases that are identified as likely being obfuscated, wherein detecting use of some obfuscation techniques increases the probability that the email message is spam more than detecting use of other obfuscation techniques.