Email analysis using fuzzy matching of text

• US 7,644,127 B2
• Filed: 03/09/2005
• Issued: 01/05/2010
• Est. Priority Date: 03/09/2004
• Status: Expired due to Fees

• Current Assignee: Gozoom.com, Inc.
• Original Assignee: Gozoom.com, Inc.
• Inventors: Yu, Tonny
• Primary Examiner(s): Caldwell; Andrew
• Assistant Examiner(s): RECEK, JASON D
• Application Number:
  US11/077,506
• Publication Number:
  US 20050262210A1
• Time in Patent Office: 1,763 Days
• Field of Search: 709/206
• US Class Current: 709/206
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

Translation of text or messages provides a message that is more reliably or efficiently analyzed for purposes as, for example, to detect spam in email messages. One translation process takes into account statistics of erroneous and intentional misspellings. Another process identifies and removes characters or character codes that do not generate visible symbols in a message displayed to a user. Another process detects symbols such as periods, commas, dashes, etc., interspersed in text such that the symbols do not unduly interfere with, or prevent, a user from perceiving a spam message. Another process can detect use of foreign language symbols and terms. Still other processes and techniques are presented to counter obfuscating spammer tactics and to provide for efficient and accurate analysis of message content. Groups of similar content items (e.g., words, phrases, images, ASCII text, etc.) are correlated and analysis can proceed after substitution of items in the group with other items in the group so that a more accurate detection of “sameness” of content can be achieved. Dictionaries are used for spam or ham words or phrases. Other features are described.

Claims

1. A method for analyzing character codes in text of a message to determine a probability that the message is spam, the method comprising:

(a) receiving a message that includes character codes in text of the message;

(b) identifying character codes of the text that are likely being used to obfuscate a word or phrase of the text;

(c) deobfuscating each word or phrase of the text that is identified at step (b) as likely being obfuscated, to produce deobfuscated text; and

(d) determining an extent that the character codes of the text are identified as likely being used to obfuscate a word or phrase of the text by determining one or more of the following

a quantity of words or phrases identified at step (b) as likely being obfuscated;

which particular words or phrases are identified at step (b) as likely being obfuscated, and

(e) analyzing the deobfuscated text by comparing the deobfuscated text to text of one or more other messages known to be spam;

(f) determining a probability that the message is spam based on both

(i) results of the analyzing the deobfuscated text, by comparing the deobfuscated text to text of one or more other messages known to be spam, performed at step (e), and

(ii) results of the determining the extent that the character codes of the text are likely being used to obfuscate a word or phrase of the text, performed at step (d);

wherein at least steps (b), (c), (d), (e) and (f) are performed by one or more processors.

2. The method of claim 1, wherein

the following rules are used at step (f), when determining the probability that the message is spam:

the higher the quantity of words or phrases of the message identified as likely being obfuscated the higher the probability that the message is spam;

identifying some particular words or phrases as likely being obfuscated increases the probability that the message is spam more than identifying other particular words or phrases as likely being obfuscated; and

the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to be spam the higher the probability the message is spam.

3. The method of claim 1, wherein one of a plurality of rules used at step (f), when determining the probability that the message is spam, specifies that the higher the quantity of words or phrases of the message identified as likely being obfuscated the higher the probability that the message is spam.

4. The method of claim 1, wherein one of a plurality of rules used at step (f), when determining the probability that the message is spam, specifies that identifying some particular words or phrases as likely being obfuscated increases the probability that the message is spam more than identifying other particular words or phrases as likely being obfuscated.

5. The method of claim 4, wherein step (f) includes taking into account whether words that are likely being obfuscated are known to be used in spam.

6. The method of claim 5, wherein step (f) includes determining that words that are likely being obfuscated are known to be used in spam is used to increase the probability that the message is spam.

7. The method of claim 1, wherein one of a plurality of rules used at step (f), when determining the probability that the message is spam, specifies that the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to be spam the higher the probability the message is spam.

8. The method of claim 1, wherein step (f) includes taking into account which obfuscation techniques were likely used to obfuscate a word or phrase of the text, when determining the probability that the message is spam.

9. The method of claim 8, wherein step (d) includes detecting use of some obfuscation techniques increases the probability that the message is spam more than detecting use of other obfuscation techniques.

10. The method of claim 1, wherein:

step (b) includes identifying one or more words that are likely misspelled;

step (c) includes replacing one or more words that are likely misspelled with one or more words that were likely intended if the one or more words were correctly spelled, when producing the deobfuscated text;

step (d) includes searching the deobfuscated text for words and phrases that are known to be used in spam;

step (e) includes comparing the deobuscated text to text of one or more other messages known to be spam to determine a degree of sameness;

step (f) includes

taking into account whether one or more word that is likely misspelled is a word or is part of a phrase that is known to be used in spam, when determining the probability that the message is spam; and

also taking into account the degree of sameness when determining the probability that the message is spam.

11. The method of claim 10, wherein step (f) includes determining that words that are likely being misspelled are known to be used in spam is used to increase the probability that the message is spam.

12. The method of claim 1, wherein the deobfuscating performed at step (c) includes one or more techniques selected from the group consisting of:

replacing a non-alphanumeric character with an alphanumeric character;

deleting a white space character;

replacing a number character with a letter character;

replacing one or more symbol character with a letter character; and

replacing a non-English character with an English character.

13. A machine-readable storage medium including instructions executable by one or more processors to analyze character codes in text of a message to determine a probability that the message is spam, machine-readable storage medium comprising:

instructions to receive a message that includes character codes in text of the message;

instructions to identify character codes of the text that are likely being used to obfuscate a word or phrase of the text;

instructions to deobfuscate each word or phrase of the text that is identified as likely being obfuscated, to produce deobfuscated text;

instructions to determine an extent that the character codes of the text are identified as likely being used to obfuscate a word or phrase of the text by determining one or more of the following

a quantity of words or phrases identified as likely being obfuscated, and

which particular words or phrases are identified as likely being obfuscated;

instructions to analyze the deobfuscated text by comparing the deobfuscated text to text of one or more other messages known to be spam; and

instructions to determine a probability that the message is spam based on both

(i) results of analysis of the deobfuscated text by comparing the deobfuscated text to text of one or more other messages known to be spam, and

(i) results of determining the extent that the character codes of the text are likely being used to obfuscate a word or phrase of the text.

14. The machine-readable storage medium of claim 13, wherein the instructions to determine a probability that the message is spam include instructions that the higher the quantity of words or phrases of the message identified as likely being obfuscated the higher the probability that the message is spam.

15. The machine-readable storage medium of claim 13, wherein the instructions to determine a probability that the message is spam include instructions that identifying some particular words or phrases as likely being obfuscated increases the probability that the message is spam more than identifying other particular words or phrases as likely being obfuscated.

16. The machine-readable storage medium of claim 15, wherein the instructions to determine a probability that the message is spam include instructions that the more similar the deobfuscated text is to the text of at least one of the one or more other messages known to be spam the higher the probability the message is spam.

17. The machine-readable storage medium of claim 13, wherein the instructions to determine a probability that the message is spam include instructions that take into account which obfuscation techniques were likely used to obfuscate a word or phrase of the text, when determining the probability that the message is spam.

18. The machine-readable storage medium of claim 13, wherein:

the instructions to identify character codes of the text that are likely being used to obfuscate a word or phrase of the text, include instructions to identify one or more words that are likely misspelled;

the instructions to deobfuscate any word or phrase of the text that is likely being obfuscated, to produce deobfuscated text, include instructions to replace one or more words that are likely misspelled with one or more words that were likely intended if the one or more words were correctly spelled;

the instructions to determine a probability that the message is spam include instructions to search the deobfuscated text for words and phrases that are known to be used in spam, and instructions that take into account whether one or more word that is likely misspelled is a word or is part of a phrase that is known to be used in spam.

19. The machine-readable storage medium of claim 13, wherein the instructions to deobfuscate any word or phrase of the text that is likely being obfuscated, to produce deobfuscated text, include instructions selected from the group consisting of:

instructions to replace a non-alphanumeric character with an alphanumeric character;

instructions to delete a white space character;

instructions to replace a number character with a letter character;

instructions to replace one or more symbol character with a letter character; and

instructions to replace a non-English character with an English character.

20. A system to analyze character codes in text of a message to determine a probability that the message is spam, comprising:

at least one processor;

machine-readable storage medium including instructions that are executable by the at least one processor, the instructions including

instructions to receive a message that includes character codes in text of the message;

instructions to identify character codes of the text that are likely being used to obfuscate a word or phrase of the text;

instructions to deobfuscate each word or phrase of the text that is identified as likely being obfuscated, to produce deobfuscated text;

instructions to determine an extent that the character codes of the text are identified as likely being used to obfuscate a word or phrase of the text by determining one or more of the following

a quantity of words or phrases identified as likely being obfuscated, and

which particular words or phrases are identified as likely being obfuscated;

instructions to analyze the deobfuscated text by comparing the deobfuscated text to text of one or more other messages known to be spam; and

instructions to determine a probability that the message is spam based on both

(i) results of analysis of the deobfuscated text, by comparing the deobfuscated text to text of one or more other messages known to be spam, and

(ii) results of determining the extent that the character codes of the text are likely being used to obfuscate a word or phrase of the text.