Detecting spam e-mail using similarity calculations

• US 20060149820A1
• Filed: 01/04/2005
• Published: 07/06/2006
• Est. Priority Date: 01/04/2005
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Rajan, Vadakkedathu T., Kephart, Jeffrey O., Crawford, Jason L., Wegman, Mark N., Segal, Richard B., Ossher, Joel
• Application Number:
  US11/028,969
• Publication Number:
  US 20060149820A1
• Time in Patent Office: Days
• US Class Current: 709/206
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

A method for detecting undesirable e-mails is disclosed. The method includes collecting a plurality of undesirable e-mails, arranging the plurality of undesirable e-mails into a plurality of groups and generating, for each group, at least one token, thereby producing a plurality of tokens for the plurality of undesirable e-mails. The method further includes receiving a first e-mail and generating at least one token for the first e-mail. The method further includes causing a comparison of the at least one token for the first e-mail with at least one of the plurality of tokens for the plurality of undesirable e-mails and identifying the first e-mail as an undesirable e-mail if the at least one token for the first e-mail matches any of the plurality of tokens for the plurality of undesirable e-mails.

Claims

1. A method for detecting undesirable e-mail, the method comprising:

collecting a plurality of undesirable e-mails;

arranging the plurality of undesirable e-mails into a plurality of groups;

generating, for each group, at least one token, thereby producing a plurality of tokens for the plurality of undesirable e-mails;

receiving a first e-mail;

generating at least one token for the first e-mail;

causing a comparison of the at least one token for the first e-mail with at least one of the plurality of tokens for the plurality of undesirable e-mails; and

identifying the first e-mail as an undesirable e-mail if the at least one token for the first e-mail matches any of the plurality of tokens for the plurality of undesirable e-mails.

2. The method of claim 1, further comprising:

deleting a first token of the plurality of tokens for the plurality of undesirable e-mails if the first token matches a token for a desirable e-mail.

3. The method of claim 1, further comprising:

deleting a first token of the plurality of tokens for the plurality of undesirable e-mails if the first token matches another token of the plurality of tokens for the plurality of undesirable e-mails.

4. The method of claim 1, wherein a token comprises a string of contiguous characters from an e-mail.

5. The method of claim 1, wherein a token comprises a string of contiguous characters of fixed length from an e-mail.

6. The method of claim 1, wherein a token comprises a string of characters from an e-mail, wherein a hash of the characters meet a criteria.

7. The method of claim 1, wherein a token comprises a k-gram including a string of 20 to 30 consecutive bytes from an e-mail.

8. The method of claim 1, wherein the first step of generating comprises:

generating, for each group, at least one token, thereby producing a plurality of tokens for the plurality of undesirable e-mails, wherein a weight based on token length is associated with each token.

9. The method of claim 1, wherein the first step of generating comprises:

generating, for each group, at least one token, thereby producing a plurality of tokens for the plurality of undesirable e-mails, wherein a weight based on token frequency is associated with each token.

10. The method of claim 1, wherein the first step of generating comprises:

generating, for each group, at least one token, thereby producing a plurality of tokens for the plurality of undesirable e-mails, wherein a weight based the relative frequency of a token within groups as compared with its frequency between groups.

11. The method of claim 1, wherein the step of causing to compare comprises:

performing a byte-by-byte comparison of the at least one token for the first e-mail with the plurality of tokens for the plurality of undesirable e-mails, wherein a match is found if the at least one token for the first e-mail is identical to at least one of the plurality of tokens for the plurality of undesirable e-mails.

12. The method of claim 1, wherein the step of identifying comprises:

identifying the first e-mail as an undesirable e-mail if the at least one token for the first e-mail matches more than one of the plurality of tokens for the plurality of undesirable e-mails.

13. The method of claim 1, further comprising:

scoring the first e-mail for undesirability based on the number of tokens for the first e-mail that match the plurality of tokens for the plurality of undesirable e-mails.

14. The method of claim 1, further comprising:

scoring the first e-mail for undesirability based on weights of the tokens for the first e-mail that match the plurality of tokens for the plurality of undesirable e-mails.

15. The method of claim 1, wherein an e-mail is deemed undesirable if the e-mail is sent to a first e-mail account.

16. The method of claim 1, wherein an e-mail is deemed undesirable if the e-mail is identified as undesirable by the user.

17. The method of claim 1, with the additional step of deleting spam-filtering countermeasures in at least one e-mail.

18. An information processing system for detecting undesirable e-mail, comprising:

a memory for collecting a plurality of undesirable e-mails;

a receiver for receiving a first e-mail; and

a processor configured for;

arranging the plurality of undesirable e-mails into a plurality of groups;

generating, for each group, at least one token, thereby producing a plurality of tokens for the plurality of undesirable e-mails;

generating at least one token for the first e-mail;

causing a comparison of the at least one token for the first e-mail with at least one of the plurality of tokens for the plurality of undesirable e-mails; and

identifying the first e-mail as an undesirable e-mail if the at least one token for the first e-mail matches any of the plurality of tokens for the plurality of undesirable e-mails.

19. The information processing system of claim 18, the processor further configured for:

deleting a first token of the plurality of tokens for the plurality of undesirable e-mails if the first token matches a token for a desirable e-mail.

20. The information processing system of claim 18, the processor further configured for:

deleting a first token of the plurality of tokens for the plurality of undesirable e-mails if the first token matches another token of the plurality of tokens for the plurality of undesirable e-mails.

21. The information processing system of claim 18, wherein a token comprises a string of contiguous characters from an e-mail.

22. The information processing system of claim 18, wherein a token comprises a string of contiguous characters of fixed length from an e-mail.

23. The information processing system of claim 18, wherein a token comprises a string of characters from an e-mail, wherein a hash of the characters meet a criteria.

24. The information processing system of claim 18, wherein a token comprises a k-gram including a string of 20 to 30 consecutive bytes from an e-mail.

25. The information processing system of claim 18, wherein an e-mail is deemed undesirable if the e-mail is sent to a first e-mail account.

26. A computer readable medium including computer instructions for detecting undesirable e-mail, the computer instructions including instructions for:

collecting a plurality of undesirable e-mails;

arranging the plurality of undesirable e-mails into a plurality of groups;

generating, for each group, at least one token, thereby producing a plurality of tokens for the plurality of undesirable e-mails;

receiving a first e-mail;

generating at least one token for the first e-mail;

causing a comparison of the at least one token for the first e-mail with at least one of the plurality of tokens for the plurality of undesirable e-mails; and

identifying the first e-mail as an undesirable e-mail if the at least one token for the first e-mail matches any of the plurality of tokens for the plurality of undesirable e-mails.

27. A method for detecting undesirable e-mail, the method comprising:

collecting a plurality of desirable and undesirable e-mails;

generating at least one token for the plurality of desirable and undesirable e-mails,

receiving a first e-mail;

generating at least one token for the first e-mail;

causing a comparison of the at least one token for the first e-mail with at least one of the plurality of tokens for the plurality of desirable or undesirable e-mails; and

identifying the first e-mail as an desirable or undesirable e-mail based on the result of the comparison between at least one token for the first e-mail with at least one of the plurality of tokens for the plurality of desirable or undesirable e-mails.

28. The method of claim 27, wherein the first generating step comprises creating at least one token for the plurality of undesirable e-mails, wherein the token does not occur more than a specified number of times in the plurality of desirable e-mails, thereby producing a plurality of tokens for the plurality of undesirable e-mails;

29. The method of claim 27, wherein the second generating step comprises creating at least two tokens for the first e-mail and the comparison step comprises comparing the at least two tokens for the first e-mail with at least two of the plurality of tokens for the plurality of desirable or undesirable e-mail.

30. The method of claim 27, wherein the first step of generating comprises:

generating, for each e-mail, at least one token, thereby producing a plurality of tokens for the plurality of undesirable e-mails, wherein a weight based on token length is associated with each token.

31. The method of claim 27, wherein the first step of generating comprises:

generating, for each group, at least one token, thereby producing a plurality of tokens for the plurality of undesirable e-mails, wherein a weight based on token frequency in desirable and undesirable e-mail is associated with each token.

32. The method of claim 27, wherein the step of causing to compare comprises:

performing a byte-by-byte comparison of the at least one token for the first e-mail with the plurality of tokens for the plurality of undesirable e-mails, wherein a match is found if the at least one token for the first e-mail is identical to at least one of the plurality of tokens for the plurality of undesirable e-mails.

33. The method of claim 27, wherein the step of causing to compare comprises:

performing a byte-by-byte comparison of the at least one token for the first e-mail with the plurality of tokens for the plurality of undesirable e-mails, wherein a match is found if the at least one token for the first e-mail is similar to at least one of the plurality of tokens for the plurality of undesirable e-mails.

34. The method of claim 27, wherein the step of identifying comprises:

identifying the first e-mail as an undesirable e-mail if the at least one token for the first e-mail matches more than one of the plurality of tokens for the plurality of undesirable e-mails.

35. The method of claim 27, further comprising:

scoring the first e-mail for undesirability based on the number of tokens for the first e-mail that match the plurality of tokens for the plurality of undesirable e-mails.

36. The method of claim 27, further comprising:

scoring the first e-mail for undesirability based on weights of the tokens for the first e-mail that match the plurality of tokens for the plurality of undesirable e-mails.

37. The method of claim 27, wherein an e-mail is deemed undesirable if the e-mail is sent to a first e-mail account.

38. The method of claim 27, wherein an e-mail is deemed undesirable if the e-mail is identified as undesirable by the user.

39. The method of claim 27, with the additional step of deleting spam-filtering countermeasures in at least one e-mail.

40. A method for detecting undesirable e-mail, the method comprising:

collecting a plurality of undesirable e-mails;

generating at least one token for the plurality of undesirable e-mails, thereby producing a plurality of tokens for the plurality of undesirable e-mails;

generating a weight associated with each of the plurality of tokens, wherein a weight is based on token length;

receiving a first e-mail;

generating at least one token for the first e-mail;

causing a comparison of the at least one token for the first e-mail with at least one of the plurality of tokens for the plurality of undesirable e-mails; and

identifying the first e-mail as an undesirable e-mail if the at least one token for the first e-mail matches any of the plurality of tokens for the plurality of undesirable e-mails.