Hash-based systems and methods for detecting and preventing transmission of unwanted e-mail

• US 8,204,945 B2
• Filed: 10/09/2008
• Issued: 06/19/2012
• Est. Priority Date: 06/19/2000
• Status: Expired due to Fees

• Current Assignee: Stragent, LLC (Oso IP, LLC)
• Original Assignee: Stragent, LLC (Oso IP, LLC)
• Inventors: Milliken, Walter Clark, Strayer, William Timothy, Milligan, Stephen Douglas
• Primary Examiner(s): Hamza, Faruk
• Application Number:
  US12/248,790
• Publication Number:
  US 20090132669A1
• Time in Patent Office: 1,349 Days
• Field of Search: 709204-207, 709223-224, 726/22
• US Class Current: 709/206
• CPC Class Codes: G06F 21/562 Static detection
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A system (120) detects transmission of potentially unwanted e-mail messages. The system (120) may receive e-mail messages and generate hash values based on one or more portions of the e-mail messages. The system (120) may then determine whether the generated hash values match hash values associated with prior e-mail messages. The system (120) may determine that one of the e-mail messages is a potentially unwanted e-mail message when one or more of the generated hash values associated with the e-mail message match one or more of the hash values associated with the prior e-mail messages.

Claims

1. A method for detecting potentially unwanted e-mail messages, comprising:

receiving a plurality of e-mail messages;

processing the plurality of e-mail messages by removing HTML comments and HTML tags from the plurality of e-mail messages;

generating one or more hash values based on one or more portions of the plurality of e-mail messages such that each of the plurality of e-mail messages has one or more corresponding generated hash values;

counting the one or more of the generated hash values associated with at least one of the plurality of e-mail messages that match one or more hash values associated with at least one prior e-mail message;

determining that the at least one of the plurality of e-mail messages is a potentially unwanted e-mail message based on the counting of the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more of the hash values associated with the at least one prior e-mail message;

generating at least one suspicion score for the at least one of the plurality of e-mail messages based on the counting of the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message; and

taking remedial action on the at least one of the plurality of e-mail messages that is determined to be the potentially unwanted e-mail message, based on the at least one suspicion score, the taking remedial action including deleting a newly received e-mail message.

2. The method of claim 1, wherein the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of variable-sized blocks of a main text of the plurality of e-mail messages.

3. The method of claim 1, wherein the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of fixed-sized blocks of a main text of the plurality of e-mail messages.

4. The method of claim 1, wherein the generating the one or more hash values includes:

performing a plurality of hashes on a main text of the plurality of e-mail messages using a plurality of different hash functions.

5. The method of claim 1, wherein the generating the one or more hash values includes:

performing a plurality of hashes on a main text of the plurality of e-mail messages using a same hash function.

6. The method of claim 1, wherein the generating the one or more hash values includes:

attempting to expand an attachment of the plurality of e-mail messages, and

hashing the attachment after attempting to expand the attachment.

7. The method of claim 1, wherein the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of variable-sized blocks of an attachment of the plurality of e-mail messages.

8. The method of claim 1, wherein the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of fixed-sized blocks of an attachment of the plurality of e-mail messages.

9. The method of claim 1, wherein the generating the one or more hash values includes:

performing a plurality of hashes on an attachment of the plurality of e-mail messages using a plurality of different hash functions.

10. The method of claim 1, wherein the generating the one or more hash values includes:

performing a plurality of hashes on an attachment of the plurality of e-mail messages using a same hash function.

11. The method of claim 1, further comprising:

comparing the generated one or more hash values to hash values corresponding to known unwanted e-mails.

12. The method of claim 11, wherein the known unwanted e-mails include at least one of e-mails containing a virus, e-mails containing a worm, or unsolicited commercial e-mails.

13. The method of claim 1, wherein the generating the one or more hash values includes:

hashing at least one of a main text or an attachment to generate one or more first hash values, and

hashing a concatenation of first and second header fields to generate a second hash value.

14. The method of claim 13, wherein the first and second header fields include a From header field and a To header field.

15. The method of claim 13, wherein the counting the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message includes:

determining a first suspicion count based on a number of the hash values associated with the at least one prior e-mail message that match the one or more first hash values, and

determining a second suspicion count based on a number of the hash values associated with the at least one prior e-mail message that match the second hash value.

16. The method of claim 15, wherein the determining that at least one of the plurality of e-mail messages is a potentially unwanted e-mail message includes:

determining that the at least one of the plurality of e-mail messages is the potentially unwanted e-mail message when the first suspicion count is significantly higher than the second suspicion count.

17. The method of claim 1, wherein the taking remedial action further includes at least one of:

bouncing the at least one of the plurality of e-mail messages,

marking the at least one of the plurality of e-mail messages with a warning,

subjecting the at least one of the plurality of e-mail messages to a virus or worm detection process, or

creating a notification message.

18. The method of claim 1, wherein the generating the one or more hash values and the counting the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message are performed incrementally as the plurality of e-mail messages are being received.

19. The method of claim 1, further comprising:

comparing the generated one or more hash values to hash values associated with one or more known legitimate mailing lists; and

passing the plurality of e-mail messages without further examination when the one or more generated hash values associated with the at least one of the plurality of e-mail messages match one or more hash values associated with the one or more known legitimate mailing lists.

20. The method of claim 19, wherein the comparing the one or more generated hash values includes:

determining whether the plurality of e-mail messages originated from the one or more known legitimate mailing lists.

21. The method of claim 1, wherein the generating the one or more hash values includes:

hashing a main text to generate a first hash value, and

hashing one or more sender-related header fields to generate one or more second hash values.

22. The method of claim 21, wherein the one or more sender-related header fields include at least one of a From header field, a Sender header field, or a Reply-To header field.

23. The method of claim 21, wherein the counting the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message includes:

determining a first suspicion count based on a number of the hash values associated with the at least one prior e-mail message that match the first hash value, and

determining one or more second suspicion counts based on a number of the hash values associated with the at least one prior e-mail message that match the one or more second hash values.

24. The method of claim 23, wherein the determining that at least one of the plurality of e-mail messages is a potentially unwanted e-mail message includes:

determining that the at least one of the plurality of e-mail messages is the potentially unwanted e-mail message when the first suspicion count is higher than the one or more second suspicion counts.

25. The method of claim 1, wherein the generating the one or more hash values includes:

hashing a main text of the plurality of e-mail messages to generate at least one main text hash, and

hashing at least one header field of the plurality of e-mail messages to generate at least one header hash.

26. The method of claim 25, wherein the counting the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message includes:

determining whether the at least one main text hash matches a substantially higher number of the hash values associated with the at least one prior e-mail message than the at least one header hash; and

wherein the determining that at least one of the plurality of e-mail messages is a potentially unwanted e-mail message includes;

determining that the at least one of the plurality of e-mail messages is the potentially unwanted e-mail message when the at least one main text hash matches a substantially higher number of the hash values associated with the at least one prior e-mail message than the at least one header hash.

27. A computer program product embodied on a non-transitory computer-readable medium, comprising:

computer code for receiving a plurality of e-mail messages;

computer code for processing the plurality of e-mail messages by removing HTML comments and HTML tags from the plurality of e-mail messages;

computer code for generating one or more hash values based on one or more portions of the plurality of e-mail messages such that each of the plurality of e-mail messages has one or more corresponding generated hash values;

computer code for counting the one or more of the generated hash values associated with at least one of the plurality of e-mail messages that match one or more hash values associated with at least one prior e-mail message;

computer code for determining that the at least one of the plurality of e-mail messages is a potentially unwanted e-mail message based on the counting of the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more of the hash values associated with the at least one prior e-mail message;

computer code for generating at least one suspicion score for the at least one of the plurality of e-mail messages based on the counting of the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message; and

computer code for taking remedial action on the at least one of the plurality of e-mail messages that is determined to be the potentially unwanted e-mail message, based on the at least one suspicion score, the taking remedial action including deleting a newly received e-mail message.

28. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of variable-sized blocks of a main text of the plurality of e-mail messages.

29. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of fixed-sized blocks of a main text of the plurality of e-mail messages.

30. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a main text of the plurality of e-mail messages using a plurality of different hash functions.

31. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a main text of the plurality of e-mail messages using a same hash function.

32. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

attempting to expand an attachment of the plurality of e-mail messages, and

hashing the attachment after attempting to expand the attachment.

33. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of variable-sized blocks of an attachment of the plurality of e-mail messages.

34. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of fixed-sized blocks of an attachment of the plurality of e-mail messages.

35. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on an attachment of the plurality of e-mail messages using a plurality of different hash functions.

36. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on an attachment of the plurality of e-mail messages using a same hash function.

37. The computer program product of claim 27, further comprising:

computer code for comparing the generated hash values to hash values corresponding to known unwanted e-mails.

38. The computer program product of claim 37, wherein the known unwanted e-mails include at least one of e-mails containing a virus, e-mails containing a worm, or unsolicited commercial e-mails.

39. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

hashing at least one of a main text or an attachment to generate one or more first hash values, and

hashing a concatenation of first and second header fields to generate a second hash value.

40. The computer program product of claim 39, wherein the first and second header fields include a From header field and a To header field.

41. The computer program product of claim 39, wherein the computer program product is operable such that the counting the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message includes:

determining a first suspicion count based on a number of the hash values associated with the at least one prior e-mail message that match the one or more first hash values, and

determining a second suspicion count based on a number of the hash values associated with the at least one prior e-mail message that match the second hash value.

42. The computer program product of claim 41, wherein the computer program product is operable such that the determining that at least one of the plurality of e-mail messages is a potentially unwanted e-mail message includes:

determining that the at least one of the plurality of e-mail messages is the potentially unwanted e-mail message when the first suspicion count is significantly higher than the second suspicion count.

43. The computer program product of claim 27, wherein the computer program product is operable such that the taking remedial action further includes at least one of:

bouncing the at least one of the plurality of e-mail messages,

marking the at least one of the plurality of e-mail messages with a warning,

subjecting the at least one of the plurality of e-mail messages to a virus or worm detection process, or

creating a notification message.

44. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values and the counting the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message are performed incrementally as the plurality of e-mail messages are being received.

45. The computer program product of claim 27, further comprising:

computer code for comparing the generated one or more hash values to has values associated with one or more known legitimate mailing lists; and

computer code for passing the plurality of e-mail messages without further examination when the one or more generated hash values associated with the at least one of the plurality of e-mail messages match one or more hash values associated with the one or more known legitimate mailing lists.

46. The computer program product of claim 45, wherein the computer program product is operable such that the comparing the generated the one or more hash values includes:

determining whether the plurality of e-mail messages originated from the one or more known legitimate mailing lists.

47. The computer program product of claim 27, wherein the generating the one or more hash values includes:

hashing a main text to generate a first hash value, and

hashing one or more sender-related header fields to generate one or more second hash values.

48. The computer program product of claim 47, wherein the computer program product is operable such that the one or more sender-related header fields include at least one of a From header field, a Sender header field, or a Reply-To header field.

49. The computer program product of claim 47, wherein the computer program product is operable such that the counting the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message includes:

determining a first suspicion count based on a number of the hash values associated with the at least one prior e-mail message that match the first hash value, and

determining one or more second suspicion counts based on a number of the hash values associated with the at least one prior e-mail message that match the one or more second hash values.

50. The computer program product of claim 49, wherein the computer program product is operable such that the determining that at least one of the plurality of e-mail messages is a potentially unwanted e-mail message includes:

determining that the at least one of the plurality of e-mail messages is the potentially unwanted e-mail message when the first suspicion count is higher than the one or more second suspicion counts.

51. The computer program product of claim 27, wherein the computer program product is operable such that the generating the one or more hash values includes:

hashing a main text of the plurality of e-mail messages to generate at least one main text hash, and

hashing at least one header field of the plurality of e-mail messages to generate at least one header hash.

52. The computer program product of claim 51, wherein the computer program product is operable such that the counting the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message includes:

determining whether the at least one main text hash matches a substantially higher number of the hash values associated with the at least one prior e-mail message than the at least one header hash; and

wherein the determining that at least one of the plurality of e-mail messages is a potentially unwanted e-mail message includes;

determining that the at least one of the plurality of e-mail messages is the potentially unwanted e-mail message when the at least one main text hash matches a substantially higher number of the hash values associated with the at least one prior e-mail message than the at least one header hash.

53. The computer program product of claim 37, wherein the hash values are stored using one or more bloom filters.

54. The computer program product of claim 27, wherein the computer program product is operable such that the one or more bloom filters are used in connection with the determining that the at least one of the plurality of e-mail messages is the potentially unwanted e-mail message.

55. The computer program product of claim 27, wherein the computer program product is operable such that the one or more corresponding generated hash values each serve as a signature for data over which the one or more corresponding generated hash values was computed.

56. The computer program product of claim 27, wherein the computer program product is operable such that the at least one suspicion score for the at least one of the plurality of e-mail messages is based on whether the count of the one or more of the generated hash values exceeds a threshold.

57. The computer program product of claim 56, wherein the threshold is a settable threshold.

58. The computer program product of claim 27, wherein the one or more generated hash values includes a plurality of generated hash values.

59. The computer program product of claim 27, wherein the one or more of the hash values associated with the at least one prior e-mail message include one or more hash values of at least one portion of content of the at least one prior e-mail message that was determined to be a known unsolicited commercial e-mail message.

60. The computer program product of claim 27, wherein the computer readable medium includes non-volatile memory.

61. The computer program product of claim 27, wherein the computer program product is operable such that the one or more hash values are generated using a hash function that provides a distribution of values over a variety of data inputs sufficient to prevent hash collisions.

62. The computer program product of claim 27, wherein the computer program product is operable such that the one or more of the generated hash values are indexed.

63. The computer program product of claim 27, wherein the computer program product is operable such that the at least one suspicion score includes a suspicion count.

64. The computer program product of claim 27, wherein the computer program product is operable such that the at least one suspicion score is based on a scoring function.

65. The computer program product of claim 27, wherein the computer program product is operable such that the at least one suspicion score is based on a scoring function that is a function of a plurality of subparts of the plurality of e-mail messages.

66. The computer program product of claim 27, wherein the computer program product is operable such that at least a portion of the computer code is executed at a server.

67. The computer program product of claim 27, wherein the computer program product is operable such that the remedial action is programmable.

68. The computer program product of claim 27, wherein the one or more of the generated hash values includes one generated hash value.

69. The computer program product of claim 27, wherein the at least one prior e-mail message includes one prior e-mail message.

70. The computer program product of claim 27, wherein the at least one suspicion score includes one suspicion score.

71. The computer program product of claim 27, wherein the one or more of the hash values associated with the at least one prior e-mail message include one or more hash values of the at least one prior e-mail message.

72. The computer program product of claim 27, wherein the one or more of the hash values associated with the at least one prior e-mail message include one or more hash values of the at least one prior e-mail message that was determined to be a known unsolicited commercial e-mail message.

73. A system including a processor and a non-transitory computer-readable medium, comprising:

means for receiving a plurality of e-mail messages;

means for processing the plurality of e-mail messages by removing HTML comments and HTML tags from the plurality of e-mail messages;

means for generating one or more hash values based on one or more portions of the plurality of e-mail messages such that each of the plurality of e-mail messages has one or more corresponding generated hash values;

means for counting the one or more of the generated hash values associated with at least one of the plurality of e-mail messages that match one or more hash values associated with at least one prior e-mail message;

means for determining that the at least one of the plurality of e-mail messages is a potentially unwanted e-mail message based on the counting of the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more of the hash values associated with the at least one prior e-mail message;

means for generating at least one suspicion score for the at least one of the plurality of e-mail messages based on the counting of the one or more of the generated hash values associated with the at least one of the plurality of e-mail messages that match the one or more hash values associated with the at least one prior e-mail message; and

means for taking remedial action on the at least one of the plurality of e-mail messages that is determined to be the potentially unwanted e-mail message, based on the at least one suspicion score, the taking remedial action including deleting a newly received e-mail message.

74. The system of claim 73, wherein the system is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of variable-sized blocks of a main text of the plurality of e-mail messages.

75. The system of claim 73, wherein the system is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of fixed-sized blocks of a main text of the plurality of e-mail messages.

76. The system of claim 73, wherein the system is operable such that the generating the one or more hash values includes:

attempting to expand an attachment of the plurality of e-mail messages, and

hashing the attachment after attempting to expand the attachment.

77. The system of claim 73, wherein the system is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of variable-sized blocks of an attachment of the plurality of e-mail messages.

78. The system of claim 73, wherein the system is operable such that the generating the one or more hash values includes:

performing a plurality of hashes on a plurality of fixed-sized blocks of an attachment of the plurality of e-mail messages.

79. The system of claim 73, further comprising:

means for comparing the generated hash values to hash values corresponding to known unwanted e-mails.

80. The system of claim 79, wherein the known unwanted e-mails include at least one of e-mails containing a virus, e-mails containing a worm, or unsolicited commercial e-mails.

81. The system of claim 73, wherein the system is operable such that the generating the one or more hash values and the counting the one or more of the generated hash values that match the hash values associated with the at least one prior e-mail message are performed incrementally as the plurality of e-mail messages are being received.

82. The system of claim 73, further comprising:

means for comparing the generated one or more hash values to hash values associated with one or more known legitimate mailing lists; and

means for passing the plurality of e-mail messages without further examination when the one or more generated hash values associated with the at least one of the plurality of e-mail messages match one or more has values associated with the one or more known legitimate mailing lists.

83. The system of claim 73, wherein the system is operable such that the generating the one or more hash values includes:

hashing a main text of the plurality of e-mail messages to generate a main text hash, and

hashing at least one header field of the plurality of e-mail messages to generate at least one header hash.