Method and apparatus for filtering email spam based on similarity measures

• US 20050108340A1
• Filed: 05/13/2004
• Published: 05/19/2005
• Est. Priority Date: 05/15/2003
• Status: Abandoned Application

• Current Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Jensen, Sandy, Gleeson, Matt, Schneider, Ken, Mantel, Eli, Medlar, Art, Hoogstrate, David
• Application Number:
  US10/846,723
• Publication Number:
  US 20050108340A1
• Time in Patent Office: Days
• US Class Current: 709/206
• CPC Class Codes: H04L 51/063 Content adaptation, e.g. replacement of unsuitable content
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

A method and system for filtering email spam based on similarity measures are described. In one embodiment, the method includes receiving an incoming email message, generating data characterizing the incoming email message based on the content of the incoming email message, and comparing the generated data with a set of data characterizing spam messages. The method further includes determining whether a resemblance between the data characterizing the incoming email message and any data item from the set of data characterizing spam messages exceeds a threshold.

Claims

1. The method comprising:

receiving an email message;

generating data characterizing the email message based on content of the email message;

comparing the data characterizing the email message with a set of data characterizing a plurality of spam messages; and

determining whether a resemblance between the data characterizing the email message and any data item within the set of data characterizing the plurality of spam messages exceeds a threshold.

2. The method of claim 1 further comprising:

marking the email message as spam if the resemblance between the data characterizing the email message and any data item within the set of data characterizing the plurality of spam messages exceeds a threshold.

3. The method of claim 1 further comprising:

receiving data characterizing a new spam message; and

storing the received data in a database.

4. The method of claim 1 further comprising:

upon receiving the email message, evaluating the email message for a presence of noise added to avoid spam filtering; and

modifying content of the email message to reduce the noise.

5. The method of claim 4 wherein evaluating the email message for the presence of noise comprises detecting at least one of formatting data, a numeric character reference, a character entity reference, and predefined URL data.

6. The method of claim 1 wherein generating the data characterizing the email message comprises:

dividing the email message into a plurality of tokens; and

calculating a plurality of hash values for the plurality of tokens.

7. The method of claim 6 wherein comparing the data characterizing the email message with the set of data characterizing the plurality of spam messages comprises:

finding, in the set of data characterizing the plurality of spam messages, one or more data items having additional information similar to additional information contained in the data characterizing the plurality of spam messages; and

comparing the subset of hash values in the data characterizing the email message with a subset of hash values in each found data item until finding a similar subset of hash values.

8. The method of claim 3 further comprising:

evaluating the new spam message for a presence of noise;

modifying content of the new spam message to reduce the noise; and

generating data characterizing the spam message based on content of the modified new spam message.

9. The method comprising:

receiving a spam message;

generating data characterizing the spam message based on content of the spam message; and

transferring the data characterizing the spam message to a server, the data characterizing the spam message being subsequently used to find incoming messages resembling the spam message.

10. The method of claim 9 further comprising:

upon receiving the spam message, evaluating the spam message for a presence of noise; and

modifying content of the spam message to reduce the noise.

11. The method of claim 9 wherein evaluating the spam message for the presence of noise comprises detecting at least one of formatting data, a numeric character reference, a character entity reference, and predefined URL data.

12. The method of claim 9 wherein generating the data characterizing the spam message comprises:

dividing the spam message into a plurality of tokens; and

calculating a plurality of hash values for the plurality of tokens.

13. A system comprising:

an incoming message parser to receive an email message;

a message data generator to generate data characterizing the email message based on content of the email message; and

a resemblance identifier to compare the data characterizing the email message with a set of data characterizing a plurality of spam messages, and to determine whether a resemblance between the data characterizing the email message and any data item within the set of data characterizing the plurality of spam messages exceeds a threshold.

14. The system of claim 13 further comprising:

a database to store data characterizing a new spam message.

15. The system of claim 13 further comprising:

a message cleaning algorithm to evaluate the email message for a presence of noise added to avoid spam filtering, and to modify content of the email message to reduce the noise.

16. A system comprising:

a spam content parser to receive a spam message;

a spam data generator to generate data characterizing the spam message based on content of the spam message; and

a spam data transmitter to transfer the data characterizing the spam message to a server, the data characterizing the spam message being subsequently used to find incoming messages resembling the spam message.

17. The system of claim 16 further comprising a noise reduction algorithm to evaluate the spam message for a presence of noise, and to modify content of the spam message to reduce the noise.

18. The system of claim 17 wherein the noise reduction algorithm is to evaluate the spam message for the presence of noise by detecting at least one of formatting data, a numeric character reference, a character entity reference, and predefined URL data.

19. An apparatus comprising:

means for receiving an email message;

means for generating data characterizing the email message based on content of the email message;

means for comparing the data characterizing the email message with a set of data characterizing a plurality of spam messages; and

means for determining whether a resemblance between the data characterizing the email message and any data item within the set of data characterizing the plurality of spam messages exceeds a threshold.

20. The apparatus of claim 19 further comprising:

means for receiving data characterizing a new spam message; and

a database to storing the received data.

21. An apparatus comprising:

means for receiving a spam message;

means for generating data characterizing the spam message based on content of the spam message; and

means for transferring the data characterizing the spam message to a server, the data characterizing the spam message being subsequently used to find incoming messages resembling the spam message.

22. The apparatus of claim 21 further comprising:

means for evaluating the spam message for a presence of noise; and

means for modifying content of the spam message to reduce the noise.

23. The apparatus of claim 21 wherein means for evaluating the spam message for the presence of noise comprises means for detecting at least one of formatting data, a numeric character reference, a character entity reference, and predefined URL data.

24. A computer readable medium comprising executable instructions which when executed on a processing system cause said processing system to perform a method comprising:

receiving an email message;

generating data characterizing the email message based on content of the email message;

comparing the data characterizing the email message with a set of data characterizing a plurality of spam messages; and

determining whether a resemblance between the data characterizing the email message and any data item within the set of data characterizing the plurality of spam messages exceeds a threshold.

25. The computer readable medium of claim 24 wherein the method further comprises:

receiving data characterizing a new spam message; and

storing the received data in a database.

26. The computer readable medium of claim 24 wherein the method further comprises:

upon receiving the email message, evaluating the email message for a presence of noise added to avoid spam filtering; and

modifying content of the email message to reduce the noise.

27. A computer readable medium comprising executable instructions which when executed on a processing system cause said processing system to perform a method comprising:

receiving a spam message;

generating data characterizing the spam message based on content of the spam message; and

transferring the data characterizing the spam message to a server, the data characterizing the spam message being subsequently used to find incoming messages resembling the spam message.

28. The computer readable medium of claim 27 wherein the method further comprises:

upon receiving the spam message, evaluating the spam message for a presence of noise; and

modifying content of the spam message to reduce the noise.

29. The computer readable medium of claim 27 wherein evaluating the spam message for the presence of noise comprises detecting at least one of formatting data, a numeric character reference, a character entity reference, and predefined URL data.