System for email processing and analysis

• US 8,918,466 B2
• Filed: 03/08/2005
• Issued: 12/23/2014
• Est. Priority Date: 03/09/2004
• Status: Expired due to Fees

• Current Assignee: Gozoom.com, Inc.
• Original Assignee: Tonny Yu
• Inventors: Yu, Tonny
• Primary Examiner(s): MACILWINEN, JOHN MOORE JAIN
• Application Number:
  US11/076,577
• Publication Number:
  US 20050262209A1
• Time in Patent Office: 3,577 Days
• Field of Search: 709/206
• US Class Current: 709/206
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 51/48 Message addressing, e.g. address format or anonymous messages, aliases

Abstract

Various features are provided for analyzing and processing email messages including determining if an email message is unwanted, and blocking unwanted messages. Email traffic is monitored by analyzing email messages addressed to known invalid email addresses. Email messages addressed to invalid email addresses are sent to a central control site for analysis. One embodiment tries to ensure that the distance between the invalid addresses and closest valid addresses is significant enough so that the invalid addresses are not inadvertently used for non-spam purposes. Another embodiment of the invention provides for distributed “thin client” processes to run on computer systems or other processing platforms. The thin clients emulate an open relay computer. Attempts at exploiting the apparent open relay computer are reported to a control center and the relay of email messages can be inhibited. Another embodiment provides for analysis and tuning of rules to detect spam and legitimate email. The approach adjusts various factors according to changing, current email data that is gathered from present, or recent, email traffic. Another embodiment takes into account statistics of erroneous and intentional misspellings. Groups of similar content items (e.g., words, phrases, images, ASCII text, etc.) are correlated and analysis can proceed after substitution of items in the group with other items in the group so that a more accurate detection of “sameness” of content can be achieved. Another embodiment uses authentication and security methods for validating email senders, detecting the sameness of messages, tracking the reputation of the sender, and tracking the behavior of the sender. Another embodiment profiles users to intelligently organize user data, including adapting spam detection according to a user'"'"'s perceived interests.

Claims

1. A computer-implemented method for analyzing an email message to determine the likelihood that the email message is spam, the method comprising:

(a) for each of a plurality of message attributes, determining a message attribute occurrence frequency within a collection of known spam email messages, the message attribute occurrence frequency specifying how often one of the plurality of message attributes occurs in the collection of known spam email messages;

(b) for each of the plurality of message attributes, determining a message attribute occurrence frequency within a collection of known non-spam email messages;

(c) determining a spam probability weight for each of the plurality of message attributes based at least in part on the determined message attribute occurrence frequencies determined at steps (a) and (b),

wherein step (c) comprises, when determining the spam probability weight for each of the plurality of message attributes,

(c.1) weighting more highly newer email messages within the collection of known spam email messages than older email messages within the collection of known spam email messages, and

(c.2) weighting more highly newer email messages within the collection of known non-spam email message than older email messages within the collection of known non-spam email messages,

wherein the weighting more highly newer email messages within the collection of known non-spam email messages at (c.2) is to a lesser extent than the weighting more highly newer email messages within the collection of known spam messages at (c.1);

(d) detecting a further email message;

(e) identifying which, if any, of the message attributes are associated with the further email message; and

(f) determining a value indicative of a likelihood that the further email message is spam based at least in part on the message attributes identified at step (e) as being associated with the further email message and spam probability weights corresponding to the identified message attributes;

wherein at least step (f) is performed using a hardware processor.

2. The method of claim 1, wherein:

step (c) comprises adjusting the spam probability weights for at least some of the message attributes based on how difficult it is to falsify the message attributes.

3. The method of claim 2, wherein:

step (c) comprises increasing the weights of message attributes that are difficult to falsify and reducing the weights of message attributes that are easy to falsify.

4. The method of claim 3, wherein message attributes that are email addresses, layout fingerprints, message fingerprints and attachment fingerprints.

5. The method of claim 1, wherein:

step (c) comprises weighting more highly message attributes associated with viewable data than message attributes associated with non-viewable data.

6. The method of claim 1, wherein:

step (c) comprises weighting more highly message attributes that comprise words, in viewable text, that are in a dictionary than message attributes that comprise words, in viewable text, that are not in the dictionary.

7. The method of claim 1, further comprising:

detecting additional email messages, and repeating steps (d), (e) and (f) for each of the additional email messages.

8. The method of claim 7, further comprising:

(g) adjusting the plurality of message attributes as new spam email messages are added to the collection of spam email messages, the adjusting comprising one or more of adding new message attributes to the plurality of message attributes and deleting message attributes from the plurality of message attributes, to accommodate changing types of spam email messages.

9. The method of claim 8, wherein:

step (g) comprises deleting attributes with the least usefulness from the plurality of message attributes, when the number of attributes for the plurality of spam email messages exceeds a specified maximum.

10. The method of claim 1, further comprising, prior to steps (a), (b), and (c):

normalizing each of the spam and non-spam messages; and

un-obfuscating the normalized email messages.

11. The method of claim 1, wherein the steps are performed by one or more hardware processors.

12. A non-transitory machine-readable storage medium including instructions executable by a processor for analyzing an email message to determine the likelihood that the email message is spam, the machine-readable storage medium comprising:

one or more instructions for determining a message attribute occurrence frequency within a collection of known spam email messages for each of a plurality of message attributes, the message attribute occurrence frequency specifying how often one of the plurality of message attributes occurs in the collection of known spam email messages;

one or more instructions for determining a message attribute occurrence frequency within a collection of known non-spam email messages for each of the plurality of message attributes;

one or more instructions for determining a spam probability weight for the message attributes based at least in part on the determined message attribute occurrence frequencies determined for each of the plurality of message attributes within the collection of known spam messages and within the collection of known non-spam messages,

wherein the one or more instructions for determining the spam probability weight for each of the plurality of message attributes comprises one or more instructions for,

weighting more highly newer email messages within the collection of known spam email messages than older email messages within the collection of known spam email messages, and

weighting more highly newer email messages within the collection of known non-spam email messages than older email messages within the collection of known non-spam email messages,

wherein the weighting more highly newer email messages within the collection of known non-spam email messages is to a lesser extent than the weighting more highly newer email messages within the collection of known spam messages;

one or more instructions for detecting a further email message;

one or more instructions for identifying which, if any, of the message attributes are associated with the further email message; and

one or more instructions for determining a value indicative of a likelihood that the further email message is spam, based at least in part on the identified message attributes for the further email message and spam probability weights corresponding to the identified message attributes.

13. The machine readable storage medium of claim 12, wherein:

the one or more instructions for determining a spam probability weight comprises one or more instructions for adjusting the spam probability weights for at least some of the message attributes based on how difficult it is to falsify the message attributes.

14. The machine readable storage medium of claim 13, wherein:

the one or more instructions for determining a spam probability weight comprises one or more instructions for increasing the weights of message attributes that are difficult to falsify and reducing the weights of message attributes that are easy to falsify.

15. The machine readable storage medium of claim 14, wherein message attributes that are difficult to falsify comprise one or more of IP addresses, URLs, URL fragments, phone numbers, email addresses, layout fingerprints, message fingerprints and attachment fingerprints.

16. The machine readable storage medium of claim 12, wherein:

the one or more instructions for determining a spam probability weight comprises one or more instructions for weighting more highly message attributes associated with viewable data than message attributes associated with non-viewable data.

17. The machine readable storage medium of claim 12, wherein:

the one or more instructions for determining a spam probability weight comprises one or more instructions for weighting more highly message attributes that comprise words, in viewable text, that are in a dictionary than message attributes that comprise words, in viewable text, that are not in the dictionary.

18. The machine readable storage medium of claim 12, further comprising:

one or more instructions for adjusting the plurality of message attributes as new spam email messages are added to the collection of spam email messages, the adjusting comprising one or more of adding new attributes to the plurality of message attributes and deleting message attributes from the plurality of message attributes, to accommodate changing types of spam email messages.

19. The machine readable storage medium of claim 18, further comprising:

one or more instructions for deleting message attributes with the least usefulness from the plurality of message attributes, when the number of message attributes for the collection of spam email messages exceeds a specified maximum.

20. The machine readable storage medium of claim 12, further comprising:

one or more instructions for normalizing each of the spam and non-spam messages; and

one or more instructions for un-obfuscating the normalized email messages.

21. A system for analyzing an email message to determine the likelihood that the email message is spam, the system comprising at least one processor and memory, and a non-transitory computer-readable medium having thereon one or more programs, which when executed by the at least on processor, cause the system to:

determine a message attribute occurrence frequency within a collection of spam email messages for each of a plurality of message attributes, the message attribute occurrence frequency specifying how often one of the plurality of message attributes occurs in the collection of known spam email messages rather than how often the one of the plurality of message attributes occurs in an individual spam email message;

determine a message attribute occurrence frequency within a collection of non-spam email messages for each of the plurality of message attributes;

determine a spam probability weight for each of the plurality of message attributes based at least in part on the determined message attribute occurrence frequencies determined for each of the plurality of message attributes, wherein determine a spam probability weight for each of the plurality of message attributes

weights more highly newer email messages within the collection of known spam email messages than older email messages within the collection of known spam email messages, and

weights more highly newer email messages within the collection of known non-spam email messages than older email messages within the collection of known non-spam email messages, but to a lesser extent than newer email messages are weighted more highly within the collection of known spam email messages;

detect a further email message;

identify which, if any, of the message attributes are associated with the further email message; and

determine a value indicative of a likelihood that the further email message is spam based at least in part on the identified message attributes and for the further email message corresponding spam probability weights corresponding to identified message attributes.