CLASSIFICATION AND CLUSTER ANALYSIS SPAM DETECTION AND REDUCTION

• US 20100082800A1
• Filed: 09/29/2008
• Published: 04/01/2010
• Est. Priority Date: 09/29/2008
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Wei, Xiaopeng, YANKOV, Dragomir, RAMARAO, Vishwanath Tumkur, WEI, Stanley
• Granted Patent: US 7,809,824 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

Multiple features of email traffic are analyzed and extracted. Feature vectors comprising the multiple features are created and cluster analysis is utilized to track spam generation even from dynamically changing or aliased IP addresses.

Claims

1. A computer-implemented method for managing email users and email traffic of an email system, comprising:

collecting usage data of email traffic handled by the email system;

generating time series data from the collected usage data;

analyzing the time series data;

analyzing geographic data of the usage data;

analyze sending data of the usage data;

analyzing content features of the email traffic;

creating a plurality of feature vectors comprising indications of;

the analyzed time series data, analyzed geographic data; and

analyzed content features; and

performing cluster analysis on the plurality of feature vectors and cluster groups of vectors into categories.

2. The method of claim 1, further comprising:

assigning one or more email delivery thresholds to at least one categorized cluster group.

3. The method of claim 1, wherein analyzing time series data comprises performing a fast fourier transform of the email traffic.

4. The method of claim 1, wherein analyzing time series data comprises performing a discrete wavelength transform of the email traffic.

5. A computer-implemented method for managing email users and email traffic of an email system, comprising:

collecting usage data of individual and aggregate usage of the email system;

generating time series data from the collected data;

analyzing the time series data;

analyzing geographic data of the individual and aggregate usage;

analyzing sending data of the individual and aggregate usage;

analyzing content features of the individual and aggregate usage;

creating a plurality of feature vectors comprising indications of;

the analyzed time series data, analyzed geographic data; and

analyzed content features;

performing cluster analysis on the plurality of feature vectors and clustering groups of vectors into categories;

assigning a set of permissions and policies to each of the categories of clustered groups, wherein each category is assigned a set and wherein there are two or more sets of permissions and policies; and

applying a first set of permissions and policies to a first clustered group of users and applying a second set of permissions and policies to a second clustered group of users.

6. The method of claim 5, wherein analyzing time series data comprise performing a fast fourier transform of incoming email traffic.

7. The method of claim 5, wherein analyzing time series data comprise performing a discrete wavelength transform of incoming email traffic.

8. The method of claim 5, wherein analyzing geographic data comprises analyzing a location associated with incoming mail, and one or more of:

a connection type, a routing type, a domain, and a subnet.

9. The method of claim 5, wherein analyzing sending data comprises analyzing message size entropy.

10. The method of claim 5, wherein analyzing sending data comprises analyzing user classification history of a message as spam.

11. The method of claim 5, wherein analyzing sending data comprises analyzing, the number of recipients of a message.

12. The method of claim 5, wherein analyzing sending data comprises analyzing the number of file attachments to a message.

13. The method of claim 5, wherein analyzing content features comprises determining or computing one or more of:

most commonly used terms within the email;

a URL within the email; and

a term frequency or inverse term frequency computation result.

14. The method of claim 5, wherein assigning a set of permissions and policies to each of the categories of clustered groups comprises assigning one or more thresholds for maximum send volume per time period.

15. An email delivery and management system, configured to:

collect usage data of email traffic handled by the email system;

generate time series data from the collected usage data;

analyze the time series data;

analyze geographic data of the usage data;

analyze sending data of the usage data;

analyze content features of the email traffic;

create a plurality of feature vectors comprising indications of;

the analyzed time series data, analyzed geographic data; and

analyzed content features; and

perform cluster analysis on the plurality of feature vectors and cluster groups of vectors into categories.

16. The system of claim 15, wherein the system is further configured to:

assign one or more email delivery thresholds to at least one categorized cluster group.

17. The system of claim 15, wherein to analyze time series data the system is configured to perform a fast fourier transform of the email traffic.

18. The system of claim 15, wherein to analyze time series data the system is configured to perform a discrete wavelength transform of the email traffic.

19. The system of claim 15, wherein to analyze geographic data the system is configured to determine a location associated with incoming mail, and one or more of:

a connection type, a routing type, a domain, and a subnet.

20. The system of claim 15, wherein to analyze sending data the system is configured to calculate message size entropy.

21. The system of claim 15, wherein to analyze sending data the system is configured to analyze a user classification history of a message as spam.

22. The system of claim 15, wherein to analyze sending data the system is configured to determine the number of recipients of a message.

23. The system of claim 15, wherein to analyze sending data the system is configured to analyze the number of file attachments to a message.

24. The system of claim 15, wherein to analyze content features the system is configured to determine one or more of:

most commonly used terms within the email;

a URL associated with the email; and

a term frequency or inverse term frequency computation result.