Method and system for automated evaluation of spam filters

• US 8,205,264 B1
• Filed: 09/04/2009
• Issued: 06/19/2012
• Est. Priority Date: 09/04/2009
• Status: Active Grant

• Current Assignee: Zscaler Incorporated
• Original Assignee: Zscaler Incorporated
• Inventors: Kailash, Kailash, Raphel, Jose, Natarajan, Sriram
• Primary Examiner(s): Pich, Ponnoreay
• Application Number:
  US12/554,873
• Time in Patent Office: 1,019 Days
• Field of Search: 726/25
• US Class Current: 726/25
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

A method for automated evaluation of a SPAM filter rates a sender-receiver pair based on traffic information related to the sender-receiver pair. A SPAM filter intercepts electronic messages transmitted between the sender-receiver pair and classifies them as either SPAM or HAM. On comparing the rating for the sender-receiver pair and the classification for an electronic message between the sender-receiver pair, the method generates a metric indicating the reliability of the spam filter. Repeating these steps for more than one electronic messages and related sender-receiver pairs, the method produces a set of metrics. One or more of the metrics from the set of metrics are used to calculate an evaluation factor for evaluating the SPAM filter.

Claims

1. A computer-implemented method for automated evaluation of a spam filter, the method comprising:

rating a sender-receiver pair based on traffic information related to the sender-receiver pair;

classifying an electronic message transmitted between the sender-receiver pair as spam or not spam using a spam filter;

comparing the rating and the classification to generate a metric indicating the reliability of the classification, wherein the metric is one of a set of metrics; and

evaluating the spam filter using an evaluation factor based on the set of metrics.

2. The method of claim 1 further comprising computing the evaluation factor based on one or more metrics from the set of metrics, wherein each metric in the set of metrics is accrued from a plurality of sender-receiver pairs.

3. The method of claim 1, wherein the evaluation factor is one of precision, recall, f-measure and accuracy.

4. The method of claim 3, wherein the respective values of f-measure and accuracy are directly proportionate to the performance of the spam filter.

5. The method of claim 1, wherein the rating belongs to an ordered list of ratings, the first rating in the ordered list indicating that the sender of the electronic message is known to generate spam and the last rating in the ordered list indicating that the sender of the electronic message is known not to generate spam.

6. The method of claim 1, wherein the rating is altered upon transmission of an electronic message between the sender-receiver pair.

7. The method of claim 1, wherein the rating is altered based on a numerical score accrued during an interval, wherein the numerical score is associated with the sender-receiver pair.

8. The method of claim 1 further comprising storing the rating in a hash table.

9. The method of claim 8, wherein the storing step further comprises:

generating a hash of the sender-receiver pair using a collision-resistant hash function; and

storing the hash with the rating of the sender-receiver pair in the hash table.

10. The method of claim 1, wherein the comparing step further includes altering the rating of the sender-receiver pair.

11. A computer-implemented system for automated evaluation of a spam filter, the system comprising:

a server computer comprising;

a rating module configured to rate a sender-receiver pair based on traffic information related to the sender-receiver pair; and

a processing module operatively coupled to the rating module, the processing module including;

a spam filter configured to classify an electronic message transmitted between the sender-receiver pair as spam or not spam;

a comparison module configured to compare the rating and the classification, generating a metric indicating the reliability of the classification, wherein the metric is one of a set of metrics; and

a computation module configured to compute an evaluation factor, based on the set of metrics, for evaluating the spam filter.

12. The system of claim 11, wherein the computation module is further configured to compute the evaluation factor based on one or more metrics from the set of metrics, wherein each metric in the set of metrics is accrued from a plurality of sender-receiver pairs.

13. The system of claim 12, wherein a counter accrues each metric in the set of metrics.

14. The system of claim 11, wherein the evaluation factor is one of precision, recall, f-measure and accuracy.

15. The system of claim 14, wherein the respective values of f-measure and accuracy are directly proportionate to the performance of the spam filter.

16. The system of claim 11, wherein the rating belongs to an ordered list of ratings, the first rating in the ordered list indicating that the sender of the electronic message is known to generate spam and the last rating in the ordered list indicating that the sender of the electronic message is known not to generate spam.

17. The system of claim 11, wherein the rating is altered upon transmission of an electronic message between the sender-receiver pair.

18. The system of claim 11, wherein the rating is altered based on a numerical score accrued during an interval, wherein the numerical score is associated with the sender-receiver pair.

19. The system of claim 11, wherein a storing module is configured to store the rating in a hash table.

20. The system of claim 19, wherein the storing module performs the steps of:

generating a hash of the sender-receiver pair using a collision-resistant hash function; and

storing the hash with the rating of the sender-receiver pair in the hash table.

21. The system of claim 11, wherein the comparison module is further configured to alter the rating of the sender-receiver pair.

22. The system of claim 11, wherein the rating module is part of the processing module.

23. A computer-implemented method for automated evaluation of a spam filter, the method comprising:

rating a sender based on traffic information related to the sender;

classifying an electronic message transmitted by the sender as spam or not spam using a spam filter;

comparing the rating and the classification to generate a metric indicating the reliability of the classification, wherein the metric is one of a set of metrics; and

evaluating the spam filter using an evaluation factor based on the set of metrics.

24. The method of claim 23 further comprising computing the evaluation factor based on one or more metrics from the set of metrics, wherein each metric in the set of metrics is accrued from a plurality of senders.

25. The method of claim 23, wherein the rating is altered upon transmission of an electronic message by the sender.

26. The method of claim 23, wherein the rating is altered based on a numerical score accrued during an interval, wherein the numerical score is associated with the sender.

27. The method of claim 23, wherein the comparing step further includes altering the rating of the sender.