Adaptive junk message filtering system

• US 20040167964A1
• Filed: 02/25/2003
• Published: 08/26/2004
• Est. Priority Date: 02/25/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Platt, John C., Rounthwaite, Robert L., Goodman, Joshua T., Heckerman, David E., Kadie, Carl M.
• Granted Patent: US 7,249,162 B2
• Time in Patent Office: Days
• US Class Current: 709/206
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

The invention relates to a system for filtering messages—the system includes a seed filter having associated therewith a false positive rate and a false negative rate. A new filter is also provided for filtering the messages, the new filter is evaluated according to the false positive rate and the false negative rate of the seed filter, the data used to determine the false positive rate and the false negative rate of the seed filter are utilized to determine a new false positive rate and a new false negative rate of the new filter as a function of threshold. The new filter is employed in lieu of the seed filter if a threshold exists for the new filter such that the new false positive rate and new false negative rate are together considered better than the false positive and the false negative rate of the seed filter.

Claims

1. A data filtering system, comprising:

a first filter for filtering messages, the first filter having associated therewith a false positive rate and a false negative rate; and

a second filter for filtering the messages, the second filter evaluated according to the false positive rate and the false negative rate of the first filter, the data used to determine the false positive rate and the false negative rate of the first filter utilized to determine a new false positive rate and a new false negative rate associated with the second filter as a function of threshold, wherein the second filter is employed in lieu of the first filter if a threshold exists for the second filter such that the new false positive rate and new false negative rate are together considered better than the false positive and the false negative rate of the first filter.

2. The system of claim 1, the false positive rate and the false negative rate determined according to messages that are labeled as junk and non-junk via employment of a user-correction process.

3. The system of claim 2, the user-correction process includes overriding an initial classification of a message, the initial classification performed automatically by the first filter upon receipt of the message.

4. The system of claim 1, the false positive rate and the false negative rate derived from content of at least one of the messages.

5. The system of claim 1, the false positive rate and the false negative rate derived from content of other user'"'"'s e-mail messages.

6. The system of claim 1, the second filter is employed when the new false positive rate is worse than that of the first filter.

7. The system of claim 1, the false positive rate and false negative rate determined after at least one of a predetermined number of junk and non-junk messages are labeled and a predetermined time has occurred.

8. The system of claim 1, the threshold selected from a plurality of generated threshold values, the selected threshold values determined by choosing at least one of an average threshold value over eligible threshold values, a threshold value with lowest false positive rate, and a threshold value that maximizes user'"'"'s expected utility based upon a p* utility function.

9. The system of claim 1, the threshold selected from a plurality of threshold values, the second filter employed only if the new false positive rate and the new false negative rate are better than the false positive rate and the false negative rate of the first filter at that threshold.

10. The system of claim 1, further comprising a plurality of secondary filters, the plurality comprising the second filter, the system employing at least one of the secondary filters in lieu of the first filter if a threshold exists for the at least one secondary filter such that the new false positive rate and new false negative rate are together considered better than the false positive and the false negative rate of the first filter.

11. The system of claim 10, the secondary filters comprising M filters (M being an integer), the system selecting filter M₁ in lieu of the first filter for a particular message, and selecting filter M₂ in lieu of the first filter for another message.

12. A computer-readable medium having computer-executable components stored thereon to effect the system of claim 1.

13. A computer comprising the system of claim 1.

14. A network comprising the system of claim 1.

15. A portable computing device comprising the system of claim 1.

16. The device of claim 15 being one of:

a personal data assistant, a telephone or a laptop computer.

17. A data filter, comprising:

a first filter for filtering messages, and having first accuracy data associated therewith; and

a second filter for filtering the messages, and having associated therewith second accuracy data, the second filter evaluated with the first accuracy data, the data used to determine the first accuracy data are utilized to determine the second accuracy data as a function of threshold, wherein the second filter is employed if a threshold exists for the second filter such that the second accuracy data is considered better than the first accuracy data.

18. The filter of claim 17, the second filter employed at least one of in combination with the first filter and in lieu of the first filter.

19. The filter of claim 17, the second filter employed when the second accuracy rate is at least the same as the first accuracy rate.

20. The filter of claim 17, at least one of the first accuracy data and the second accuracy data comprises a false positive rate and a false negative rate.

21. The filter of claim 17, the first accuracy data and the second accuracy data determined based at least upon one of message text and message content.

22. The filter of claim 17, the first filter being a seed filter derived from processing other user e-mail data.

23. The filter of claim 17, the first accuracy data determined according to a user-correction process wherein a user reviews the data, which data is a message, and tags the message as one of a junk message and a non-junk message.

24. The filter of claim 23, the user-correction process includes overriding an initial classification of the message, the initial classification performed automatically by the first filter when the message is received.

25. The filter of claim 17, the second filter employed when the threshold value that is used to employ the second filter is at least as efficient as the first filter based upon a p* function where N is at least twenty.

26. The filter of claim 17, the first accuracy data determined according to a predetermined set of data known with a high degree of certainty to be correct.

27. The filter of claim 26, the predetermined set of data including at least one of a message that is tagged by the user as a non-junk message, a message that is read and deleted by the user, a message that is forwarded by the user, and a message that is replied to by the user.

28. The filter of claim 17, the first accuracy data determined by a probability value that is received from a calibrated filter, which probability value is used to estimate a false positive rate of the first accuracy data.

29. The filter of claim 17, the first accuracy data utilized to generate an expectation value.

30. The filter of claim 29, the second filter employed only if the actual number of user corrections is at least the same as the expectation value.

31. The filter of claim 17, the threshold selected from a plurality of threshold values, the new filter employed only if the second accuracy data is better than the first accuracy data at that threshold.

32. A method of facilitating data filtering, comprising:

determining a false positive rate and a false negative rate associated with a seed filter;

training a new filter utilizing seed data associated with the seed filter, the seed data used in determining a new false positive rate and a new false negative rate of the new filter as a function of threshold; and

employing the new filter in lieu of the seed filter if a threshold exists for the new filter such that the new false positive and new false negative rates are together considered better than that of the seed filter.

33. The method of claim 32, generating in part the seed data based upon labeling messages as junk and non-junk via employment of a user-correction process.

34. The method of claim 32, the user-correction process includes overriding an initial classification of the message(s), the initial classification performed automatically by the seed filter receiving the message(s).

35. The method of claim 32, further comprising deriving the seed data from content of an e-mail message.

36. The method of claim 32, further comprising deriving the seed data from content of other user'"'"'s e-mail messages.

37. The method of claim 32, further comprising employing the new filter when the new false positive rate is worse than that of the seed filter.

38. The method of claim 32, further comprising determining the false positive rate and false negative rate after at least one of a predetermined number of junk and non-junk messages are labeled, and a predetermined time has occurred.

39. The method of claim 32, employing the new filter if the new false positive and new false negative rates are together considered better than that of the seed filter at a current threshold setting and a weighted value is at least zero.

40. The method of claim 39, increasing the weighted value when performance of the new filter is better at the current threshold setting than at a non-current threshold setting, which non-current threshold setting is a threshold other than the current threshold, and decreasing the weighted value when performance of the new filter is worse at the current threshold setting than at the non-current threshold setting.

41. A method of filtering data, comprising:

receiving at least a first filter and a second filter;

determining a first accuracy data of the first filter;

training the second filter utilizing the first accuracy data;

determining a second accuracy data of the second filter as a function of threshold; and

employing the second filter when a predetermined threshold value is reached.

42. The method of claim 41, the second filter employed at least one of in combination with the first filter and in lieu of the first filter.

43. The method of claim 41, the second filter employed when the second accuracy rate is at least the same as the first accuracy rate.

44. The method of claim 41, at least one of the first accuracy data and the second accuracy data comprises a false positive rate and a false negative rate.

45. The method of claim 41, the first accuracy data and the second accuracy data determined based at least upon one of e-mail text and e-mail content.

46. The method of claim 41, the first filter being a seed filter derived from processing other user e-mail data.

47. The method of claim 41, the first accuracy data determined according to a user-correction process wherein a user reviews the data, which data is an e-mail message, and tags the e-mail message as one of a junk message and a non-junk message.

48. The method of claim 47, the user-correction process includes overriding an initial classification of the e-mail message, the initial classification performed automatically by the first filter when the e-mail message is received.

49. The method of claim 41, further comprising employing the second filter when the threshold value that is used to generate the second filter is at least as efficient as the first filter based upon a p* function where N is at least twenty.

50. The method of claim 41, the first accuracy data determined according to a predetermined set of data known with a high degree of certainty to be correct.

51. The method of claim 50, the predetermined set of data including at least one of a message that is tagged by the user as a non-junk message, a message that is read and deleted by the user, a message that is forwarded by the user, and a message that is replied to by the user.

52. The method of claim 41, the first accuracy data determined by a probability value that is received from a calibrated filter, which probability value is used to estimate a false positive rate of the first accuracy data.

53. The method of claim 41, further comprising generating an expectation value based upon the first accuracy data.

54. The method of claim 53, employing the second filter only if the actual number of user corrections is at least the same as the expectation value.

55. A graphical user interface for facilitating data filtering, comprising:

a filter interface adapted to communicate with a configuration system for configuring a filter;

at least one input of the filter interface for receiving filter configuration information for configuring the filter; and

at least one output of the filter interface for committing the filter configuration information to the configuration system to generate.

56. The interface of claim 55 operable to commit the filter configuration information for processing to generate an updated message filter.

57. The interface of claim 55 operable to provide a plurality of user-selectable filter levels including at least one of default, enhanced, and exclusive.

58. The interface of claim 55, further comprising a user mailbox interface for facilitating tagging of a message as junk.

59. The interface of claim 58, the user mailbox interface facilitating access to at least one of a junk mail folder and an old junk mail folder.

60. A computer-readable medium having computer executable instructions for configuring a message filter, comprising:

a first filter for filtering messages, the first filter having associated therewith a false positive rate and a false negative rate; and

a second filter for filtering the messages, the second filter evaluated according to the false positive rate and the false negative rate, the data used to determine the false positive rate and false negative rate of the first filter are utilized to determine a new false positive rate and a new false negative rate of the second filter as a function of threshold, wherein the second filter is employed in lieu of the first filter if a threshold exists for the second filter such that the new false positive rate and new false negative rate are together considered better than the false positive and the false negative rate of the first filter.

61. The computer-readable medium of claim 60, the false positive rate and the false negative rate determined according to a user-correction process wherein a user reviews the data, which data is an e-mail message, and tags the e-mail message as one of a junk message and a non-junk message.

62. The computer-readable medium of claim 60, the second filter employed when the second accuracy rate is at least the same as the first accuracy rate.

63. A data filtering system, comprising:

means for receiving messages;

first means for filtering the messages, the first means for filtering having associated therewith a false positive rate and a false negative rate; and

new means for filtering the messages, the new means for filtering being trained according to the false positive rate and the false negative rate of the first means for filtering, the data used to determine the false positive rate and the false negative rate of the first filter are utilized to determine a new false positive rate and a new false negative rate associated with the new filtering means as a function of threshold, wherein the new filtering means is employed in lieu of the first filtering means if a threshold exists for the new filtering means such that the new false positive rate and new false negative rate are together considered better than the false positive and the false negative rate of the first filtering means.