SYSTEMS AND METHODS FOR IDENTIFYING UNWANTED OR HARMFUL ELECTRONIC TEXT

• US 20100205123A1
• Filed: 08/08/2007
• Published: 08/12/2010
• Est. Priority Date: 08/10/2006
• Status: Abandoned Application

• Current Assignee: Trustees of Tufts College
• Original Assignee: Trustees of Tufts College
• Inventors: Wachman, Gabriel, Brokley, Carla E., Sculley, D.
• Application Number:
  US12/376,970
• Publication Number:
  US 20100205123A1
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 21/562 Static detection
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00

Abstract

The present invention relates to systems and methods for identifying and removing unwanted or harmful electronic text (e.g., spam). In particular, the present invention provides systems and methods utilizing inexact string matching methods and machine learning and non-learning methods for identifying and removing unwanted or harmful electronic text.

Claims

1. A method for identifying unwanted or harmful electronic text comprising:

analyzing electronic text using an inexact string matching algorithm to identify unwanted or harmful text, if present in said electronic text, wherein said inexact string matching algorithm utilizes a database generated by machine learning method.

2. The method of claim 1, wherein said electronic text is contained in an electronic mail message.

3. The method of claim 1, wherein said electronic text is contained in an instant message.

4. The method of claim 1, wherein said electronic text is contained in a webpage.

5. The method of claim 1, wherein said inexact string matching algorithm is provided by a processor accessing a computer readable medium.

6. The method of claim 5, wherein said processor is provided on a computer.

7. The method of claim 5, wherein said processor is provided on a personal digital assistant.

8. The method of claim 5, wherein said processor is provided on a phone.

9. The method of claim 1, wherein said inexact string matching algorithm is provided by an electronic service provided over an electronic communication network.

10. The method of claim 1, wherein said inexact string matching algorithm is configured to analyze overlapping n-grams.

11. The method of claim 10, wherein said inexact string matching algorithm is configured to analyze overlapping n-grams comprising wildcard features.

12. The method of claim 11, wherein said wildcard features comprise fixed wildcard features.

13. The method of claim 10, wherein said inexact string matching algorithm is configured to analyze overlapping n-grams comprising mismatch features.

14. The method of claim 10, wherein said inexact string matching algorithm is configured to analyze overlapping n-grams comprising gappy features.

15. The method of claim 1, wherein said inexact string matching algorithm is configured to analyze a substring of text contained in said electronic text, wherein said substring is analyzed with and without gaps, wildcards, and mismatches.

16. The method of claim 1, wherein said inexact string matching algorithm is configured to analyze a sequence of features including one or more of n-grams, wildcard features, mismatch features, gappy features, substring features, repetition features, transposition features, transformation features, and at-a-distance features.

17. The method of claim 1, wherein said inexact string matching algorithm is configured to analyze a combination features including two or more of n-grams, wildcard features, mismatch features, gappy features, substring features, repetition features, transposition features, transformation features, and at-a-distance features.

18. The method of claim 1, wherein said inexact string matching algorithm is configured to analyze a number of features found in said electronic text or a substring of said electronic text, wherein said features are selected from the group consisting of:

n-grams, wildcard features, mismatch features, gappy features, substring features, repetition features, transposition features, transformation features, and at-a-distance features.

19. The method of claim 1, wherein said inexact string matching algorithm is configured to analyze features found in said electronic text or a substring of said electronic text, wherein said features are selected from the group consisting of:

n-grams, wildcard features, mismatch features, gappy features, substring features, repetition features, transposition features, transformation features, and at-a-distance features, and wherein said features are analyzed using a Kernel method to represent the features implicitly.

20. The method of claim 1, wherein said machine learning method is a supervised learning method.

21. The method of claim 20, wherein said supervised learning method is an on-line linear classifier.

22. The method of claim 21, wherein said on-line linear classifier is perceptron algorithm with margins.

23. The method of claim 1, wherein said machine learning method is an unsupervised learning method.

24. The method of claim 1, wherein said machine learning method is a semi-supervised learning method.

25. The method of claim 1, wherein said machine learning method is an active learning method.

26. The method of claim 1, wherein said machine learning method is an anomaly detection method.

27. The method of claim 1, wherein said machine learning method stores feature information in said database generated by said inexact string matching algorithm.

28. The method of claim 27, wherein said feature information is simplified prior to storage.

29. The method of claim 28, wherein said simplifying is conducted using a process selected from the group consisting of mutual information and principle component analysis.

30. The method of claim 27, wherein said feature information is transformed prior to storage in said database.

31. The method of claim 30, wherein said transforming is conducted using a process selected from the group consisting of rank approximation, latent semantic indexing, and smoothing.

32. The method of claim 1, wherein said unwanted or harmful electronic text is unwanted advertising.

33. The method of claim 1, wherein said unwanted or harmful electronic text is adult content.

34. The method of claim 1, wherein said unwanted or harmful electronic text is illegal content.

35. The method of claim 1, wherein said inexact string matching algorithm is configured to identify a feature using one or more of n-grams, wildcard features, mismatch features, gappy features, substring features, repetition features, transposition features, transformation features, and at-a-distance features, wherein a score is assigned based on a mathematical function associated with said features.

36. The method of claim 35, wherein said score is assigned based on a function depending on the number of times the features occur in said electronic text.

37. The method of claim 35, wherein said score is assigned based on a function depending on the existence of said features in said electronic text.

38. The method of claim 35, wherein said score is assigned based on a function depending on the relative frequency of the functions in said electronic text.

39. The method of claim 1, wherein said machine learning method utilizes said inexact string matching algorithm.

40. The method of claim 39, wherein said machine learning method utilizes said inexact string matching algorithm to explicitly generate features of said electronic text.

41. The method of claim 39, wherein said machine learning method utilizes said inexact string matching algorithm to implicitly generate features of said electronic text.

42. The method of claim 1, wherein said electronic text is contained in a larger electronic text document.

43. The method of claim 1, wherein said electronic text is transformed with an algorithm that edits the electronic text prior to using said inexact string matching algorithm.

44. The method of claim 1, further comprising the step of generating a score that indicates the level of harmfulness of said electronic text.

45. A system comprising a processor and a computer readable medium configured to carry out the method of claim 1.

46. A system comprising a computer readable medium encoding an algorithm configured to carry out the method of claim 1.