Binary filter using pattern recognition

• US 6,144,934 A
• Filed: 09/18/1996
• Issued: 11/07/2000
• Est. Priority Date: 09/18/1996
• Status: Expired due to Term

• Current Assignee: McAfee, LLC
• Original Assignee: Secure Computing Corporation (McAfee, LLC)
• Inventors: Boebert, William E., Stockwell, Edward B., Vanyo, Heidi A.
• Primary Examiner(s): Chang, Vivian
• Assistant Examiner(s): Edouard, Patrick N.
• Application Number:
  US08/715,333
• Time in Patent Office: 1,511 Days
• Field of Search: 704/1, 704/3, 704/9, 704/8, 395/575, 395/600, 395/650, 395/200.01, 707/500, 707/513, 382/185
• US Class Current: 704/1
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 51/234 for tracking messages
• CPC Class Codes: H04L 63/0245 Filtering by information in the payload
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

An electronic message filtering system and method is described. A message is received as input to the filter and decomposed into a set of components. The set of components is then processed through a pattern matching algorithm to determine if the message contents contains patterns inherent in a specified pattern, such as a natural language. The results of the pattern match analysis are output by the filter.

Claims

1. A method of detecting if an electronic message is in a particular format, comprising the steps of:

decomposing the message into a set of components according to a predetermined component size; and

applying pattern recognition logic to the set of components as a function of the predetermined component size to determine if the electronic message is predominately natural language text, wherein applying pattern recognition logic includes;

determining an observed phi of the electronic message;

determining a plain-text phi of the electronic message as a function of the component size; and

comparing the observed phi to the plain-text phi to determine if the electronic message includes data in the particular format.

2. The method of claim 1, wherein the step of decomposing the message comprises the step of breaking the message into words.

3. The method of claim 1, wherein the step of decomposing the message comprises the step of breaking the message into N-grams, wherein N equals the component size.

4. The method of claim 3, wherein N=3.

5. The method of claim 1, wherein the step of decomposing the message includes breaking the message into N-grams, wherein N equals the component size and wherein the step of applying pattern recognition logic comprises:

analyzing the frequency of the N-grams with N-gram analysis;

applying a heuristic capable of distinguishing between natural language text and another message format; and

determining, based on the N-gram analysis and application of the heuristic, whether the electronic message is likely to be natural language text.

6. The method of claim 5, wherein the heuristic counts the number of null characters in the electronic message.

7. The method of claim 5, wherein the heuristic calculates a ratio of lowercase to uppercase characters in the electronic message.

8. The method of claim 5, wherein the heuristic calculates a ratio of punctuation to total valid characters in the electronic message.

9. A method of filtering an electronic message to detect a particular format, comprising the steps of:

decomposing the electronic message into a set of components according to a predetermined component size;

determining an observed phi of the electronic message;

determining a plain-text phi of the electronic message as a function of the component size; and

comparing the observed phi to the plain-text phi to determine if the electronic message includes data in the particular format.

10. The method of claim 9, wherein the step of decomposing the message comprises the step of breaking the message into words.

11. The method of claim 9, wherein the step of decomposing the message comprises the step of breaking the message into N-grams, wherein N equals the component size.

12. The method of claim 11, wherein the step of applying pattern recognition logic comprises:

analyzing the frequency of the N-grams with N-gram analysis;

applying a heuristic capable of distinguishing between natural language text and another message format; and

determining, based on the N-gram analysis and application of the heuristic, whether the electronic message is likely to be natural language text.

13. A method of filtering an electronic message, comprising the steps:

receiving an electronic message from an electronic mail structure;

decomposing the message into a set of components according to a predetermined component size;

applying pattern recognition logic to the set of components as a function of the predetermined component size to determine if the electronic message is predominately natural language text, wherein applying pattern recognition logic includes;

determining an observed phi of the electronic message;

determining a plain-text phi of the electronic message as a function of the component size; and

comparing the observed phi to the plain-text phi to determine if the electronic message includes data in the particular format; and

issuing an affirmative response to the electronic mail structure if the electronic message is predominately natural language text.

14. The method of claim 13, wherein the step of decomposing the message comprises breaking the message into N-grams, wherein N equals the component size.

15. The method of claim 14, wherein N=3.

16. The method of claim 13, wherein the step of decomposing the message comprises breaking the message into N-grams, wherein N equals the component size.

17. The method of claim 16, wherein the step of applying pattern recognition logic comprises:

analyzing the frequency of the N-grams with N-gram analysis;

applying a heuristic capable of distinguishing between natural language text and another message format; and

determining, based on the N-gram analysis and application of the heuristic, whether the electronic message is likely to be natural language text.

18. A method of filtering an electronic message to detect a particular format, comprising the steps:

receiving an electronic message from an electronic mail structure;

decomposing the message into a set of components according to a predetermined component size;

determining an observed phi of the electronic message;

determining a plain-text phi of the electronic message as a function of the component size; and

comparing the observed phi to the plain-text phi to determine if the electronic message includes data in the particular format.

19. An electronic message filter, comprising:

a filter initialization module;

a message analysis module, wherein the message analysis module includes;

decomposition means for decomposing electronic messages into a set of components according to a predetermined component size; and

pattern recognition means for determining if the set of components has characteristics expected in an electronic message which is predominately natural language text, wherein the pattern recognition means determines an observed phi of the electronic message and compares the observed phi to a plain-text phi of the electronic message to determine if the electronic message includes data in the particular format, wherein the plain-text phi is determined as a function of the predetermined component size; and

a clean-up module;

wherein the message analysis module determines if the electronic messages should be passed based on whether the electronic messages are predominately natural language text.

20. The electronic message filter of claim 19, wherein the filter initialization module comprises storage means for storing execution parameters for the message analysis module.

21. The electronic message filter of claim 19, wherein the decomposition means is an N-gram decomposition module, wherein N equals the component size.

22. The electronic message filter of claim 19, wherein the pattern recognition means includes means for applying a set of rules based on a heuristic associated with the particular format.

23. The electronic message filter of claim 19, wherein the pattern recognition means includes means for applying a phi-test to the set of components as a function of the component size.

24. A computer program product, comprising:

a computer usable medium having a computer readable code means embodied therein for causing electronic messages to be analyzed as to whether the contents of each electronic message comprises a natural language, the computer readable program code means in said computer program product comprising;

computer readable program code means for causing a computer to read a first electronic message;

computer readable program code means for causing a computer to decompose the first electronic message into components according to a predetermined component size; and

computer readable program code means for causing a computer to perform a pattern analysis of the components to identify whether the first electronic message is predominately natural language text, wherein the program code means causes the computer to perform a pattern analysis of the components in order to determine an observed phi of the electronic message and compare the observed phi to a plain-text phi of the electronic message to determine if the electronic message includes data in the particular format, wherein the plain-text phi is determined as a function of the predetermined component size.

25. The computer program product of claim 24, wherein the program code for causing a computer to decompose the first electronic message includes program code for breaking the message into N-grams, wherein N equals the component size and wherein the program code for causing a computer to perform a pattern analysis includes:

program code for analyzing the frequency of the N-grams with N-gram analysis;

program code for applying a heuristic capable of distinguishing between natural language text and another message format; and

program code for determining, based on the N-gram analysis and application of the heuristic, whether the electronic message is likely to be natural language text.