AUTOMATIC LEXICON GENERATION SYSTEM FOR DETECTION OF SUSPICIOUS E-MAILS FROM A MAIL ARCHIVE

• US 20100057720A1
• Filed: 03/16/2009
• Published: 03/04/2010
• Est. Priority Date: 08/26/2008
• Status: Active Grant

• Current Assignee: Manipal Technologies Limited (Manipal Group)
• Original Assignee: Saraansh Software Solutions Pvt Ltd.
• Inventors: Panakkal, Sajay S., Hedge, Dinesh R., Naik, Vinay M., Kalyan, Malathi, Singh, Ram Lokendar, Dattasharma, Abhi
• Granted Patent: US 8,321,204 B2
• Time in Patent Office: Days
• US Class Current: N/A
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification

Abstract

A system for generating a lexicon of words, organized into weighted categories, from a user defined set of example documents for detecting suspicious e-mails from a mail archive is provided. The system uses a set of example documents and e-mails given by the user to probabilistically find possible lists of critical words. The obtained list is now applied on an archive of e-mails. The system generates an inverted index on the mails from the archive to facilitate search for the key phrases. User feedback is taken on the results obtained and corrections in the lexicon made if necessary. Thus, the mails are scanned based on user feedback, user defined words and automatically generated word list. These lists constantly adapt as e-mails in the archive change. The system then combines all these to present the user with several possible sets of keywords and their relative importance that can be used as a policy for a desired level of accuracy. The system also shows the user any change if the set is modified. Finally, the system searches through the entire mail archive to find suspicious e-mails.

Claims

1. An automatic lexicon generation system to identify and construct a list of English phrases from a user specified set of example e-mails and documents written in English, said phrases being a set of relevant key phrases useful for identifying information leak in an archive of e-mails, said system comprises:

a) means (102) to identify a set of important key phrases from a user specified set of example e-mails leaking information and documents leaking information, written in English, using frequency analysis, word stemming, and removal of common words and domain specific words (FIG. 1, FIG. 4);

b) means (102, 506, 511) to identify a set of important key phrases from a user specified set of example e-mails not leaking information and documents not leaking information, written in English, using frequency analysis, word stemming, and removal of common words (FIG. 1, FIG. 5);

c) means (405, 511) to identify a set of relevant phrases and to assign a label, one of “

very highly sensitive”

, “

highly sensitive”

, “

sensitive”

, “

not sensitive”

or “

sensitive”

to each of the phrases of said set (FIG. 4, FIG. 5);

(d) means (613) for assigning weights to each of said key phrases (FIG. 6);

e) means (614) for building multiple key phrase lists and weights from said important key phrases (FIG. 6);

f) means (615) for presenting said lists of key phrases to the user for simulation and for storing the final approved list as weighted category lexicon (FIG. 6)

g) means (716, 717) for using said list of phrases on an archive of e-mails and documents written in English for identifying any e-mail leaking information (FIG. 7).

2. The system as claimed in claim 1, wherein said means of identifying important key phrases from a user specified set of documents leaking information and e-mails leaking information and documents not leaking information and e-mails not leaking information written in English further comprises:

a) means (400) of accepting a set of English documents from the user (FIG. 4, STEP 0);

b) means (400, 401) of accepting some parts of the documents as critical or not critical (FIG. 4, STEP 0, STEP 1);

c) means (506) of accepting a set of English e-mails from the user (FIG. 5, STEP 6);

d) means (506, 507) of accepting some parts of the e-mails as critical or not critical (FIG. 5, STEP 6, STEP 7);

e) means (402, 403, 508, 509) of rejecting common words (FIG. 4, STEP 2 and STEP 3, FIG. 5, STEP 8 and STEP 9);

f) means (402, 403, 508, 509) for stemming words (FIG. 4, STEP 2 and STEP 3, FIG. 5, STEP 8 and STEP 9);

g) means (404, 510) for creating a frequency table of the words (FIG. 4, STEP 4, FIG. 5, STEP 10); and

h) means (405, 511) for identifying words from said frequency table which occur with higher frequency only in documents or e-mails leaking information and with higher frequency only in documents or e-mails not leaking information and assigning a label, one of “

very highly sensitive”

, “

highly sensitive”

, “

sensitive”

, “

not sensitive”

or “

safe”

to said identified words (FIG. 4, STEP 5 and FIG. 5, STEP 11).

3. The system as claimed in claim 1, wherein said means of assigning weights to identified important key phrases further comprises:

a) means (613) for finding ratios of frequencies of said important key phrases and classifying them according to some preset norms (FIG. 6, STEP 13);

b) means (613) for sorting said important key phrases according to said ratios (FIG. 6, STEP 13); and

c) means (613) for assigning weights proportional to the ratios spread over a pre-defined range (FIG. 6, STEP 13).

4. The system as claimed in claim 1, wherein said means of finding multiple key phrase list and weights further comprises:

a) means (614) for merging the key phrases and weights obtained in steps 5, 11, 13 and 14 Into alternate lists (FIG. 6, STEP 14).

5. The system as claimed in claim 1, wherein said means of presenting the lists to the user for simulation further comprises:

a) means (615) for displaying the lists to the user together with controls so that the user can modify, delete, change or add to the lists at the user'"'"'s discretion and can simulate the effect of such lists on a pre-defined set of mails and the final approved list can be stored as weighted category lexicon (FIG. 6, STEP 15).

6. The system of claim 1, wherein said means of identifying e-mails leaking information from an archive of e-mail and documents written in English further comprises:

a) means (716) for scanning the e-mails and documents in the archive for the phrases in said list of important phrases (FIG. 7, STEP 16);

b) means (716) for building an inverted index for each of the phrases for each of the documents and e-mails (FIG. 7, STEP 16); and

c) means (717) for classifying an e-mail as leaking information if said score crosses a threshold (FIG. 7, STEP 17).