Automatic lexicon generation system for detection of suspicious e-mails from a mail archive

• US 8,321,204 B2
• Filed: 03/16/2009
• Issued: 11/27/2012
• Est. Priority Date: 08/26/2008
• Status: Active Grant

• Current Assignee: Manipal Technologies Limited (Manipal Group)
• Original Assignee: Saraansh Software Solutions Pvt Ltd.
• Inventors: Kalyan, Malathi, Dattasharma, Abhi, Singh, Ram Lokendar, Panakkal, Sanjay Subhash, Hedge, Dinesh Rama, Naik, Vinay Manjunath
• Primary Examiner(s): SKED, MATTHEW J
• Application Number:
  US12/404,770
• Publication Number:
  US 20100057720A1
• Time in Patent Office: 1,352 Days
• Field of Search: None
• US Class Current: 704/10
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification

Abstract

A system for generating a lexicon of words, organized into weighted categories, from a user defined set of example documents for detecting suspicious e-mails from a mail archive is provided. The system uses a set of example documents and e-mails given by the user to probabilistically find possible lists of critical words. The obtained list is now applied on an archive of e-mails. The system generates an inverted index on the mails from the archive to facilitate search for the key phrases. User feedback is taken on the results obtained and corrections in the lexicon made if necessary. Thus, the mails are scanned based on user feedback, user defined words and automatically generated word list. These lists constantly adapt as e-mails in the archive change. The system then combines all these to present the user with several possible sets of keywords and their relative importance that can be used as a policy for a desired level of accuracy. The system also shows the user any change if the set is modified. Finally, the system searches through the entire mail archive to find suspicious e-mails.

Claims

1. An automatic lexicon generation system to identify an information leak in an archive of e-mails, said system comprises:

a database;

a user console; and

a hardware tool in communication with the user console and configured to perform the following;

receive a set of example e-mails leaking information and documents leaking information via the user console wherein, portions of the e-mails leaking information and documents leaking information are identified as critical or not critical;

identify a first set of important key phrases from the set of example e-mails leaking information and documents leaking information using frequency analysis, word stemming, and removal of common words and domain specific words based on the identified critical or not critical portions of the e-mails leaking information and documents leaking information;

receive a set of example e-mails not leaking information and documents not leaking information via the user console;

identify a second set of important key phrases from the set of example e-mails not leaking information and documents not leaking information using frequency analysis, word stemming, and removal of common words;

create a set of relevant phrases based on the first set of important key phrases and the second set of important key phrases;

create a frequency table of the relevant phrases;

identify words from the frequency table which occur with higher frequency in documents or e-mails leaking information and with higher frequency in documents or e-mails not leaking information and assign a label of “

very highly sensitive”

, “

highly sensitive”

, “

sensitive”

, “

not sensitive”

, “

safe”

, or “

sensitive”

to each of the phrases of said set of relevant phrases;

assign weights to each of said relevant phrases;

build multiple key phrase lists and weights from said relevant phrases;

display said multiple key phrase lists on the user console for simulation;

receive approval of said multiple key phrase lists from the user console;

store the approved said multiple key phrase lists as weighted category lexicon in the database; and

apply said multiple key phrase lists on an archive of e-mails and documents for identifying any e-mail leaking information.

2. The system as claimed in claim 1, wherein said assign weights to each of said relevant phrases further comprises:

finding ratios of frequencies of said important key phrases and classifying them according to some preset norms;

sorting said important key phrases according to said ratios; and

assigning weights proportional to the ratios spread over a pre-defined range.

3. The system as claimed in claim 1, wherein said build multiple key phrase list and weights further comprises:

merging the key phrases and weights into alternate lists.

4. The system as claimed in claim 1, wherein said display said multiple key phrase lists on the user console for simulation further comprises:

displaying the lists together with controls so that a user can modify, delete, change or add to the lists at the user'"'"'s discretion and can simulate the effect of such lists on a pre-defined set of mails and the final approved list can be stored as weighted category lexicon.

5. The system of claim 1, wherein said identifying e-mails leaking information from the archive of e-mail and documents further comprises:

scanning the e-mails and documents in the archive for phrases in said multiple key phrase lists;

building an inverted index for each of the phrases for each of the documents and e-mails; and

classifying an e-mail as leaking information if said score crosses a threshold.

6. A method of detecting suspicious documents from an archive comprising:

receiving via a user console a plurality of example e-mails leaking information and documents leaking information via the user console wherein, portions of the e-mails leaking information and documents leaking information are indicated as critical or not critical;

identifying, by a hardware tool, a first set of important key phrases from the plurality of example e-mails leaking information and documents leaking information using frequency analysis, word stemming, and removal of common words and domain specific words, wherein the identifying is based on the critical or not critical indications of the e-mails leaking information and documents;

receiving a plurality of example e-mails not leaking information and documents not leaking information via the user console;

identifying a second set of important key phrases from the plurality of example e-mails not leaking information and documents not leaking information using frequency analysis, word stemming, and removal of common words by the hardware tool;

creating a set of relevant phrases based on the first set of important key phrases and the second set of important key phrases by the hardware tool;

creating a frequency table of the relevant phrases by the hardware tool;

identifying words from the frequency table which occur with higher frequency in documents or e-mails leaking information and with higher frequency in documents or e-mails not leaking information by the hardware tool;

assigning one of “

very highly sensitive,”

“

highly sensitive,”

“

sensitive,”

“

not sensitive,”

“

safe,”

or “

sensitive”

to the phrases of the set of relevant phrases based on the words from the frequency table which occur with higher frequency in documents or e-mails leaking information and with higher frequency in documents or e-mails not leaking information by the hardware tool;

assigning weights to the relevant phrases after assigning a label by the hardware tool;

building multiple key phrase lists and weights from the relevant phrases by the hardware tool;

displaying the multiple key phrase lists on the user console;

receiving from the user console an indication of at least one of a modification, a deletion, or an addition to the multiple key phrase lists and creating modified multiple key phrase lists;

simulating the effect of the indication of at least one of the modification, the deletion, or the addition to the multiple key phrase lists on a pre-defined set of emails by the hardware tool;

receiving approval of the indication from the user console;

modifying the multiple key phrase lists based on the received indication from the user console;

storing the multiple key phrase lists as a weighted category lexicon by the hardware tool; and

applying the multiple key phrase lists on an archive of e-mails and documents for identifying any e-mail leaking information by the hardware tool.