Computer-implemented system and method for identifying related messages

• US 9,384,250 B2
• Filed: 12/15/2014
• Issued: 07/05/2016
• Est. Priority Date: 03/19/2001
• Status: Expired due to Term

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: FTI Consulting Technology LLC (FTI Consulting Incorporated)
• Inventors: Kawai, Kenji, McDonald, David T.
• Primary Examiner(s): NGUYEN, PHONG H
• Application Number:
  US14/571,282
• Publication Number:
  US 20150100595A1
• Time in Patent Office: 568 Days
• Field of Search: 707/626, 707/664, 707/692, 707/737, 707/752
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 51/08 Annexed information, e.g. attachments
• CPC Class Codes: H04L 51/216 Handling conversation history, e.g. grouping of messages in sessions or threads
• CPC Class Codes: H04L 51/42 Mailbox-related aspects, e.g. synchronisation of mailboxes
• CPC Class Codes: Y10S 707/99937 Sorting
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A system and method for identifying related messages are provided. A set of messages, each having a header, sender and transmission time, is obtained. A message is selected from the set. A body of the selected message is compared to a body of a further message in the set. The further message is labeled as a duplicate of the selected message when the bodies match. The duplicate labeling of the further message is verified when the header, sender, and transmission time of the further message matches the header, sender, and transmission time of the selected message. The duplicate messages are removed from the set. The remaining messages are sorted in order of message length. A shorter message is compared with a longer message and is marked as a near duplicate of the longer message when the body of the shorter message is included in the body of the longer message.

Claims

1. A computer-implemented system for identifying related messages, comprising:

a set of messages, each comprising a header, sender, and transmission time; and

a processor to execute modules for processing the message set, comprising;

a selection module to select one of the messages from the set;

a content comparison module to compare a body of the selected message to a body of a further message in the set;

a labeling module to label the further message as a duplicate of the selected message when the bodies match;

a verification module to verify the duplicate labeling of the further message when the header, sender, and transmission time of the further message matches the header, sender, and transmission time of the selected message;

a removal module to remove the messages with verified duplicate labels;

a sorting module to sort the remaining messages of the set in order of message length;

a length comparison module to compare a shorter message comprising a short text body with a message comprising a longer text body and to determine that the body of the shorter message is included in the body of the longer message;

a marker module to mark the shorter message as a near duplicate of the longer message;

an extractor module to extract metadata from each of the messages in the set;

a compiler to compile the messages of the set into a master array based on the extracted metadata;

a topic module to determine topics of the messages in the master array and to sort the messages in the master array based on the topics;

a unique message comparison module to select another one of the messages and to compare the other selected message with a next message; and

a unique message determination module to determine that the topics of the other selected message and the next message do not match, to determine that the other selected message is a first message of the topic, and to mark the other selected message as a unique message.

2. A system according to claim 1, further comprising:

a database to store the verified duplicate labeling for the further message with an identification of the further message.

3. A system according to claim 1, further comprising:

a length determination module to determine a length of the shorter message and the longer message by searching each remaining message in the set for thread markers, counting an occurrence of each thread marker in each message, and sorting the remaining messages based on the counted occurrences of thread markers.

4. A system according to claim 3, wherein the thread markers comprise at least one of separators and subject line indicators.

5. A system according to claim 1, further comprising:

a message selection module to select a further one of the messages from the master array and to compare the further selected message with a next message; and

a message determination module to determine that the further selected and next messages are from a common conversation thread when the topics match.

6. A system according to claim g claim 1, further comprising:

a message grouping module to group with the unique message, those messages that are labeled as duplicate and near duplicate of the unique message.

7. A system according to claim 1, further comprising:

a logging module to maintain a log of the unique messages and associated duplicate and near duplicate messages.

8. A computer-implemented method for identifying related messages, comprising:

obtaining a set of messages, each comprising a header, sender, and transmission time;

selecting one of the messages from the set;

comparing a body of the selected message to a body of a further message in the set;

labeling the further message as a duplicate of the selected message when the bodies match;

verifying the duplicate labeling of the further message when the header, sender, and transmission time of the further message matches the header, sender, and transmission time of the selected message;

removing the messages with verified duplicate labels;

sorting the remaining messages of the set in order of message length;

comparing a shorter message comprising a short text body with a message comprising a longer text body and determining that the body of the shorter message is included in the body of the longer message;

marking the shorter message as a near duplicate of the longer message;

extracting metadata from each of the messages in the set;

compiling the messages of the set into a master array based on the extracted metadata;

determining topics of the messages in the master array;

sorting the messages in the master array based on the topics;

selecting another one of the messages and comparing the other selected message with a next message;

determining that the topics of the other selected message and the next message do not match;

determining that the other selected message is a first message of the topic; and

marking the other selected message as a unique message.

9. A method according to claim 8, further comprising:

storing the verified duplicate labeling for the further message with an identification of the further message.

10. A method according to claim 8, further comprising:

determining a length of the shorter message and the longer message, comprising;

searching each remaining message in the set for thread markers;

counting an occurrence of each thread marker in each message; and

sorting the remaining messages based on the counted occurrences of thread markers.

11. A method according to claim 10, wherein the thread markers comprise at least one of separators and subject line indicators.

12. A method according to claim 8, further comprising:

selecting a further one of the messages from the master array and comparing the further selected message with a next message; and

determining that the further selected and next messages are from a common conversation thread when the topics match.

13. A method according to claim 8, further comprising:

grouping with the unique message, those messages that are labeled as duplicate and near duplicate of the unique message.

14. A method according to claim 8, further comprising:

maintaining a log of the unique messages and associated duplicate and near duplicate messages.