System and method for processing message threads

• US 8,108,397 B2
• Filed: 11/15/2010
• Issued: 01/31/2012
• Est. Priority Date: 03/19/2001
• Status: Expired due to Term

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: FTI Consulting Technology LLC (FTI Consulting Incorporated)
• Inventors: Kawai, Kenji, McDonald, David T.
• Primary Examiner(s): NGUYEN, PHONG H
• Application Number:
  US12/946,797
• Publication Number:
  US 20110067037A1
• Time in Patent Office: 442 Days
• Field of Search: 707/626, 707/736, 707/737, 707/822
• US Class Current: 707/737
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 51/08 Annexed information, e.g. attachments
• CPC Class Codes: H04L 51/216 Handling conversation history, e.g. grouping of messages in sessions or threads
• CPC Class Codes: H04L 51/42 Mailbox-related aspects, e.g. synchronisation of mailboxes
• CPC Class Codes: Y10S 707/99937 Sorting
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A system and method for processing message threads is provided. A plurality of messages, each comprising a message body, is grouped by conversation thread. The message bodies of the messages are compared. Each message recursively contained in at least one other message is identified as a near duplicate message. An attachment sequence is generated for at least part of each attachment associated with one or more of the messages. The attachment sequences associated with the near duplicate messages are compared. Each near duplicate message having an attachment sequence not matching the attachment sequence of any other near duplicate message is identified as a unique message.

Claims

1. A system for processing message threads, comprising:

a thread module to group a plurality of messages, each comprising a message body, by conversation thread;

a message body comparer module to compare the message bodies of the messages;

a near duplicate module to identify each message body recursively contained in at least one other message as a near duplicate message and to identify each message body matching at least one other message as an exact duplicate message;

an attachment module to generate an attachment sequence for at least part of each attachment associated with one or more of the messages;

a message sequence module to generate a message sequence for each of the messages;

a concatenator module to concatenate the message sequence and the attachment sequence into a compound digest for each message;

a digest comparer module to compare the compound digests of each message;

marker module to mark each near duplicate message and exact duplicate message having a compound digest not matching the compound digest of any other message as a unique message; and

a processor to execute each of the modules, which are stored on a computer-readable storage medium.

2. The system according to claim 1, further comprising:

a unique message module to identify each message not recursively contained in at least one other message as a unique message.

3. The system according to claim 1, further comprising:

a message sequence comparer module to compare the message sequences of the messages; and

an exact duplicate module to identify each message having a message sequence matching the message sequence of at least one other message as an exact duplicate message.

4. The system according to claim 3, further comprising:

a unique message sequence module to identify each message having a message sequence not matching the message sequence of any other message as a unique message.

5. The system according to claim 1, further comprising:

a sorter to sorting the messages in each conversation thread by one of ascending length order and descending length order.

6. The system according to claim 5, further comprising:

a length module to determine length order based on thread markers comprising at least one of keywords, delimiter strings, and relative location within each message.

7. The system according to claim 1, further comprising:

a log to identify a source location of each unique message and cross- referencing the exact duplicate messages and near duplicate messages associated with each unique message.

8. The system according to claim 1, wherein the attachment sequences comprise hash code digests of the attachments attached to each message.

9. The system according to claim 1, further comprising:

a shadow store to store the unique messages.

10. The system according to claim 1, further comprising:

a duplicate attachment module to mark each message having an attachment sequence matching at least one other near duplicate message as a near duplicate message.

11. A method processed by a computer processor for processing message threads, comprising:

grouping a plurality of messages, each comprising a message body, by conversation thread;

comparing the message bodies of the messages;

identifying each message body recursively contained in at least one other message as a near duplicate message and each message body matching at least one other message as an exact duplicate message;

generating an attachment sequence for at least part of each attachment associated with one or more of the messages;

generating a message sequence for each of the messages;

concatenating the message sequence and the attachment sequence into a compound digest for each message;

comparing the compound digest of each message; and

marking each near duplicate message and exact duplicate message having a a compound digest not matching the compound digest of any other message as a unique message.

12. The method according to claim 11, further comprising:

identifying each message not recursively contained in at least one other message as a unique message.

13. The method according to claim 11, further comprising:

comparing the message sequences of the messages; and

identifying each message having a message sequence matching the message sequence of at least one other message as an exact duplicate message.

14. The method according to claim 13, further comprising:

identifying each message having a message sequence not matching the message sequence of any other message as a unique message.

15. The method according to claim 11, further comprising:

sorting the messages in each conversation thread by one of ascending length order and descending length order.

16. The method according to claim 15, further comprising:

determining length order based on thread markers comprising at least one of keywords, delimiter strings, and relative location within each message.

17. The method according to claim 11, further comprising:

maintaining a log identifying a source location of each unique message and cross-referencing the exact duplicate messages and near duplicate messages associated with each unique message.

18. The method according to claim 11, wherein the attachment sequences comprise hash code digests of the attachments attached to each message.

19. The method according to claim 11, further comprising:

storing the unique messages in a shadow store.

20. The method according to claim 11, further comprising:

marking each message having an attachment sequence matching at least one other near duplicate message as a near duplicate message.