System and method for evaluating a structured message store for message redundancy

• US 7,035,876 B2
• Filed: 10/13/2004
• Issued: 04/25/2006
• Est. Priority Date: 03/19/2001
• Status: Expired due to Term

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: Attenex Corp. (FTI Consulting Incorporated)
• Inventors: Kawai, Kenji, McDonald, David T.
• Primary Examiner(s): AMSBURY, WAYNE P
• Application Number:
  US10/965,564
• Publication Number:
  US 20050055359A1
• Time in Patent Office: 559 Days
• Field of Search: 707/104.1, 707/102, 709/201
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 51/08 Annexed information, e.g. attachments
• CPC Class Codes: H04L 51/216 Handling conversation history, e.g. grouping of messages in sessions or threads
• CPC Class Codes: H04L 51/42 Mailbox-related aspects, e.g. synchronisation of mailboxes
• CPC Class Codes: Y10S 707/99937 Sorting
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A system and method for evaluating a structured message store for message redundancy is described. A header and a message body are extracted from each of a plurality of messages maintained in a structured message store. A substantially unique hash code is calculated over at least part of the header and over the message body of each message. The messages are grouped by the hash codes. One such message is identified as a unique message within each group. In a further embodiment, the messages are grouped by conversation thread. The message body for each message within each conversation thread group is compared. At least one such message within each conversation thread group is identified as a unique message.

Claims

1. A computer-readable medium encoded with stored message records for redundancy processing, comprising:

hash codes that are each determined from at least part of a message header plus a message body for each of a plurality of electronic messages; and

metadata to store the electronic messages with matching hash codes into groups and to identify one such electronic message within each group as unique;

further hash codes that are each determined from one or more attachment of at least one such electronic message; and

metadata to identify the electronic messages within each group with matching further hash codes as exact duplicates.

2. A computer-readable medium according to claim 1, further comprising:

metadata to identify the electronic message in a group comprising a single electronic message as unique.

3. A computer-readable medium according to claim 1, further comprising:

metadata to identify one such electronic message out of the electronic messages within a length-ordered conversation group having matching longest lengths as unique.

4. A computer-readable medium according to claim 3, further comprising:

metadata to identify each such electronic message within each length-ordered conversation group having a subset relationship to those electronic messages having matching longest lengths as near duplicate.

5. A computer-readable medium according to claim 1, wherein the metadata further comprises content extracted from fields of each electronic message.

6. A client system, for storing electronic messages for redundancy processing, comprising:

a plurality of electronic messages that each comprise a message header plus a message body, at least part of which is used to determine a hash code for each electronic message; and

an interface providing access to the electronic messages, wherein the electronic messages with matching hash codes are assigned into groups and one such electronic message within each group is identified as unique, wherein at least one such electronic message further comprises one or more attachments, at least part of which is used to determine a further hash code and the electronic messages within each group with matching further hash codes are identified as exact duplicates.

7. A client system according to claim 6, wherein the electronic message in a group comprising a single electronic message is identified as unique.

8. A client system according to claim 6, wherein at least one such electronic message assigned into a length-ordered conversation group having matching longest lengths is identified as unique.

9. A client system according to claim 8, wherein each such electronic message within each length-ordered conversation group having a subset relationship to those electronic messages having matching longest lengths is identified as near duplicate.

10. A client system according to claim 6, wherein the hash code is generated with a fixed length, independent of content length.

11. An analysis system processing electronic messages for redundancy, comprising:

a plurality of electronic messages that each comprise a message header plus a message body wherein at least one such electronic message further comprises one or more attachments; and

a processor configured to determine hash codes from at least a part of each electronic message header plus message body and to assign the electronic messages with matching hash codes into groups, wherein one such electronic message within each group is identified as unique and to determine further hash codes from each attachment, the electronic messages within each group with matching further hash codes are identified as exact duplicates.

12. An analysis system according to claim 11, wherein the electronic message in a group comprising a single electronic message is identified as unique.

13. An analysis system according to claim 11, wherein the processor is further configured to assign the electronic messages in a conversation thread into conversation groups and to order the electronic messages within each conversation group by length, one such electronic message out of those electronic messages within each conversation group having matching longest lengths is identified as unique.

14. An analysis system according to claim 13, wherein each such electronic message within each conversation group having a subset relationship to those electronic messages having matching longest lengths is identified as near duplicate.

15. An analysis system according to claim 13, wherein at least one such electronic message further comprises one or more attachments, and the processor is further configured to determine further hash codes from each attachment, each such electronic message within each conversation group having non-matching attachment hash codes is identified as unique.

16. An analysis system according to claim 13, wherein the hash code is generated with a fixed length, independent of content length.

17. An analysis system according to claim 11, wherein the hash codes are determined using a one-way function.

18. A computer-readable medium encoded with structured records storing electronic messages for redundancy processing, comprising:

structured data for a plurality of electronic messages, comprising at least one of;

a group of electronic messages with matching hash codes; and

a unique electronic message identified within each group; and

wherein the hash codes are determined from at least part of a message header plus a message body for each electronic message and the structured data further comprises one or more attachments with further hash codes for each attachment, wherein the electronic messages within each group with matching further hash codes are identified as exact duplicates.

19. A computer-readable medium according to claim 18, wherein the unique electronic message is the only electronic message within the group.

20. A computer-readable medium according to claim 18, wherein the structured data further comprises a conversation group of electronic messages in a conversation thread, wherein the electronic messages within each conversation group by length are ordered by length and one such electronic message out of those electronic messages within each conversation group having matching longest lengths is identified as unique.

21. A process for processing electronic messages for redundancy, comprising:

accessing a plurality of electronic messages that each comprise a message header plus a message body, wherein at least one such electronic message further comprises one or more attachments;

determining hash codes from at least a part of each message header plus message body;

assigning the electronic messages with matching hash codes into groups and identifying one such electronic message within each group as unique;

determining further hash codes from each attachment; and

identifying the electronic messages within each group with matching further hash codes as exact duplicates.

22. A process according to claim 21, further comprising:

identifying the electronic message in a group comprising a single electronic message as unique.

23. A process according to claim 21, further comprising:

assigning the electronic messages in a conversation thread into conversation groups;

ordering the electronic messages within each conversation group by length; and

identifying one such electronic message out of those electronic messages within each conversation group having matching longest lengths as unique.

24. A process according to claim 23, further comprising:

identifying each such electronic message within each conversation group having a subset relationship to those electronic messages having matching longest lengths as near duplicate.

25. A process according to claim 23, wherein at least one such electronic message further comprises one or more attachments, further comprising:

determining further hash codes from each attachment; and

identifying each such electronic message within each conversation group having non-matching attachment hash codes as unique.

26. A process according to claim 21, wherein the hash code is generated with a fixed length, independent of content length.

27. A process according to claim 21, wherein the hash codes are determined using a one-way function.

28. A computer-readable storage medium holding code for performing the process according to claim 21.