System and method for dynamically evaluating latent concepts in unstructured documents

• US 7,313,556 B2
• Filed: 12/14/2005
• Issued: 12/25/2007
• Est. Priority Date: 08/31/2001
• Status: Expired due to Term

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: Attenex Corp. (FTI Consulting Incorporated)
• Inventors: Gallivan, Dan, Kawai, Kenji
• Primary Examiner(s): Breene, John
• Assistant Examiner(s): Ly, Anh
• Application Number:
  US11/304,406
• Publication Number:
  US 20060089947A1
• Time in Patent Office: 741 Days
• Field of Search: 707/102, 707/6, 707/100, 707/101, 707/104.1, 715/530, 715/531, 704/9, 704/10, 706/45
• US Class Current: 1/1
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system and method for dynamically evaluating latent concepts in unstructured documents is disclosed. A multiplicity of concepts are extracted from a set of unstructured documents into a lexicon. The lexicon uniquely identifies each concept and a frequency of occurrence. A frequency of occurrence representation is created for the documents set. The frequency representation provides an ordered corpus of the frequencies of occurrence of each concept. A subset of concepts is selected from the frequency of occurrence representation filtered against a pre-defined threshold. A group of weighted clusters of concepts selected from the concepts subset is generated. A matrix of best fit approximations is determined for each document weighted against each group of weighted clusters of concepts.

Claims

1. A computer-readable storage medium storing computer executable program code to be executed by a computer system, the computer executable program code comprising the method of:

storing data on a computer-readable storage medium, comprising;

tabulating frequencies of occurrence of terms from within a set of documents;

correlating and selecting two or more of the terms to generate weighted groups of themes based on at least one of upper and lower edge conditions on the frequencies of occurrence;

determining an inner product for each theme as a function of the frequencies of occurrence and the weighted groups of themes; and

assigning each document into a set of clusters based on the inner product of the theme comprising those terms from the document; and

accessing code for the data stored on the computer-readable storage medium.

2. The storage medium according to claim 1, wherein the semantic relatedness of those terms collectively comprised within a theme is bounded.

3. The storage medium according to claim 1, further comprising:

multidimensional data that models each term, wherein each term corresponds to a different dimension within a logically-defined concept space.

4. The storage medium according to claim 1, wherein the upper and lower limits are set in a range substantially comprising from one percent to fifteen percent where the documents comprise email.

5. A system for identifying clustered groups of semantically-related documents, comprising:

a memory, comprising;

frequencies of occurrence of terms tabulated from within a set of documents; and

weighted groups of themes that include two or more of the terms, which are correlated and selected to generate the themes based on at least one of upper and lower edge conditions on the frequencies of occurrence; and

a processor, comprising;

a text analyzer to determine an inner product for each theme as a function of the frequencies of occurrence and the weighted groups of themes; and

a visualization module to assign each document into a cluster based on the inner product of the theme comprising those terms from the document.

6. The system according to claim 5, wherein the semantic relatedness of each theme is iteratively redetermined, and each document is reassigned into the cluster based on the theme comprising those terms from the document that reflect a most-recently determined strongest semantic relatedness.

7. The system according to claim 5, wherein, for the set of n documents, the semantic relatedness s_r is determined as an inner product of the frequencies of occurrence and group weights, using the following equation:

8. The system according to claim 5, wherein the semantic relatedness of those terms collectively comprised within a theme is bounded.

9. The system according to claim 5, further comprising:

a two-dimensional visualization of the themes to render each cluster for output by semantic relatedness.

10. The system according to claim 5, further comprising:

multidimensional data to model each term, wherein each term corresponds to a different dimension within a logically-defined concept space.

11. The system according to claim 5, wherein the upper and lower limits are set in a range substantially comprising from one percent to fifteen percent where the documents comprise email.

12. A computer-readable storage medium storing computer executable program code to be executed by a computer system, the computer executable program code comprising the method of:

collectively organizing structured data as a database record, comprising;

tabulating frequencies of occurrence of terms from within a set of documents;

correlating and selecting two or more of the terms to generate weighted groups of themes based on at least one of upper and lower edge conditions on the frequencies of occurrence;

determining an inner product for each theme as a function of the frequencies of occurrence and the weighted groups of themes; and

assigning each document into a cluster based on the inner product of the theme comprising those terms from the document; and

accessing code for the structured data of the database record on the computer-readable storage medium.

13. The storage medium according to claim 12, wherein the semantic relatedness of each theme is iteratively redetermined, and each document is reassigned into the cluster based on the theme comprising those terms from the document that reflect a most-recently determined strongest semantic relatedness.

14. The storage medium according to claim 12, wherein each cluster is rendered for output within a two-dimensional visualization of the themes by semantic relatedness.

15. The storage medium according to claim 12, wherein the upper and lower limits are set in a range substantially comprising from one percent to fifteen percent where the documents comprise email.

16. A method for identifying clustered groups of semantically-related documents, comprising:

tabulating frequencies of occurrence of terms within a set of documents;

correlating weighted groups of themes including two or more of the terms;

selecting the terms to generate the themes based on at least one of upper and lower edge conditions on the frequencies of occurrence;

determining an inner product for each theme as a function of the frequencies of occurrence and the weighted groups of themes; and

assigning each document into a cluster based on the inner product of the theme comprising those terms from the document.

17. The method according to claim 16, further comprising;

iteratively redetermining the semantic relatedness of each theme; and

reassigning each document into the cluster based on the theme comprising those terms from the document that reflect a most-recently determined strongest semantic relatedness.

18. The method according to claim 16, wherein, for the set of n documents, the semantic relatedness s_r is determined as an inner product of the frequencies of occurrence and group weights, using the following equation:

19. The method according to claim 16, further comprising:

bounding the semantic relatedness of those terms collectively comprised within a theme.

20. The method according to claim 16, further comprising:

rendering each cluster for output within a two-dimensional visualization of the themes by semantic relatedness.

21. The method according to claim 16, further comprising:

modeling each term as multidimensional data, wherein each term corresponds to a different dimension within a logically-defined concept space.

22. The method according to claim 16, wherein the upper and lower limits are set in a range substantially comprising from one percent to fifteen percent where the documents comprise email.

23. A system for identifying clustered groups of semantically-related documents, comprising:

a set of stored documents that each comprise at least one term, wherein each term corresponds to a different dimension within a logically-defined multi-dimensional concept space;

a database comprising a memory to store records that each comprises;

frequencies of occurrence of terms tabulated from within a set of documents; and

weighted groups of themes that include two or more of the terms, which are correlated and selected to generate the themes based on at least one of upper and lower edge conditions on the frequencies of occurrence;

a processor, comprising;

a text analyzer to determine an inner product for each theme as a function of the frequencies of occurrence and the weighted groups of themes; and

a visualization module to assign each document into a cluster based on the inner product of the theme comprising those terms from the document; and

a display system to render each cluster for output as a two-dimensional visualization of the themes by semantic relatedness.

24. The system according to claim 23, wherein the semantic relatedness of each theme is iteratively redetermined, and each document is reassigned into the cluster based on the theme comprising those terms from the document that reflect a most-recently determined strongest semantic relatedness.

25. The system according to claim 23, wherein the semantic relatedness of those terms collectively comprised within a theme is bounded.

26. The system according to claim 23, wherein the upper and lower limits are set in a range substantially comprising from one percent to fifteen percent where the documents comprise email.