Computer-implemented system and method for identifying relevant documents for display

• US 9,195,399 B2
• Filed: 05/12/2014
• Issued: 11/24/2015
• Est. Priority Date: 08/31/2001
• Status: Expired due to Term

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: FTI Consulting Technology LLC (FTI Consulting Incorporated)
• Inventors: Gallivan, Dan, Kawai, Kenji
• Primary Examiner(s): Spieler, William
• Application Number:
  US14/275,802
• Publication Number:
  US 20140250087A1
• Time in Patent Office: 561 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A computer-implemented system and method for identifying relevant documents for display are provided. Themes for a set of documents are generated. The documents are clustered based on the themes. A matrix including an inner product of document frequency occurrences and cluster concept weightings for each theme is generated for the documents. From the matrix, documents most relevant to a particular theme are identified, and the relevant documents are displayed.

Claims

1. A computer-implemented system for identifying relevant documents for display, comprising:

themes for a set of documents;

an extraction module to extract noun phrases from the documents as concepts;

a theme generator to group two or more of the concepts as one such theme;

a frequency table that identifies each of the concepts and a frequency of occurrence of each concept within each of the documents in the set;

a graph generator to generate a graph of the concepts, comprising;

an x-axis of the graph defining the concepts;

a y-axis of the graph defining a number of the documents that reference each concept; and

a mapping module to map the concepts on the graph in order of descending number of referring documents;

a cluster module to cluster the documents based on the themes;

a matrix for the documents comprising an inner product of document frequency occurrences and cluster concept weightings for each theme;

an identification module to identify from the matrix, documents most relevant to a particular theme; and

a display to present the relevant documents.

2. A system according to claim 1, further comprising:

an assignment module to assign an identifier to each of the concepts, wherein each identifier is a monotonically increasing integer value.

3. A system according to claim 1, further comprising:

for each document, a histogram comprising a normalized representation of the frequencies of occurrence for those concepts extracted from one such document.

4. A system according to claim 1, further comprising:

a value determination module to identify a median value of the frequencies of occurrence along the x-axis;

a threshold module to set edge conditions based on the frequencies of occurrence; and

a selection module to select those documents with concepts that fall within the edge conditions as forming a subset of documents that include latent concepts.

5. A system according to claim 1, further comprising:

a cluster mapping module to map one such document to one of the clusters based on a similarity of the document to the documents in the cluster, wherein the distance is quantified by a distance.

6. A system according to claim 5, further comprising:

a distance determination module to calculate the distance using the following equation;

7. A system according to claim 1, further comprising:

a removal module to remove duplicates of the relevant documents prior to display.

8. A computer-implemented method for identifying relevant documents for display, comprising:

generating themes for a set of documents, comprising;

extracting noun phrases from the documents as concepts; and

grouping two or more of the concepts as one such theme;

generating a frequency table that identifies each of the concepts and a frequency of occurrence of each concept within each of the documents in the set;

generating a graph of the concepts, comprising;

defining an x-axis of the graph as the concepts;

defining a y-axis of the graph as a number of the documents that reference each of the concepts; and

mapping the concepts on the graph in order of descending number of referring documents;

clustering the documents based on the themes;

generating a matrix for the documents comprising an inner product of document frequency occurrences and cluster concept weightings for each theme;

identifying from the matrix, documents most relevant to a particular theme; and

displaying the relevant documents.

9. A method according to claim 8, further comprising:

assigning an identifier to each of the concepts, wherein each identifier is a monotonically increasing integer value.

10. A method according to claim 8, further comprising:

generating for each document, a histogram comprising a normalized representation of the frequencies of occurrence for those concepts extracted from one such document.

11. A method according to claim 8, further comprising:

identifying a median value of the frequencies of occurrence along the x-axis;

setting edge conditions based on the frequencies of occurrence; and

selecting those documents with concepts that fall within the edge conditions as forming a subset of documents that include latent concepts.

12. A method according to claim 8, further comprising:

mapping one such document to one of the clusters based on a similarity of the document to the documents in the cluster, wherein the distance is quantified by a distance.

13. A method according to claim 12, further comprising:

calculating the distance using the following equation;

14. A method according to claim 8, further comprising:

removing duplicates of the relevant documents prior to display.