System and method for displaying relationships between electronically stored information to provide classification suggestions via nearest neighbor

• US 8,572,084 B2
• Filed: 07/09/2010
• Issued: 10/29/2013
• Est. Priority Date: 07/28/2009
• Status: Active Grant

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: FTI Consulting Incorporated
• Inventors: Nussbaum, Nicholas I., Knight, William C.
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): VU, THONG H
• Application Number:
  US12/833,880
• Publication Number:
  US 20110029527A1
• Time in Patent Office: 1,208 Days
• Field of Search: 707/3, 707/728, 707/742, 707/706, 707/791, 707/737, 706/12, 706/14, 706/55, 706/20
• US Class Current: 707/737
• CPC Class Codes: G06F 16/287 Visualization; Browsing
• CPC Class Codes: G06F 16/3322 using system suggestions G06F16/3325 takes precedence
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/358 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/047 Pattern matching networks; Rete networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

A system and for providing reference documents as a suggestion for classifying uncoded documents is provided. Reference electronically stored information items and a set of uncoded electronically stored information items are designated. Each of the reference information items are previously classified. At least one uncoded electronically stored information item is compared with the reference electronically stored information items. One or more of the reference electronically stored information items similar to the at least one uncoded electronically stored information items are identified. Relationships are depicted between the at least one uncoded electronically stored information item and the similar reference electronically stored information items for classifying the at least one uncoded electronically stored information item.

Claims

1. A system for providing reference documents as a suggestion for classifying electronically stored information using nearest neighbor, comprising:

a clustering module to provide a set of uncoded electronically stored information items and a different set of reference electronically stored information items that are each classified with a code;

a similarity module to compare at least one of the uncoded electronically stored information items from the set with the set of reference electronically stored information items and to identify one or more of the reference electronically stored information items that are similar to the at least one uncoded electronically stored information item;

a processing module to process the classification codes associated with the similar reference electronically stored information items, comprising;

a type module to determine a number of different types of the classification codes associated with the similar reference electronically stored information items;

a presence module to determine one or more of a presence and absence of the similar reference electronically stored information items with each type of the different classification codes; and

a quantity module to determine for each type of the classification codes a quantity of the similar reference electronically stored information items;

a suggestion module to display a visual classification suggestion based on at least one of the presence and the absence and the quantity and the number of the types via a display of the at least one uncoded electronically stored information item and the similar reference electronically stored information items;

a receipt module to receive a classification code of one of the types for the at least one uncoded electronically stored information item from a human reviewer based on the suggestion; and

a processor to execute the modules.

2. A system according to claim 1, further comprising:

a reference set module to generate the set of reference electronically stored information items, comprising at least one of;

a comparison module to obtain a set of electronically stored information items, to identify one or more electronically stored information items that are dissimilar from each other electronically stored information item, and to assign a classification code to each of the dissimilar electronically stored information items, as the reference electronically stored information items; and

a reference clustering module to group electronically stored information items for a document review project into one or more clusters, to select one or more of the electronically stored information items in at least one cluster, and to assign a classification code to each of the selected electronically stored information items, as the reference electronically stored information items.

3. A system according to claim 1, further comprising:

a score module to form a score vector for each uncoded electronically stored information item and each reference electronically stored information item; and

the similarity module to calculate a similarity metric by comparing the score vectors for the uncoded electronically stored information items and the reference electronically stored information items.

4. A system according to claim 3, wherein the similarity metric is calculated according to the following equation:

5. A system according to claim 1, further comprising:

a generation module to automatically generate an additional classification code for the at least one of the uncoded electronically stored information item; and

an assignment module to assign one of the automatically generated classification code and the classification code received from the user to the at least one uncoded reference electronically stored information item.

6. A method for providing reference documents as a suggestion for classifying electronically stored information using nearest neighbor, comprising the steps of:

designating a set of uncoded electronically stored information items and a different set of reference electronically stored information items that are each classified with a code;

comparing at least one of the uncoded electronically stored information items from the set with the set of reference electronically stored information items and identifying one or more of the reference electronically stored information items that are similar to the at least one uncoded electronically stored information item;

processing the classification codes associated with the similar reference electronically stored information items, comprising;

determining a number of different types of the classification codes associated with the similar reference electronically stored information items;

determining one or more of a presence and absence of the similar reference electronically stored information items with each type of the different classification codes; and

determining for each type of the classification codes a quantity of the similar reference electronically stored information items;

displaying a visual classification suggestion based on at least one of the presence and the absence and the quantity and the number of the types via a display of the uncoded electronically stored information item and the similar reference electronically stored information items; and

receiving a classification code of one of the types for the at least one uncoded electronically stored information item from a human reviewer based on the suggestion,

wherein the steps are performed by a suitably programmed computer.

7. A method according to claim 6, further comprising:

generating the reference electronically stored information items from a set of electronically stored information items, comprising at least one of;

identifying the electronically stored information items that are dissimilar from each other electronically stored information item and assigning a classification code to each of the dissimilar electronically stored information items, as the reference electronically stored information items; and

grouping a set of electronically stored information items associated with a document review project into one or more clusters, selecting one or more of the electronically stored information items in at least one cluster, and assigning a classification code to each of the selected electronically stored information items, as the reference electronically stored information items.

8. A method according to claim 6, further comprising:

forming a score vector for each uncoded electronically stored information item and each reference electronically stored information item; and

calculating a similarity metric by comparing the score vectors for the uncoded electronically stored information items and the reference electronically stored information items in the reference set.

9. A method according to claim 8, wherein the similarity metric is calculated according to the following equation:

10. A method according to claim 6, further comprising:

automatically generating an additional classification code for the at least one uncoded electronically stored information items; and

assigning one of the generated classification code and the classification code received from the user to the at least one uncoded reference electronically stored information item.

11. A system for identifying reference documents for use in classifying uncoded documents, comprising:

a database to store a set of reference documents that are each classified with a code;

a clustering module to designate a set of clusters each comprising uncoded documents from a different set than the reference documents;

a similarity module to select at least one uncoded document from the set, to compare the at least one uncoded document with each of the reference documents, and to identify one or more reference documents that satisfy a threshold of similarity with the at least one uncoded document;

a processing module to process the classification codes associated with the similar reference documents, comprising;

a type module to determine a number of different types of the classification codes associated with the similar reference documents;

a presence module to determine one or more of a presence and absence of the similar reference documents with each type of the different classification codes; and

a quantity module to determine for each type of the classification codes a quantity of the similar reference documents;

a suggestion module to display a visual classification suggestion based on at least one of the presence and the absence and the quantity and the number of the types via a display of the at least one of the uncoded document and the similar reference documents;

a classification module receiving a classification code of one of the types for the at least one uncoded document from a human reviewer based on the suggestion; and

a processor to execute the modules.

12. A system according to claim 11, further comprising:

a reference set module to generate the set of reference documents, comprising at least one of;

a reference similarity module to obtain a set of documents, to identify the documents that are dissimilar from each other document, and to assign a classification code to each of the dissimilar documents; and

a reference cluster module to generate clusters of documents for a document review project, to select one or more of the documents in at least one of the clusters, and to assign a classification code to each of the documents.

13. A system according to claim 11, wherein the one or more reference documents are selected from at least one of a predefined, customized, or arbitrary reference document set.

14. A system according to claim 11, wherein the similarity module comprises:

a score module to form a score vector for each uncoded document and each reference document; and

a vector similarity module to calculate a similarity metric between the score vectors for the uncoded documents and reference documents.

15. A system according to claim 14, wherein the similarity metric is calculated according to the following equation:

16. A method for identifying reference documents for use in classifying uncoded documents, comprising the steps of:

designating a set of reference documents that are each classified with a code;

designing a set of clusters each comprising uncoded documents from a different set than the reference documents;

selecting at least one uncoded document from the set and comparing the at least one uncoded document with each of the reference documents and identifying one or more reference documents that satisfy a threshold of similarity with the at least one uncoded document;

a processing module to process the classification codes associated with the similar reference documents, comprising;

determining a number of different types of the classification codes associated with the similar reference documents;

determining one or more of a presence and absence of the similar reference documents with each type of the different classification codes; and

determining a quantity of the similar reference documents for each type of the different classification codes;

displaying a visual classification suggestion based on at least one of the presence and the absence and the quantity and the number of the types via a display of the at least one of the uncoded document and the similar reference documents; and

receiving a classification code of one of the types for the at least one uncoded document from a human reviewer based on the suggestion,

wherein the steps are performed by a suitably programmed computer.

17. A method according to claim 16, further comprising:

generating the set of reference documents, comprising at least one of;

obtaining a set of documents, identifying the documents that are dissimilar from each other document, and assigning a classification code to each of the dissimilar documents; and

generating clusters of documents, selecting one or more of the documents in at least one of the clusters and assigning a classification code to each of the documents.

18. A method according to claim 16, wherein the one or more reference documents are selected from at least one of a predefined, customized, or arbitrary reference document set.

19. A method according to claim 16, further comprising:

forming a score vector for each uncoded document and each reference document; and

calculating a similarity metric between the score vectors for the uncoded documents and reference documents.

20. A method according to claim 19, wherein the similarity metric is calculated according to the following equation: