System And Method For Generating A Reference Set For Use During Document Review

• US 20110047156A1
• Filed: 08/24/2010
• Published: 02/24/2011
• Est. Priority Date: 08/24/2009
• Status: Active Grant

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: FTI Consulting Incorporated
• Inventors: Conwell, John, McNee, Sean M., Knight, William C.
• Granted Patent: US 8,612,446 B2
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

A system and method for providing generating reference sets for use during document review is provided. A collection of unclassified documents is obtained. Selection criteria are applied to the document collection and those unclassified documents that satisfy the selection criteria are selected as reference set candidates. A classification code is assigned to each reference set candidate. A reference set is formed from the classified reference set candidates. The reference set is quality controlled and shared between one or more users.

Claims

1. A method for generating a reference set for use during document review, comprising:

obtaining a collection of unclassified documents;

applying selection criteria to the collection and selecting those unclassified documents that satisfy the selection criteria as reference set candidates;

assigning a classification code to each reference set candidate; and

forming a reference set from the classified reference set candidates, wherein the reference set comprises coded documents that are quality controlled and shared between one or more reviewers.

2. A method according to claim 1, further comprising:

refining the reference set by reducing a number of reference set candidates included in the reference set.

3. A method according to claim 1, wherein the selection criteria comprises random sampling, further comprising:

randomly ordering the unclassified documents into a list; and

selecting a predetermined number of the unclassified documents from the list as the reference set candidates.

4. A method according to claim 1, wherein the selection criteria comprises random sampling, further comprising:

assigning document identification values to each of the unclassified documents;

selecting one or more of the document identification values; and

identifying the unclassified documents associated with the selected document identification values as the reference set candidates.

5. A system for generating a reference set for use during document review, comprising:

a collection of unclassified documents;

a selection module to apply selection criteria to the collection and to select those unclassified documents that satisfy the selection criteria as reference set candidates;

a classification module to assign a classification code to each reference set candidate; and

a data module to form a reference set from the classified reference set candidates, wherein the reference set comprises coded document that are quality controlled and shared between one or more reviewers.

6. A system according to claim 5, further comprising:

refining the reference set by reducing a number of reference set candidates included in the reference set.

7. A system according to claim 5, wherein the selection criteria comprises random sampling, further comprising:

randomly ordering the unclassified documents into a list; and

selecting a predetermined number of the unclassified documents from the list as the reference set candidates.

8. A system according to claim 5, wherein the selection criteria comprises random sampling, further comprising:

assigning document identification values to each of the unclassified documents;

selecting one or more of the document identification values; and

identifying the unclassified documents associated with the selected document identification values as the reference set candidates.

9. A method for generating a reference set via clustering, comprising:

obtaining a collection of documents;

grouping the documents into clusters of documents;

selecting one or more documents from at least one cluster as reference set candidates;

assigning a classification code to each of the reference set candidates; and

grouping the classified reference set candidates as the reference set.

10. A method according to claim 9, further comprising:

building a hierarchical tree of the clusters; and

traversing the hierarchical tree to identify the reference set candidates.

11. method according to claim 9, further comprising:

applying a size threshold to the reference set candidates; and

selecting the reference set candidates for inclusion in the reference set when the size threshold is satisfied.

12. A method according to claim 9, further comprising:

applying a size threshold to the reference set candidates; and

reclustering the reference set candidates when the size threshold is not satisfied until the size threshold is satisfied.

13. A method for generating a reference set via seed documents, comprising:

obtaining a collection of documents;

identifying one or more seed documents;

comparing the seed documents to the document collection and identifying those documents similar to the seed documents as reference set candidates;

applying a size threshold to the reference set candidates; and

grouping the reference set candidates as the reference set when the size threshold is satisfied.

14. A method according to claim 13, further comprising:

refining the reference set by reducing a number of reference set candidates included in the reference set.

15. A method according to claim 13, further comprising:

selecting the seed documents from at least one of a current document set and a previously defined document set.

16. A method according to claim 13, further comprising:

determining the seed documents using a keyword search.

17. A method for generating a training set for use during document review, comprising:

assigning classification codes to a set of documents;

receiving further classification codes assigned to the same set of documents;

comparing the classification code for at least one document with the further classification code for that document;

determining whether a disagreement exists between the assigned classification code and the further classification code for at least one document;

identifying those documents with disagreeing classification codes as training set candidates;

applying a stop threshold to the training set candidates; and

grouping the training set candidates as a training set when the stop threshold is satisfied.

18. A method according to claim 17, further comprising:

assigning further classification codes to the documents for which a disagreement exists.

19. A method according to claim 17, wherein the classification codes are assigned by a machine and the further classification codes are assigned by a reviewer.

20. A method according to claim 17, wherein the classification codes are assigned by a machine and the further classification codes are assigned by a further machine.