Computer-implemented system and method for generating a training set for use during document review

• US 9,489,446 B2
• Filed: 12/16/2013
• Issued: 11/08/2016
• Est. Priority Date: 08/24/2009
• Status: Expired due to Fees

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: FTI Consulting Incorporated
• Inventors: Knight, William C., McNee, Sean M.
• Primary Examiner(s): Fleurantin, Jean B
• Application Number:
  US14/108,257
• Publication Number:
  US 20140108312A1
• Time in Patent Office: 1,058 Days
• Field of Search: 707/708, 707/257, 707/737
• US Class Current: 1/1
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

A computer-implemented system and method for generating a training set for use during document review is provided. Classification codes are assigned to a set of documents. Further classification codes are assigned to the same set of documents. The classification code for at least one document is compared with the further classification code for that document. A determination regarding whether a disagreement exists between the assigned classification code and the further classification code for at least one document is made. Those documents with disagreeing classification codes are identified as training set candidates. A stop threshold is applied to the training set candidates and the training set candidates are grouped as a training set when the stop threshold is satisfied.

Claims

1. A computer-implemented method for generating a training set for use during document review, comprising:

assigning classification codes to a set of documents;

receiving further classification codes assigned to the same set of documents;

comparing the classification code for at least one document with the further classification code for that document;

determining whether a disagreement exists between the assigned classification code and the further classification code for at least one document;

identifying those documents with disagreeing classification codes as training set candidates;

applying a stop threshold to the training set candidates, wherein the stop threshold comprises one of a percentage of disagreement, a number of documents with disagreeing classifications, and a zero-defect test; and

designating the training set candidates as a training set when the stop threshold is satisfied.

2. A method according to claim 1, further comprising:

assigning further classification codes to the documents for which a disagreement exists.

3. A method according to claim 1, wherein the classification codes are assigned by a machine and the further classification codes are assigned by a reviewer.

4. A method according to claim 1, wherein the classification codes are assigned by a machine and the further classification codes are assigned by a further machine.

5. A method according to claim 1, further comprising:

automatically classifying documents when a disagreement between the assigned classification code and the further classification code for the at least one document does not exist.

6. A method according to claim 1, further comprising:

associating a confidence score to one or more of the automatically classified documents.

7. A method according to claim 1, further comprising:

identifying the set of documents based on at least one of random selection, a predetermined list of documents, and document review requests.

8. A method according to claim 1, further comprising:

receiving the further classification codes from one or more users;

training a classifier by analyzing the classification codes from the one or more users.

9. A method according to claim 1, further comprising:

identifying at least one of new documents and new classification codes; and

retraining for document classification based on the new documents and new classification codes.

10. A computer-implemented system for generating a training set for use during document review, comprising:

an assignment module to assign classification codes to a set of documents;

a classification receipt module to receive further classification codes assigned to the same set of documents;

a comparison module to compare the classification code for at least one document with the further classification code for that document;

a determination module to determine whether a disagreement exists between the assigned classification code and the further classification code for at least one document;

an identification module to identify those documents with disagreeing classification codes as training set candidates;

a training set module to apply a stop threshold to the training set candidates and to designate the training set candidates as a training set when the stop threshold is satisfied, wherein the stop threshold comprises one of a percentage of disagreement, a number of documents with disagreeing classifications, and a zero-defect test; and

a processor to execute the modules.

11. A system according to claim 10, further comprising:

a document assignment module to assign further classification codes to the documents for which a disagreement exists.

12. A system according to claim 10, wherein the classification codes are assigned by a machine and the further classification codes are assigned by a reviewer.

13. A system according to claim 10, wherein the classification codes are assigned by a machine and the further classification codes are assigned by a further machine.

14. A system according to claim 10, further comprising:

a classifier to automatically classify documents when a disagreement between the assigned classification code and the further classification code for the at least one document does not exist.

15. A system according to claim 10, further comprising:

a scoring module to associate a confidence score to one or more of the automatically classified documents.

16. A system according to claim 10, further comprising:

a document determination module to identify the set of documents based on at least one of random selection, a predetermined list of documents, and document review requests.

17. A system according to claim 10, further comprising:

a training module to train a classifier by analyzing the further classification codes, wherein the further classification codes are received from one or more users.

18. A system according to claim 10, further comprising:

a retraining module to identify at least one of new documents and new classification codes and to retrain for document classification based on the new documents and new classification codes.