Computer-implemented system and method for generating document training sets

• US 10,332,007 B2
• Filed: 11/07/2016
• Issued: 06/25/2019
• Est. Priority Date: 08/24/2009
• Status: Active Grant

• Current Assignee: Nuix North America Inc. (Nuix Ltd.)
• Original Assignee: Nuix North America Inc. (Nuix Ltd.)
• Inventors: Knight, William C., McNee, Sean M.
• Primary Examiner(s): Fleurantin, Jean B
• Application Number:
  US15/345,471
• Publication Number:
  US 20170076203A1
• Time in Patent Office: 960 Days
• Field of Search: 707708, 707737
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

A computer-implemented system and method for generating document training sets is provided. Unclassified documents are provided to two or more classifiers. A classification code assigned to each unclassified document is received. A determination is made as to whether a disagreement exists between classification codes assigned to a common unclassified document via different classifiers. The common unclassified document with a disagreement in classification codes are provided for further review. Results of the further review include one of a new classification code and confirmation of one of the assigned classification codes. The unclassified documents for which a disagreement exists are grouped as a training set.

Claims

1. A computer-implemented method for generating document training sets, comprising:

providing a set of unclassified documents to each of two or more trained classifiers and receiving a classification code assigned to each unclassified document from each classifier;

comparing via a server the classification codes assigned to each unclassified document by two or more of the classifiers, wherein the server comprises a central processing unit, memory, an input port to receive the set of unclassified documents, and an output port to provide a training set for a matter;

determining for at least one of the unclassified documents that a disagreement exists between the classification codes from the two or more classifiers;

providing via the server for further review the unclassified document with a disagreement in classification codes, wherein results of the further review comprise one of a new classification code and confirmation of one of the assigned classification codes;

generating the training set for the matter via the server by grouping the unclassified documents for which the disagreement exists; and

generating a further training set for a same or different matter, comprising;

training two or more other classifiers by identifying features within one or more coded documents, classifying the features, and utilizing the classified features for training the other classifiers;

identifying via the other classifiers one or more features within at least one of the unclassified documents;

assigning by each of the other classifiers, a classification code to each of the identified features;

comparing the classification codes assigned to each feature;

determining whether a disagreement exists between the classification codes assigned to at least one of the features via the other classifiers;

providing the features with a disagreement in classification codes for further review, wherein results of the further review comprise one of a new classification code and confirmation of one of the assigned classification codes; and

grouping as the further training set the unclassified documents associated with the features for which a disagreement exists.

2. A method according to claim 1, further comprising:

comparing the classification codes for each of the unclassified documents with the results of the further review;

identifying those classification codes that differ from the results of the further review; and

re-training the classifiers that assigned the classification codes that differ from the results.

3. A method according to claim 1, further comprising:

training the classifiers with coded documents, each coded document assigned with a classification code.

4. A method according to claim 1, wherein the feature classification is performed by an individual.

5. A method according to claim 1, further comprising:

combining the training set and the further training set as a combined training set.

6. A method according to claim 5, further comprising:

providing the combined training set to one of an individual for manual classification and to one or more of the other classifiers.

7. A method according to claim 1, further comprising:

highlighting areas of the unclassified documents that include the features in disagreement; and

identifying via the highlighted areas, areas of the unclassified documents that are interesting.

8. A method according to claim 1, wherein the further review is performed by an individual.

9. A computer-implemented system for generating document training sets, comprising:

a set of unclassified documents provided to each of two or more trained classifiers, wherein a classification code assigned to each unclassified document from each classifier; and

a server comprising a central processing unit, memory, an input port to receive the set of unclassified documents, and an output port to provide a training set for a matter, wherein the central processing unit is configured to;

compare the classification codes assigned to each unclassified document by two or more of the classifiers;

determine for at least one of the unclassified documents that a disagreement exists between the classification codes from the two or more classifiers;

provide for further review the unclassified document with a disagreement in classification codes;

receiving results of the further review comprising one of a new classification code and confirmation of one of the assigned classification codes;

generate the training set for the matter by grouping the unclassified documents for which the disagreement exists; and

generate a further training set for a same or different matter, comprising;

train two or more other classifiers by identifying features within one or more coded documents, classifying the features, and utilizing the classified features for training the classifiers;

identify via the other classifiers one or more features within at least one of the unclassified documents;

assign by each of the other classifiers, a classification code to each of the identified features;

compare the classification codes assigned to each feature;

determine whether a disagreement exists between the classification codes assigned to at least one of the features via the other classifiers;

provide the features with a disagreement in classification codes for further review, wherein results of the further review comprise one of a new classification code and confirmation of one of the assigned classification codes; and

group as the further training set the unclassified documents associated with the features for which a disagreement exists.

10. A system according to claim 9, further comprising:

a further comparison module to compare the classification codes for each of the unclassified documents with the results of the further review;

an identification module to identify those classification codes that differ from the results of the further review; and

a re-training module to re-train the classifiers that assigned the classification codes that differ from the results.

11. A system according to claim 9, further comprising:

a training module to train the classifiers with coded documents, each coded document assigned with a classification code.

12. A system according to claim 9, wherein the feature classification is performed by an individual.

13. A system according to claim 9, further comprising:

a combination module to combine the training set and the further training set as a combined training set.

14. A system according to claim 13, further comprising:

a delivery module to provide the combined training set to one of an individual for manual classification and to one or more of the other classifiers.

15. A system according to claim 9, further comprising:

an identification module to highlight areas of the unclassified documents that include the features in disagreement and to identify via the highlighted areas, areas of the unclassified documents that are interesting.

16. A system according to claim 9, wherein the further review is performed by an individual.