Enhancing training of predictive coding systems through user selected text

• US 9,058,327 B1
• Filed: 05/17/2012
• Issued: 06/16/2015
• Est. Priority Date: 01/25/2012
• Status: Expired due to Fees

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Lehrman, Gary, Rangan, Venkat, Wiggins, Nelson, Desai, Malay
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Zidanic, Michael
• Application Number:
  US13/474,602
• Time in Patent Office: 1,125 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/93 Document management systems

Abstract

An exemplary predictive coding system can be programmed to update a plurality of training documents based on a portion of a training document selected by a user. The predictive coding system generates a machine learning engine based on the updated plurality of training documents. The predictive coding system predicts a classification for one or more remaining documents from the plurality of training documents using the machine learning engine.

Claims

1. A method comprising:

updating, by a predictive coding system, a set of training documents based on a selected portion of a training document of the set to obtain an updated set of training documents;

searching, by the predictive coding system, content within other training documents in the updated set using a machine learning engine based on the selected portion and variations of the selected portion;

determining a probability measure for the content; and

classifying, by the predictive coding system, a second training document containing the content based on the probability measure.

2. The method of claim 1, further comprising:

presenting the set of training documents in a graphical user interface (GUI) of a selection tool;

receiving, by the selection tool, a selection of the selected portion of the training document and a training classification; and

associating the training classification with the selected portion of the training document.

3. The method of claim 2, wherein updating the set of training documents comprises:

creating a new training document comprising the selected portion of the training document;

associating the training classification with the new training document; and

including the new training document in the updated set of training documents.

4. The method of claim 2, further comprising:

receive a marking from the selection tool;

determine that the selected portion of the training document is a positive contribution to the training classification when the marking is positive; and

determine that the selected portion of the training document is a negative contribution to the training classification when the marking is negative.

5. The method of claim 1, further comprising:

identifying additional documents to add to the updated set;

receiving input for the additional documents; and

modifying the updated set of training documents to include at least one of the additional documents based on the received input associated with the additional documents.

6. The method of claim 1, further comprising:

identifying one or more exemplar documents used by the predictive coding system to classify a prediction document in the updated set of updated training documents; and

identifying a region of the prediction document used by the predictive coding system to determine the classification for the prediction document in the updated set of training documents.

7. A non-transitory computer readable storage medium having instructions that, when executed by a processing device, cause the processing device to perform operations comprising:

updating a set of training documents based on a selected portion of a training document of the set to obtain an updated set of training documents;

searching content within other training documents in the updated set using a machine learning engine based on the selected portion and variations of the selected portion;

determining a probability measure of the content; and

classifying a second training document containing the content based on the probability measure.

8. The non-transitory computer readable storage medium of claim 7, further comprising:

presenting the set of training documents in a graphical user interface (GUI) of a selection tool;

receiving, by the selection tool, a selection of a portion of the training document and a training classification; and

associating the training classification with the portion of the training document.

9. The non-transitory computer readable storage medium of claim 8, further comprising:

creating a new training document comprising the selected portion of the training document;

associating the training classification with the new training document; and

including the new training document in the updated set of training documents.

10. The non-transitory computer readable storage medium of claim 8, further comprising:

receiving a marking from the selection tool;

determining that the selected portion of the training document is a positive contribution to the training classification when the marking is positive; and

determining that the selected portion of the training document is a negative contribution to the training classification when the marking is negative.

11. The non-transitory computer readable storage medium of claim 7, further comprising:

identifying additional documents to add to the updated set;

receiving input for the additional documents; and

modifying the updated set of training documents to include at least one of the additional documents based on the received input associated with the additional documents.

12. The non-transitory computer readable storage medium of claim 7, further comprising:

identifying one or more exemplar documents used by a predictive coding system to classify a prediction document in the updated set of training documents; and

identifying a region of the prediction document used by the predictive coding system to determine the classification for the prediction document in the updated set of training documents.

13. A system comprising:

a memory; and

a processing device coupled to the memory, wherein the processing device is configured to;

update a set of training documents based on a selected portion of a training document of the set to obtain an updated set of training documents;

search content within other training documents in the updated set using a machine learning engine based on the selected portion and variations of the selected portion; and

determine a probability measure of the content; and

classify a second training document containing the content based on the probability measure.

14. The system of claim 13, wherein the processing device is further configured to:

present the set of training documents in a graphical user interface (GUI) of a selection tool;

receive, by the selection tool, a content selection of a portion of the training document and a training classification; and

associate the training classification with the portion of the training document.

15. The system of claim 14, wherein the processing device is further configured to:

create a new training document comprising the selected portion of the training document;

associate the training classification with the new training document; and

include the new training document in the updated set of training documents.

16. The system of claim 14, further comprising:

receive a marking from the selection tool;

determine that the selected portion of the training document is a positive contribution to the training classification when the marking is positive; and

determine that the selected portion of the training document is a negative contribution to the training classification when the marking is negative.

17. The system of claim 13, wherein the processing device is further configured to:

identify additional documents to add to the updated set;

receive input for the additional documents; and

modify the updated set of training documents to include at least one of the additional documents based on the received input associated with the additional documents.

18. The system of claim 13, wherein the processing device is further configured to:

identifying one or more exemplar documents used by a predictive coding system to classify a prediction document in the updated set of updated training documents; and

identifying a region of the prediction document used by the predictive coding system to determine the classification for the prediction document in the updates set of the updated training documents.