User interface and workflow for performing machine learning

• US 8,682,814 B2
• Filed: 03/01/2011
• Issued: 03/25/2014
• Est. Priority Date: 12/14/2010
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: DiCorpo, Phillip, Sawant, Shitalkumar S., Kauffman, Sally, Galindez, Alan Dale, Jaiswal, Sumesh, Aggarwal, Ashish
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Misir, Dave
• Application Number:
  US13/038,299
• Publication Number:
  US 20120150773A1
• Time in Patent Office: 1,120 Days
• Field of Search: 706/11, 706/14, 706/15, 706/46
• US Class Current: 706/12
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06F 2221/2107 File encryption
• CPC Class Codes: G06F 2221/2141 Access rights, e.g. capability lists, access control lists, access tables, access matrices
• CPC Class Codes: G06F 2221/2147 Locking files
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/0631 Resource planning, allocation, distributing or scheduling for enterprises or organisations
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

A computing device receives a training data set that includes a plurality of positive examples of sensitive data and a plurality of negative examples of sensitive data via a user interface. The computing device analyzes the training data set using machine learning to generate a machine learning-based detection (MLD) profile that can be used to classify new data as sensitive data or as non-sensitive data. The computing device displays a quality metric for the MLD profile in the user interface.

Claims

1. A method comprising:

receiving a training data set that includes a plurality of positive examples of sensitive data and a plurality of negative examples of sensitive data via a user interface;

analyzing the training data set using machine learning to train a machine learning-based detection (MLD) profile, the MLD profile to be used to classify new data as sensitive data or as non-sensitive data;

displaying a quality metric for the MLD profile in the user interface; and

if the quality metric fails to meet a quality threshold, performing the following comprising;

modifying the training data set in response to user input; and

analyzing the modified training data set to retain the MLD profile.

2. The method of claim 1, wherein analyzing the training data set comprises:

performing feature extraction on the training data set to generate a feature set that comprises features of the positive examples and features of the negative examples;

generating a classification model from the training data set; and

computing the quality metric, wherein the quality metric includes at least one of a false positive rating, a false negative rating or a memory utilization rating.

3. The method of claim 2, further comprising:

receiving a user selection of a memory allocation via the user interface before analyzing the training data set, wherein the memory utilization rating complies with the memory allocation.

4. The method of claim 2, further comprising:

enabling a deploy operation if the false positive rating is within a false positive threshold and the false negative rating is within a false negative threshold;

receiving a user request to perform the deploy operation via the user interface; and

in response to receiving the user request, adding the MLD profile to a data loss prevention (DLP) policy of a DLP system.

5. The method of claim 2, further comprising:

identifying in the user interface at least one of data that caused false positives and data that caused false negatives from the training data set.

6. The method of claim 1, further comprising:

receiving categorization information for the positive examples of sensitive data and for the negative examples of sensitive data; and

identifying in the user interface categories of data to add to the training data set to improve the quality metric.

7. The method of claim 1, further comprising:

displaying a new quality metric for the MLD profile in the user interface.

8. The method of claim 1, wherein the training data set was collected by a data loss prevention (DLP) policy of a DLP system, the plurality of negative examples of sensitive data comprising documents that were misclassified as sensitive documents by the DLP policy, the method further comprising:

deploying the MLD profile to the DLP policy.

9. The method of claim 1, further comprising:

receiving a selection of a sensitivity threshold via the user interface for the MLD profile; and

controlling a sensitivity threshold setting for the MLD profile based on the selection.

10. A non-transitory computer readable storage medium including instructions that, when executed by a processing device, cause the processing device to perform a method comprising:

receiving a training data set that includes a plurality of positive examples of sensitive data and a plurality of negative examples of sensitive data via a user interface;

analyzing the training data set using machine learning to train a machine learning- based detection (MLD) profile, the MLD profile to be used to classify new data as sensitive data or as non-sensitive data;

displaying a quality metric for the MLD profile in the user interface; and

if the quality metric fails to meet a quality threshold, performing the following comprising;

modifying the training data set in response to user input; and

analyzing the modified training data set to retrain the MLD profile.

11. The non-transitory computer readable storage medium of claim 10, wherein analyzing the training data set comprises:

performing feature extraction on the training data set to generate a feature set that comprises features of the positive examples and features of the negative examples;

generating a classification model from the training data set; and

computing the quality metric, wherein the quality metric includes at least one of a false positive rating, a false negative rating or a memory utilization rating.

12. The non-transitory computer readable storage medium of claim 11, the method further comprising:

receiving a user selection of a memory allocation via the user interface before analyzing the training data set, wherein the memory utilization rating complies with the memory allocation.

13. The non-transitory computer readable storage medium of claim 11, the method further comprising:

enabling a deploy operation if the false positive rating is within a false positive threshold and the false negative rating is within a false negative threshold;

receiving a user request to perform the deploy operation via the user interface; and

in response to receiving the user request, adding the MLD profile to a data loss prevention (DLP) policy of a DLP system.

14. The non-transitory computer readable storage medium of claim 11, the method further comprising:

identifying in the user interface at least one of data that caused false positives and data that caused false negatives from the training data set.

15. The non-transitory computer readable storage medium of claim 10, the method further comprising:

receiving categorization information for the positive examples of sensitive data and for the negative examples of sensitive data; and

identifying in the user interface categories of data to add to the training data set to improve the quality metric.

16. The non-transitory computer readable storage medium of claim 10, the method further comprising:

displaying a new quality metric for the MLD profile in the user interface.

17. The non-transitory computer readable storage medium of claim 10, wherein the training data set was collected by a data loss prevention (DLP) policy of a DLP system, the plurality of negative examples of sensitive data comprising documents that were misclassified as sensitive documents by the DLP policy, the method further comprising:

deploying the MLD profile to the DLP policy.

18. The non-transitory computer readable storage medium of claim 10, the method further comprising:

receiving a selection of a sensitivity threshold via the user interface for the MLD profile; and

controlling a sensitivity threshold setting for the MLD profile based on the selection.

19. A computing device comprising:

a memory to store instructions for a machine learning manager; and

a processing device to execute the instructions, wherein the instructions cause the processing device to;

provide a user interface for the machine learning manager;

receive a training data set that includes a plurality of positive examples of sensitive data and a plurality of negative examples of sensitive data via the user interface;

analyze the training data set using machine learning to train a machine learning-based detection (MLD) profile to be used to classify new data as sensitive data or as non-sensitive data;

display a quality metric for the MLD profile in the user interface; and

if the quality metric fails to meet a quality threshold, performing the following comprising;

modify the training data set in response to user input; and

analyze the modified training data set to retrain the MLD profile.

20. The computing device of claim 19, wherein analyzing the training data set comprises:

performing feature extraction on the training data set to generate a feature set that comprises features of the positive examples and features of the negative examples;

generating a classification model from the training data set; and

computing the quality metric, wherein the quality metric includes at least one of a false positive rating, a false negative rating or a memory utilization rating.

21. The computing device of claim 20, further comprising the instructions to cause the processing device to:

identify in the user interface at least one of data that caused false positives and data that caused false negatives from the training data set.

22. The computing device of claim 19, wherein the training data set was collected by a data loss prevention (DLP) policy of a DLP system, the plurality of negative examples of sensitive data comprising documents that were misclassified as sensitive documents by the DLP policy, further comprising the instructions to cause the processing device to:

deploy the MLD profile to the DLP policy.

23. The computing device of claim 19, further comprising the instructions to cause the processing device to:

receive a selection of a sensitivity threshold via the user interface for the MLD profile; and

control a sensitivity threshold setting for the MLD profile based on the selection.