User interface and workflow for performing machine learning

• US 9,177,261 B2
• Filed: 02/19/2014
• Issued: 11/03/2015
• Est. Priority Date: 03/01/2011
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Sawant, Shitalkumar S., Kauffman, Sally, Galindez, Alan Dale, Jaiswal, Sumesh, Aggarwal, Ashish, DiCorpo, Phillip
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Misir, Dave
• Application Number:
  US14/184,581
• Publication Number:
  US 20140172760A1
• Time in Patent Office: 622 Days
• Field of Search: 726 26- 27, 706/12
• US Class Current: 1/1
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06F 2221/2107 File encryption
• CPC Class Codes: G06F 2221/2141 Access rights, e.g. capability lists, access control lists, access tables, access matrices
• CPC Class Codes: G06F 2221/2147 Locking files
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/0631 Resource planning, allocation, distributing or scheduling for enterprises or organisations
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

A computing device receives a training data set that includes a plurality of positive examples of sensitive data and a plurality of negative examples of sensitive data. The computing device analyzes the training data set using machine learning to generate a machine learning-based detection (MLD) profile that can be used to classify new data as sensitive data or as non-sensitive data. The computing device computes a quality metric for the MLD profile.

Claims

1. A method comprising:

receiving, by a processing device, a training data set that includes a plurality of positive examples of sensitive data and a plurality of negative examples of sensitive data;

receiving a selection of a memory allocation;

analyzing, by the processing device, the training data set using machine learning to train a machine learning-based detection (MLD) profile, the MLD profile to be used to classify new data as sensitive data or as non-sensitive data; and

computing, by the processing device, a quality metric for the MLD profile, the quality metric comprising a memory utilization rating, wherein the memory utilization rating complies with the memory allocation.

2. The method of claim 1, wherein:

analyzing the training data set comprises performing feature extraction on the training data set to generate a feature set that comprises features of the positive examples and features of the negative examples and generating a classification model from the training data set; and

the quality metric comprises at least one of a false positive rating or a false negative rating.

3. The method of claim 2, further comprising:

enabling a deploy operation if the false positive rating is within a false positive threshold and the false negative rating is within a false negative threshold;

receiving a user request to perform the deploy operation; and

in response to receiving the user request, adding the MLD profile to a data loss prevention (DLP) policy of a DLP system.

4. The method of claim 2, further comprising:

identifying at least one of data that caused false positives and data that caused false negatives from the training data set.

5. The method of claim 1, further comprising:

receiving categorization information for the positive examples of sensitive data and for the negative examples of sensitive data; and

identifying categories of data to add to the training data set to improve the quality metric.

6. The method of claim 1, further comprising:

modifying the training data set in response to user input if the quality metric fails to meet a quality threshold;

analyzing the modified training data set to retrain the MLD profile; and

computing a new quality metric for the MLD profile.

7. The method of claim 1, wherein the training data set was collected by a data loss prevention (DLP) policy of a DLP system, the plurality of negative examples of sensitive data comprising documents that were misclassified as sensitive documents by the DLP policy, the method further comprising:

deploying the MLD profile to the DLP policy.

8. The method of claim 1, further comprising:

receiving a selection of a sensitivity threshold for the MLD profile; and

controlling a sensitivity threshold setting for the MLD profile based on the selection.

9. A non-transitory computer readable storage medium including instructions that, when executed by a processing device, cause the processing device to perform operations comprising:

receiving, by the processing device, a training data set that includes a plurality of positive examples of sensitive data and a plurality of negative examples of sensitive data;

receiving categorization information for the positive examples of sensitive data and for the negative examples of sensitive data;

analyzing, by the processing device, the training data set using machine learning to train a machine learning-based detection (MLD) profile, the MLD profile to be used to classify new data as sensitive data or as non-sensitive data;

computing, by the processing device, a quality metric for the MLD profile; and

identifying categories of data to add to the training data set to improve the quality metric.

10. The non-transitory computer readable storage medium of claim 9, wherein:

analyzing the training data set comprises performing feature extraction on the training data set to generate a feature set that comprises features of the positive examples and features of the negative examples and generating a classification model from the training data set; and

the quality metric comprises at least one of a false positive rating, a false negative rating or a memory utilization rating.

11. The non-transitory computer readable storage medium of claim 10, the operations further comprising:

receiving a user selection of a memory allocation before analyzing the training data set, wherein the memory utilization rating complies with the memory allocation.

12. The non-transitory computer readable storage medium of claim 10, the operations further comprising:

enabling a deploy operation if the false positive rating is within a false positive threshold and the false negative rating is within a false negative threshold;

receiving a user request to perform the deploy operation; and

in response to receiving the user request, adding the MLD profile to a data loss prevention (DLP) policy of a DLP system.

13. The non-transitory computer readable storage medium of claim 10, the operations further comprising:

identifying at least one of data that caused false positives and data that caused false negatives from the training data set.

14. The non-transitory computer readable storage medium of claim 9, the operations further comprising:

modifying the training data set in response to user input if the quality metric fails to meet a quality threshold;

analyzing the modified training data set to retrain the MLD profile; and

computing a new quality metric for the MLD profile.

15. The non-transitory computer readable storage medium of claim 9, wherein the training data set was collected by a data loss prevention (DLP) policy of a DLP system, the plurality of negative examples of sensitive data comprising documents that were misclassified as sensitive documents by the DLP policy, the operations further comprising:

deploying the MLD profile to the DLP policy.

16. The non-transitory computer readable storage medium of claim 9, the operations further comprising:

receiving a selection of a sensitivity threshold for the MLD profile; and

controlling a sensitivity threshold setting for the MLD profile based on the selection.

17. A computing device comprising:

a memory to store instructions for a machine learning manager; and

a processing device to execute the instructions, wherein the instructions cause the processing device to;

receive a training data set that includes a plurality of positive examples of sensitive data and a plurality of negative examples of sensitive data;

analyze the training data set using machine learning to train a machine learning-based detection (MLD) profile to be used to classify new data as sensitive data or as non-sensitive data;

compute a quality metric for the MLD profile;

determine that the quality metric fails to meet a quality threshold;

modify the training data set;

analyze the modified training data set to retrain the MLD profile; and

compute a new quality metric for the MLD profile.

18. The computing device of claim 17, wherein to analyze the training data set the processing device is to:

perform feature extraction on the training data set to generate a feature set that comprises features of the positive examples and features of the negative examples; and

generate a classification model from the training data set;

wherein the quality metric comprises at least one of a false positive rating, a false negative rating or a memory utilization rating.

19. The computing device of claim 17, wherein the processing device is further to:

receive categorization information for the positive examples of sensitive data and for the negative examples of sensitive data; and

identify categories of data to add to the training data set to improve the quality metric.

20. The computing device of claim 17, wherein the processing device is further to:

receive a user selection of a memory allocation before analyzing the training data set, the quality metric comprising a memory utilization rating, wherein the memory utilization rating complies with the memory allocation.