Systems and methods for probabilistic data classification

• US 8,296,301 B2
• Filed: 01/30/2008
• Issued: 10/23/2012
• Est. Priority Date: 01/30/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Lunde, Norman R.
• Primary Examiner(s): WILSON, KIMBERLY LOVEL
• Application Number:
  US12/022,676
• Publication Number:
  US 20090192979A1
• Time in Patent Office: 1,728 Days
• Field of Search: 707/738, 707/999.001
• US Class Current: 707/738
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/90 Details of database functions independent of the retrieved data types
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

A system for performing data classification operations. In one embodiment, the system comprises a filesystem configured to store a plurality of computer files and a scanning agent configured to traverse the filesystem and compile data regarding the attributes and content of the plurality of computer files. The system also comprises an index configured to store the data regarding attributes and content of the plurality of computer files and a file classifier configured to analyze the data regarding the attributes and content of the plurality of computer files and to classify the plurality of computer files into one or more categories based on the data regarding the attributes and content of the plurality of computer files. Results of the file classification operations can be used to set appropriate security permissions on files which include sensitive information or to control the way that a file is backed up or the schedule according to which it is archived.

Claims

1. A computer system comprising:

a filesystem configured to store a plurality of computer files in a computer memory;

a plurality of scanning agents implemented on one or more computer processors, wherein the plurality of scanning agents are configured to traverse the filesystem and compile attributes and content indexes about the plurality of computer files wherein the attributes and content indexes are stored in one or more databases that are stored separately from the filesystem; and

a file classifier comprising one or more computer processors, wherein the file classifier is configured to receive user input wherein the user selects a first set of attributes and content indexes from the one or more databases stored separately from a corresponding first set of computer files in the filesystem,

wherein the file classifier is configured to analyze the user input to determine a set of classification rules such that the classification rules are derived from accessing the first set of the attributes and content indexes in the one or more databases stored separately from the corresponding first set of computer files stored in the filesystem, wherein the set of classification rules are derived without directly accessing the first set of computer files stored in the filesystem,

wherein the file classifier is further configured to classify a second set of computer files stored in the filesystem without accessing the filesystem based on a calculated probability derived from a corresponding second set of attributes and context indexes in the one or more databases stored separately from the filesystem.

2. The system of claim 1, wherein the filesystem comprises one or more data storage devices coupled to a plurality of client computers via a Storage Area Network (SAN), a Network Attached Storage (NAS) unit, or some combination of the two.

3. The system of claim 1, further comprising an index configured to store metadata regarding the plurality of computer files.

4. The system of claim 3, wherein the metadata comprises a file size, name, path, type, or date of creation or modification.

5. The system of claim 3, wherein the metadata comprises information regarding a category that a file has been identified as being a member of.

6. The system of claim 3, wherein the content indexes comprise at least a file content index configured to store information regarding the content of the plurality of computer files.

7. The system of claim 6, wherein the content indexes comprise at least a keyword present in a computer file.

8. The system of claim 1, wherein the file classifier is configured to be trained to recognize files belonging to a category based on a user-provided training data set.

9. The system of claim 8, wherein the file classifier determines that a file should be classified as a member of a category based on a probability threshold.

10. The system of claim 9, wherein a computer file is classified as a member of a category only if the probability that it should be classified as a member of the category exceeds a user-defined threshold.

11. The system of claim 1, wherein the file classifier uses at least one of the group consisting of a Naï

ve Bayesian and a strict Bayesian algorithm.

12. The system of claim 1, wherein the file classifier is configured reduce a field of possible candidate member computer files of a category based on an explicit rule prior to applying a more computationally expensive classification algorithm to identify member computer files of the category.

13. The system of claim 1, wherein the system is configured to alter security access restrictions on a file based upon a category into which the file is classified.

14. The system of claim 1, wherein the system is configured to alter a data backup schedule for a file based upon a category into which the file is classified.

15. The system of claim 1, wherein the system is configured to alter a data migration plan for a file based upon a category into which the file is classified.

16. A method comprising:

traversing a filesystem and compiling data regarding attributes and content indexes about a plurality of computer files stored in the filesystem, wherein the attributes and content indexes are stored in one or more databases that are stored separately from the filesystem;

receiving user input wherein the user selects a first set of attributes and content indexes from the one or more databases stored separately from a corresponding first set of computer files in the filesystem;

analyzing the user input and data regarding the first set of attributes and content indexes about the corresponding first set of computer files stored in the filesystem to derive a set of classification rules from the first set of attributes and content indexes stored separately from the corresponding first set of the corresponding first set of computer files without directly accessing the first set of computer files stored in the filesystem; and

classifying a second set of computer files stored in the filesystem without accessing the filesystem into one or more categories based on a calculated probability derived from a corresponding second set of attributes and context indexes in the one or more databases stored separately from the filesystem.

17. The method of claim 16, further comprising training a file classifier to recognize files belonging to a category based on a user-provided training data set.

18. The method of claim 16, further comprising comparing the calculated probability with a predefined threshold to determine whether a file should be classified as a member of a category.

19. The method of claim 18, wherein a computer file is classified as a member of a category only if the probability that it should be classified as a member of the category exceeds a user-defined threshold.

20. The method of claim 16, further comprising altering a data backup schedule for a file based upon a category into which the file is classified.

21. The method of claim 16, further comprising altering a data migration plan for a file based upon a category into which the file is classified.

22. A computer system comprising:

means for traversing a filesystem and compiling data regarding attributes and content indexes about a plurality of computer files stored in the filesystem in computer memory, wherein the attributes and content indexes are stored in one or more databases that are stored separately from the filesystem, and wherein the means for traversing the filesystem comprises one or more computer processors;

means for receiving user input wherein the user selects a first set of attributes and content indexes from the one or more databases stored separately from a corresponding first set of computer files in the filesystem;

means for analyzing the user input and data regarding the first set of attributes and content indexes about the corresponding first set of computer files stored in the filesystem to derive a set of classification rules from the first set of attributes and content indexes stored separately from the corresponding first set of computer files without directly accessing the first set of computer files stored in the filesystem; and

means for classifying a second set of computer files stored in the filesystem without accessing the filesystem into one or more categories based on a calculated probability derived from a corresponding second set of attributes and context indexes in the one or more databases stored separately from the filesystem.

23. The system of claim 22, wherein the means for classifying comprises a Naï

ve Bayes classifier.