Systems and methods for classifying and transferring information in a storage network

• US 7,668,884 B2
• Filed: 11/28/2006
• Issued: 02/23/2010
• Est. Priority Date: 11/28/2005
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Brockway, Brian, Schwartz, Jeremy A., Ngo, David, Prahlad, Anand, Muller, Marcus S.
• Primary Examiner(s): Trujillo; James
• Assistant Examiner(s): Moser; Bruce M
• Application Number:
  US11/564,119
• Publication Number:
  US 20070185925A1
• Time in Patent Office: 1,183 Days
• Field of Search: 707/10, 707/200, 707/202
• US Class Current: 707/672
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1466 to make the backup process non-disruptive
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/285 : Clustering or classification
• CPC Class Codes: G06F 2201/80 : Database-specific techniques
• CPC Class Codes: G06F 3/0604 : Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 : by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0649 : Lifecycle management
• CPC Class Codes: G06F 3/0685 : Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: Y10S 707/99942 : Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99948 : Application of database or data structure, e.g. distributed, multimedia, or image
• CPC Class Codes: Y10S 707/99955 : Archiving or backup

Abstract

Systems and methods for data classification to facilitate and improve data management within an enterprise are described. The disclosed systems and methods evaluate and define data management operations based on data characteristics rather than data location, among other things. Also provided are methods for generating a data structure of metadata that describes system data and storage operations. This data structure may be consulted to determine changes in system data rather than scanning the data files themselves.

Claims

1. A method of classifying data generated within a computer system, wherein the computer system includes a file system and a data storage device, the method comprising:

intercepting the generated data, wherein the intercepted data is to be stored at the data storage device;

generating a log entry within a log, wherein the log contains entries associated with the intercepted data;

analyzing entries within the log to identify entries that satisfy at least one data management criterion, wherein the data management criterion is associated with data storage operations to be performed on the generated data;

storing the intercepted data generated within the computer system on the data storage device;

analyzing at least some of the data stored on the data storage device;

based upon the analysis of the data, generating metadata that describes the data, wherein the generated metadata does not include metadata generated by the file system, or wherein the generated metadata includes both metadata generated by the file system and metadata other than metadata generated by the file system, including metadata indicating a period of time before which a data storage operation is to be performed on the data; and

adding the generated metadata to a metabase, wherein the generated metadata describes the intercepted data associated with the identified log entries and refers to the data stored on the data storage device, but wherein the metabase does not include the data.

2. The method of claim 1 further comprising accessing the data storage device to retrieve additional information related to the intercepted data, and adding the retrieved additional information to the metabase.

3. The method of claim 1 wherein intercepting comprises trapping the generated data and wherein the data is generated by an application program.

4. The method of claim 1 wherein analyzing entries within the log comprises parsing the log entries into a predetermined format that includes a file reference number.

5. The method of claim 1 wherein the data management criterion identifies a particular user and wherein analyzing entries identifies entries related to intercepted data that has been accessed by the identified user and classifies the intercepted data by characteristics associated with the user.

6. The method of claim 1 wherein the data management criterion identifies a department within an organization and wherein analyzing entries identifies entries related to data that has been accessed by a member of the identified department and classifies the data based on the identified department.

7. The method of claim 1 wherein adding metadata comprises associating one or more predefined classifications with the data.

8. The method of claim 1 wherein adding metadata comprises associating one or more user-defined classifications with the data.

9. The method of claim 1 wherein adding metadata comprises associating one or more classifications with the data based on the similarity of the data to previously classified data and storing the metadata based on the classifications.

10. The method of claim 1 wherein analyzing at least some of the data stored on the data storage device includes analyzing content of at least some of the data stored on the data storage device, and wherein generating metadata that describes the data includes generating metadata based upon the analysis of the content.

11. The method of claim 1 wherein the intercepting, the generating, the analyzing entries, the storing, the analyzing at least some of the data, and the adding are performed as part of a unitary process.

12. The method of claim 1 wherein the intercepting, the generating, the analyzing entries, the storing, the analyzing at least some of the data, and the adding are performed at the computing system.

13. The method of claim 1 wherein the generated metadata does not indicate a pattern of access to the data upon which data storage operations to be performed on the data are based.

14. A computing system for classifying data stored within a computer system, wherein the computer system includes a file system and an application program configured to generate program data to be employed by the computer system, the computing system comprising:

a processor;

a memory;

a monitor agent configured to process at least part of the program data to create at least one log entry related to the program data;

a data collection agent configured to analyze the at least one log entry and select at least part of the program data associated with the at least one log entry;

a data classification agent configured to identify characteristics of and assign classifications to the selected program data, wherein a classification associates the selected program data with other program data having similar characteristics, wherein the data classification agent assigns classifications to the selected program data by

analyzing the selected program data; and

assigning classifications based upon the analysis of the selected program data; and

a metabase configured to store metadata information about the selected program data, wherein the metadata information contains at least the assigned classifications of the selected program data, wherein the metadata information does not contain metadata generated by the file system, or wherein the metadata information contains both metadata generated by the file system and metadata other than metadata generated by the file system, including metadata relating to data storage operations, and wherein the metabase is configured to receive requests from the computer system or other computing systems to perform data management operations based on the assigned classifications.

15. The system of claim 14 wherein the monitor agent processes program data by capturing a request to store the program data.

16. The system of claim 14 wherein the monitor agent processes program data by traversing a file system containing the program data.

17. The system of claim 14 wherein the monitor agent is further configured to identify changes in the program data.

18. The system of claim 14 wherein the data classification agent identifies characteristics of the selected program data that are not stored with the selected program data.

19. The system of claim 14 wherein the data classification agent identifies characteristics including an aging criteria.

20. The system of claim 14 wherein the data classification agent identifies characteristics including a pattern of use.

21. The system of claim 14 wherein the data collection agent captures additional information about the selected program data as metadata describing the selected program data and wherein the metadata includes at least one of the following types of information:

an owner, a last modified time, a size, an application that generated the selected program data, a user that generated the selected program data, header information, a creation date, a file type, a last accessed time, an application type, a location, a frequency of change, a business unit, usage trends associated with the selected program data, or aging information.

22. The system of claim 14 wherein the data collection agent captures additional information about the selected program data if changes are detected in the program data.

23. The system of claim 14 wherein the monitor agent is further configured to create a log entry when at least part of the program data changes, and the monitor agent is a filter driver.

24. The system of claim 14 wherein the monitor agent is further configured to create a log entry when at least part of the program data changes, and the monitor agent is Microsoft Change Journal.

25. The system of claim 14 wherein the monitor agent is further configured to create a log entry when at least part of the program data changes, and the monitor agent is distinct from the data classification agent.

26. The system of claim 14 wherein the monitor agent is further configured to create a log entry when at least part of the program data changes.

27. The system of claim 14 further comprising a change journal database configured to store log entries created by the monitor agent.

28. The system of claim 14 wherein the metabase is further configured to perform the data management operations without traversing a data store where the program data is stored.

29. A computing system for classifying data stored within a computing device, the computing system comprising:

a processor;

a memory;

a means for storing data;

a means for collecting stored data comprising a data collection agent configured to identify data objects stored by the means for storing data and obtain information about identified data objects;

a means for classifying data comprising a data classification agent for

analyzing content of the identified data objects; and

assigning classifications to identified data objects based upon the analysis of the content of the identified data objects,

wherein the classifications are related to one or more applications that may use the data object, and wherein the classifications do not include metadata generated by the file system, or wherein the classifications include both metadata generated by the file system and metadata other than metadata generated by the file system, including metadata relating to data storage operations; and

a means for storing metadata comprising a metabase configured to store information about identified data objects and their classifications.

30. The system of claim 29 wherein the means for collecting stored data further comprises a means for journaling changes.

31. The system of claim 29 further comprising a means for administering the configuration of data movement to and from the means for storing data.

32. A computer-readable medium storing instructions that when executed by a computing system cause the computing system to perform a method for classifying data stored in a data storage system, the method comprising:

receiving a file to be stored in a data storage device;

analyzing the file to determine characteristics associated with the file;

determining one or more of multiple categorizations for the file based on the determined characteristics, wherein the categorizations identify trends associated with the file that are related to trends identified about other files;

storing the one or more categorizations in a metabase that stores metadata and other information regarding the file and other data, wherein the metadata does not include metadata generated by the file system, or wherein the generated metadata includes both metadata generated by the file system and metadata other than metadata generated by the file system, including metadata relating to data storage operations; and

performing a storage operation on the file with respect to the data storage device, based at least in part on the one or more categorizations for the file, wherein performing a storage operation on the file includes

receiving a query that indicates one of the one or more categorizations;

querying the metabase with the query;

determining that the file satisfies the query; and

accessing the file.

33. The computer-readable medium of claim 32 wherein the system is part of a network and the computing system is coupled to the data storage device through the network.

34. The computer-readable medium of claim 32 wherein the computing system executes an agent specialized to manipulate data related to an application that is associated with the file.