Systems and methods for classifying and transferring information in a storage network

• US 7,707,178 B2
• Filed: 11/28/2006
• Issued: 04/27/2010
• Est. Priority Date: 11/28/2005
• Status: Expired due to Fees

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Brockway, Brian, Schwartz, Jeremy A., Ngo, David, Prahlad, Anand, Muller, Marcus S.
• Primary Examiner(s): Cottingham; John R.
• Assistant Examiner(s): Reyes; Mariela D
• Application Number:
  US11/564,215
• Publication Number:
  US 20070198570A1
• Time in Patent Office: 1,246 Days
• Field of Search: None
• US Class Current: 707/617
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1466 to make the backup process non-disruptive
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/285 : Clustering or classification
• CPC Class Codes: G06F 2201/80 : Database-specific techniques
• CPC Class Codes: G06F 3/0604 : Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 : by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0649 : Lifecycle management
• CPC Class Codes: G06F 3/0685 : Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: Y10S 707/99942 : Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99948 : Application of database or data structure, e.g. distributed, multimedia, or image
• CPC Class Codes: Y10S 707/99955 : Archiving or backup

Abstract

Systems and methods for data classification to facilitate and improve data management within an enterprise are described. The disclosed systems and methods evaluate and define data management operations based on data characteristics rather than data location, among other things. Also provided are methods for generating a data structure of metadata that describes system data and storage operations. This data structure may be consulted to determine changes in system data rather than scanning the data files themselves.

Claims

1. A method for updating a centralized database of metadata in a network comprising multiple computing devices storing multiple data objects, wherein the method is performed by one or more computing systems, each computing system having a processor and memory, the method comprising:

receiving at a first time from one or more of the multiple computing devices multiple entries for the centralized database of metadata,

wherein an entry includes information identifying a data object and information identifying one or more data classifications associated with the data object, wherein the one or more data classifications are generated using one or more characteristics of the data object, and

wherein one or more of the multiple computing devices had been disconnected from the network for a period of time prior to the first time;

determining one or more rules based on one or more policies to apply to the multiple entries, wherein the one or more policies provide an order of preference by which the multiple entries are to be integrated into the centralized database of metadata;

applying the one or more rules to the multiple entries;

based on the application of the one or more rules, after the period of time where the one or more of the multiple computing devices were disconnected from the network, determining by the one or more computing systems, the order of preference by which to integrate the multiple entries into the centralized database of metadata; and

integrating the multiple entries into the centralized database of metadata according to the determined order of preference, wherein integrating includes—

searching, within the centralized database of metadata, for metadata entries previously associated with a first data object;

if metadata entries previously associated with the first data object are not found, then;

adding metadata entries associated with the first data object to the centralized database of metadata;

or

if metadata entries previously associated with the first data object are found, then;

updating the metadata entries in the centralized database of metadata with information associated with data classifications for the data objects based on the information identifying the first data object and information identifying the one or mere data classifications associated with the first data object.

2. The method of claim 1, further comprising receiving information from a change journal entry of a change journal.

3. The method of claim 1, further comprising receiving information identifying a data object that has been deleted.

4. The method of claim 1, further comprising identifying a data object that has been modified by a particular user.

5. The method of claim 1, further comprising identifying a data object that has been moved to a new storage location.

6. The method of claim 1, further comprising identifying a data object that has been copied from a first computing device to a second computing device.

7. The method of claim 1 wherein the one or more data classifications specify a storage retention policy for the data object.

8. The method of claim 1 wherein the one or more data classifications specify a storage aging policy for the data object.

9. The method of claim 1 wherein the one or more data classifications specify an access history of the data object.

10. A computer system for updating a database of metadata associated with data objects stored in a data store, the computer system comprising:

a processor;

a memory;

a data object monitoring component, wherein the data object monitoring component is configured to monitor a data store containing data objects for changes to the data store;

a data object classification component, in communication with the data object monitoring component, wherein the data object classification component is configured to;

receive from the data object monitoring component information identifying multiple data objects;

determine one or more characteristics of the multiple data objects; and

generate, based on the one or more characteristics, one or more data classifications for the multiple data objects;

a database search component, in communication with the data object classification component, wherein the database search component is configured to search a database for database entries associated with the multiple data objects, wherein the database entries store information identifying multiple data objects and one or more data classifications for the multiple data objects; and

a metadata integration component in communication with the database object classification component, wherein the metadata generation component is configured to;

determine multiple database entries, wherein the multiple database entries include the information identifying the multiple data objects and the one or more data classifications for the multiple data objects;

receive one or more rules to apply to the multiple database entries, wherein the one or more rules provide an order of preference by which the multiple database entries are to be integrated into the

apply the one or more rules to the multiple database entries;

based on the application of the one or more rules, determine the order of preference by which to integrate the multiple database entries into the database; and

integrate the multiple database entries into the database according to the determined order of preference

Wherein the metadata integration component integrates the multiple database entries into the database beginning at a first time, and

Wherein the database had been disconnected from the computer system for a period of time prior to the first time.

11. The computer system of claim 10, further comprising:

a secondary storage component, wherein the storage component is configured to store a secondary copy of the multiple data objects in a secondary data store along with a second iteration of the multiple database entries generated by the metadata generation component.

12. The computer system of claim 10, further comprising:

a data object storage component, wherein the data object storage component is configured to store a secondary copy of the multiple data objects in a secondary data store associated with the database storing information identifying data objects with information associated with data classifications for the data objects.

13. The computer system of claim 10, wherein the data classifications associated with the identified data object include information identifying an entity associated with the identified data object, and the created or updated metadata includes an identifier for the entity assigned by a management component that manages the data within the data store.

14. The computer system of claim 10, wherein the data classifications associated with the identified data object include information identifying one or more clients associated with storage of the identified data object to the data store, and the created or updated metadata includes identifiers for the one or more clients assigned by a management component that manages the data within the data store.

15. The computer system of claim 10, wherein the data classifications associated with the identified data object include information identifying the storage policy associated with storage of the identified data object to the data store, and the created or updated metadata includes information from the storage policy.

16. A computer system for updating a database of metadata associated with data stored in a data store, the computer system comprising:

a processor;

a memory;

a data monitoring component, wherein the data monitoring component is configured to monitor a data store containing data objects for changes to the data objects;

a data classification component, in communication with the data monitoring component, wherein the data classification component is configured to;

receive information from the data monitoring component that identifies multiple data objects; and

determine information that describes one or more characteristics associated with the multiple data objects; and

generate, based on the one or more characteristics, one or more data classifications for the multiple data objects;

a metadata generation component in communication with the data classification component, wherein the metadata generation component is configured to generate multiple metadata entries for the multiple data objects based on the one or more data classifications associated with the multiple data objects;

a metadata update component in communication with the metadata generation component, wherein the metadata update component is configured to;

receive one or more rules to apply to the multiple metadata entries, wherein the one or more rules provide an order of preference by which the multiple metadata entries are to be integrated into the database of metadata;

apply the one or more rules to the multiple metadata entries;

based on the application of the one or more rules, determine the order of preference by which to integrate the multiple metadata entries into the database of metadata; and

integrate the multiple metadata entries into the database of metadata according to the determined order of preference; and

Wherein the metadata update component integrates the multiple metadata entries into the database of metadata beginning at a first time, and

Wherein the database of metadata had been disconnected from the computer system for a period of time prior to the first time

a data storage component, wherein the data storage component is configured to store a secondary copy of the data objects in a secondary data store along with a secondary copy of the multiple metadata entries generated by the metadata generation component.

17. The computer system of claim 16, wherein the one or more characteristics associated with the data include characteristics that identify resources used by a data management component when changing the data in the data store.

18. A computer-readable storage medium whose stored contents cause a data storage system to perform a method for updating metadata in a metabase, the method comprising:

receiving an indication of changes made to data objects contained by a data store managed by the data storage system; and

for data objects that have changed;

identifying one or more characteristics of the changed data objects;

based upon the identified one or more characteristics, generating one or more classifications for the changed data objects;

generating multiple metadata entries, wherein a metadata entry includes information identifying a changed data object and the one or more classifications generated for the chanced data object;

receiving one or more rules to apply to the multiple metadata entries, wherein the one or more rules provide an order of preference by which the multiple metadata entries are to be integrated into a metabase associated with the data store;

applying the one or more rules to the multiple metadata entries;

based on the application of the one or more rules, determining the order of preference by which to integrate the multiple metadata entries into the metabase; and

integrating the multiple metadata entries into the metabase according to the determined order of preference b3

Wherein the multiple metadata entries are integrated into the metabase beginning at a first time, and

Wherein the metabase had been disconnected from the data storage system for a period of time prior to the first time.

19. The computer-readable storage medium of claim 18, wherein the method further comprises:

storing an additional instance of the generated multiple metadata entries in the data store.

20. The computer-readable storage medium of claim 18, wherein integrating the multiple metadata entries into the metabase according to the determined order of preference includes:

searching, within the metabase for one or more metadata entries previously associated with a first data object;

if one or more metadata entries associated with the first data object are not found, then adding metadata entries associated with the first data object to the metabase;

or

if one or more metadata entries associated with the first data object are found, then updating the metadata entries in the metabase to include the information identifying the first data object and the one or more classifications generated for the first data object.