Systems and methods for using metadata to enhance data management operations

• US 7,613,752 B2
• Filed: 11/28/2006
• Issued: 11/03/2009
• Est. Priority Date: 11/28/2005
• Status: Expired due to Fees

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Brockway, Brian, Ngo, David, Schwartz, Jeremy Alan, Prahlad, Anand, Muller, Marcus S.
• Primary Examiner(s): Trujillo; James
• Assistant Examiner(s): Moser; Bruce M
• Application Number:
  US11/564,130
• Publication Number:
  US 20070185914A1
• Time in Patent Office: 1,071 Days
• Field of Search: 707/204
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1466 to make the backup process non-disruptive
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/285 : Clustering or classification
• CPC Class Codes: G06F 2201/80 : Database-specific techniques
• CPC Class Codes: G06F 3/0604 : Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 : by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0649 : Lifecycle management
• CPC Class Codes: G06F 3/0685 : Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: Y10S 707/99942 : Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99948 : Application of database or data structure, e.g. distributed, multimedia, or image
• CPC Class Codes: Y10S 707/99955 : Archiving or backup

Abstract

Systems and methods for managing electronic data are disclosed. Various data management operations can be performed based on a metabase formed from metadata. Such metadata can be identified from an index of data interactions generated by a journaling module, and obtained from their associated data objects stored in one or more storage devices. In various embodiments, such processing of the index and storing of the metadata can facilitate, for example, enhanced data management operations, enhanced data identification operations, enhanced storage operations, data classification for organizing and storing the metadata, cataloging of metadata for the stored metadata, and/or user interfaces for managing data. In various embodiments, the metabase can be configured in different ways. For example, the metabase can be stored separately from the data objects so as to allow obtaining of information about the data objects without accessing the data objects or a data structure used by a file system.

Claims

1. A method for managing the transfer of electronic data in a storage system, the method comprising:

receiving input associated with tagging of information shared among a group of users, wherein the input identifies the criteria associated with the tagging of the information;

receiving a monitor index having a plurality of entries identifying information about data interactions made by at least one application to a plurality of data objects on first and second storage devices, the entries in the monitor index being populated by a journaling application that monitors the data interactions;

automatically determining with one or more computer processors, one or more entries in the monitor index that satisfy the criteria associated with the tagging of the information shared among the group of users;

automatically tagging one or more of the entries in the monitor index that satisfy the criteria such that tagged entries and non-tagged entries exist, wherein tagging the entries occurs separately from the monitoring of data interactions by the journaling application, and wherein the tagged entries identify the entries associated with information shared among the group of users;

accessing the data objects associated with the tagged entries, to obtain information directly from the data objects associated with the tagged entries;

updating, based on the tagged entries, metadata stored in a metabase with information about the data interactions associated with the tagged entries and the information obtained directly from the data objects associated with the tagged entries;

evaluating a selected criteria for transferring at least one data object from the first storage device to the second storage device, said evaluating comprising accessing the metadata stored in the metabase;

identifying a first data object, of the plurality of data objects, that satisfies the selected criteria;

moving the first data object to the second storage device; and

updating the metabase to include the location of the first data object on the second storage device when the first data object is associated with a tagged entry in the metabase.

2. The method of claim 1, further comprising performing at least one transferring operation on the first data object based on the selected criteria.

3. The method of claim 1, wherein said accessing of the metadata stored in the metabase is performed without accessing the data objects stored in the first storage device.

4. The method of claim 1, further comprising accessing one or more metadata associated with the corresponding data objects again one or more times to update the metabase.

5. The method of claim 1, wherein said identifying further comprises providing a file reference number associated with the first data object.

6. The method of claim 1, wherein said identifying further comprises providing a path indicative of a location of the first data object on the first storage device.

7. The method of claim 1, wherein the selected criteria comprises at least one of a schedule policy, a storage policy and a migration policy.

8. The method of claim 1, additionally comprising copying the first data object from a first location on the first storage device to a second location on the second storage device.

9. The method of claim 8, additionally comprising updating the metabase to reflect the second location of the first data object on the second storage device.

10. The method of claim 9, wherein the metabase comprises first and second metabases, wherein the first metabase is associated with the first storage device and the second metabase is associated with the second storage device.

11. The method of claim 10, wherein said updating comprises updating one or more records of the second metabase.

12. The method of claim 11, additionally comprising removing records on the first metabase associated with the metadata of the first data object.

13. The method of claim 8, additionally comprising storing on the first storage device reference data that identifies the second location of the first data object on the second storage device.

14. A system for managing the transfer of electronic data between a plurality of storage devices, the system comprising:

a first storage device having a plurality of data objects;

a second storage device;

a selected criteria for copying at least one of the plurality of data objects on the first storage device to the second storage device;

at least one metabase having a plurality of records comprising metadata associated with the plurality of data objects;

a classification module executing in one or more computer processors, the classification module configured to evaluate a monitor index generated by a journal program, the monitor index comprising entries about interactions between at least one application and the plurality of data objects, the classification module is further configured to automatically tag one or more of the entries in the monitor index that related to information shared among a group of users such that tagged entries and non-tagged entries exist, wherein tagging the entries occurs separately from the journaling application, the classification module further configured to update the metadata stored in the at least one metabase based on interactions identified by the monitor index, wherein the classification module further updates the metabase with information obtained from accessing the data objects associated with the tagged entries; and

a management module configured to access the at least one metabase to identify a first data object, of the plurality of data objects, that satisfies said selected criteria for copying at least one of the plurality of data objects on the first storage device to the second storage device and updating the metabase to include the location of the first data object on the second storage device when the first data object is associated with a tagged entry.

15. The system of claim 14, wherein the interactions identified by the monitor index comprise interactions that modify metadata of at least a portion of the plurality of data objects.

16. The system of claim 14, wherein the management module accesses the at least one metabase without accessing the plurality of data objects on the first storage device.

17. The system of claim 14, wherein the journal program is different than the at least one application.

18. The system of claim 14, wherein the classification module is located outside an input/output (I/O) path between the at least one application and the first storage device.

19. The system of claim 14, wherein the classification module is managed by a first processor and the interactions are managed by a second processor.

20. The system of claim 14, wherein the management module is further configured to transfer the first data object from the first storage device to the second storage device.

21. The system of claim 20, wherein the first storage device comprises a faster access time than the second storage device.

22. The system of claim 14, wherein the at least one metabase comprises a first metabase and a second metabase, wherein records stored in the first metabase correspond to metadata of the plurality of data objects stored on the first storage device, and records stored in the second metabase correspond to metadata of a second plurality of data objects stored on the second storage device.

23. The system of claim 22, further comprising a master index identifying information stored on the first and second metabases.

24. The system of claim 14, wherein at least a portion of the at least one metabase is stored on the first storage device.

25. A system for managing the transfer of electronic data stored on a plurality of storage devices, the system comprising:

first means for storing a first plurality of data objects;

second means for storing a second plurality of data objects;

means for identifying at least one of the first plurality of data objects to be transferred from the first storing means to the second storing means;

third means for storing in a metabase, metadata associated with the first plurality of data objects; and

means for processing a journal file comprising a list of entries associated with interactions between at least one application and the plurality of data objects, said processing means further configured to update said third storing means based on interactions identified by the journal file;

means for tagging one or more of the entries such that tagged entries and non-tagged entries exist, wherein tagging the entries occurs separately from the monitoring of data interactions by the journaling application, and wherein the tagged entries are associated with information shared among a group of users; and

means for updating, based on the tagged entries, metadata stored in a metabase with information obtained directly from the data objects associated with the tagged entries.

26. The system of claim 25, wherein said metadata is stored in the metabase separate from the entire contents of the data objects.

27. The system of claim 25, wherein said identified interactions comprise interactions that modify metadata of at least a portion of the plurality of data objects.

28. The system of claim 25, wherein said accessing of said third storing means is performed without accessing the plurality of data objects on the first storing means.

29. The system of claim 25, wherein said selected criteria comprises a data storage migration policy.