Systems and methods for using metadata to enhance storage operations

• US 8,271,548 B2
• Filed: 11/28/2006
• Issued: 09/18/2012
• Est. Priority Date: 11/28/2005
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Prahlad, Anand, Schwartz, Jeremy Alan, Ngo, David, Brockway, Brian, Muller, Marcus S.
• Primary Examiner(s): GIRMA, ANTENEH B
• Application Number:
  US11/564,177
• Publication Number:
  US 20070192385A1
• Time in Patent Office: 2,121 Days
• Field of Search: 711/114
• US Class Current: 707/812
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1466 to make the backup process non-disruptive
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/285 : Clustering or classification
• CPC Class Codes: G06F 2201/80 : Database-specific techniques
• CPC Class Codes: G06F 3/0604 : Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 : by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0649 : Lifecycle management
• CPC Class Codes: G06F 3/0685 : Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: Y10S 707/99942 : Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99948 : Application of database or data structure, e.g. distributed, multimedia, or image
• CPC Class Codes: Y10S 707/99955 : Archiving or backup

Abstract

Systems and methods for managing electronic data are disclosed. Various data management operations can be performed based on a metabase formed from metadata. Such metadata can be identified from an index of data interactions generated by a journaling module, and obtained from their associated data objects stored in one or more storage devices. In various embodiments, such processing of the index and storing of the metadata can facilitate, for example, enhanced data management operations, enhanced data identification operations, enhanced storage operations, data classification for organizing and storing the metadata, cataloging of metadata for the stored metadata, and/or user interfaces for managing data. In various embodiments, the metabase can be configured in different ways. For example, the metabase can be stored separately from the data objects so as to allow obtaining of information about the data objects without accessing the data objects or a data structure used by a file system.

Claims

1. A method for managing electronic data in a storage system, the method comprising:

storing a plurality of data objects in at least one first storage device;

receiving a log having a plurality of entries identifying data interactions between at least one application executing on one or more first computer processors of a first computing system and one or more of the plurality of data objects stored in a file system, the log entries being populated by a journaling application;

obtaining first metadata from the log entries about the plurality of data objects, wherein the first metadata comprises at least information about the data interactions;

accessing the plurality of data objects identified by the log entries to obtain second metadata from the plurality of data objects, wherein the second metadata comprises at least data properties about the plurality of data objects;

updating, with one or more second computer processors of a second computing system different than the first computing system and in communication with the first computing system via a network, a metabase with the first metadata and the second metadata, wherein the metabase is separate from the log populated by the journaling application, and wherein the metabase is separate from the plurality of data objects stored in the file system;

accessing a user-defined storage policy associated with at least one of the plurality of data objects, wherein the user-defined storage policy comprises one or more storage criteria associated with migrating at least one of the plurality of data objects to one or more second storage devices, the storage criteria defining one or more of;

a period of time to store data in the first storage device before migrating the data to the one or more second storage devices;

which of the one or more second storage devices to migrate data to from the first storage device; and

a network pathway to utilize in migrating data from the first storage device to the one or more second storage devices;

storing the user-defined storage policy in the metabase wherein the user-defined storage policy is stored in the metabase in association with the first and second metadata about the plurality of data objects;

using the one or more second computer processors of the second computing system to query the metabase, without directly accessing the plurality of data objects in the file system, to identify with one or more computer processors, at least one of the plurality of data objects that satisfies the storage criteria of the user-defined storage policy; and

migrating at least one of the plurality of data objects from the first storage device to the one or more second storage devices based on the storage criteria of the user-defined storage policy stored in the metabase and further based on the first or second metadata about the plurality of data objects stored in the metabase.

2. The method of claim 1, wherein the first and second metadata in the metabase are stored separate from entire content of each of the plurality of data objects.

3. The method of claim 1, wherein the identifying is performed without accessing the plurality of data objects on the storage device.

4. The method of claim 1, wherein the updating comprises:

processing the log to select at least one data interaction that modifies the first metadata of at least one of the plurality of data objects; and

updating the first metadata in the metabase based on the selected at least one data interaction.

5. The method of claim 4, wherein the identifying is performed without accessing a data structure associated with storage of the plurality of data objects on the storage device.

6. The method of claim 5, wherein the identifying is performed without accessing a file access table (FAT) associated with storage of the plurality of data objects on the storage device.

7. The method of claim 1, further comprising accessing a user-defined policy that further identifies selection criteria comprising modifications made to the at least one of the plurality of data objects by one or more of the group of users.

8. The method of claim 7, wherein the selection criteria identifies first or second metadata relating to at least one of a read operation, a write operation, and an access operation associated with the data object.

9. The method of claim 8, wherein the access operation comprises at least one of web-page access, electronic gaming interaction, chat-room access, instant messaging, communication interaction, or multimedia interaction.

10. The method of claim 9, wherein the selection criteria involving the access operation allows at least one of identifying, capturing, classifying, or tracking of the access operation.

11. The method of claim 1, further comprising accessing the data objects one or more times to update the metabase.