Method and system for managing metadata in a storage environment

• US 8,595,237 B1
• Filed: 02/17/2010
• Issued: 11/26/2013
• Est. Priority Date: 02/17/2010
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Agarwal, Gaurav, Agarwal, Manish M., Jobanputra, Varun, Shukla, Vikram, Chaudhary, Anant, Johnson, Sloan, Jeyachandran, Sridher
• Primary Examiner(s): Ehichioya, Fred I
• Assistant Examiner(s): MACKES, KRIS E
• Application Number:
  US12/706,974
• Time in Patent Office: 1,378 Days
• Field of Search: 707/741
• US Class Current: 707/741
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/316 Indexing structures

Abstract

Method and system is provided for managing metadata for a plurality of data containers that are stored at one or more storage volumes in a storage system. The metadata is collected from one or more storage volumes and then provided to a catalog module. The catalog module pre-processes the metadata and then generates a searchable data structure. The searchable data structure may then be used to respond to a user request for information regarding the storage system.

Claims

1. A machine implemented method for a storage system, comprising:

configuring a data storage volume for collecting metadata for a plurality of data containers stored at the data storage volume when a first snapshot of the data storage volume is generated;

wherein the metadata includes an attribute that is associated with the plurality of data containers and the metadata comprises a baseline image for a searchable data structure;

configuring a storage volume as a catalog volume for storing metadata associated with the plurality of data containers;

collecting metadata for the plurality of data containers by a processor executable metadata collection module executed by the plurality of computing nodes;

wherein after the first snapshot is generated for the data storage volume, metadata is collected only for data containers that were changed after the first snapshot was generated;

pre-processing the collected metadata by extracting one or more fields from the collected metadata by a processor executable catalog module executed by the plurality of computing nodes; and

storing the pre-processed metadata in the searchable data structure for responding to a user query requesting information regarding the plurality of data containers;

wherein the searchable data structure is stored at the catalog volume by the catalog module and includes a first searchable segment for storing metadata for a plurality of directory entries of the data storage volume and a second searchable segment for storing metadata for non-directory based data containers stored at the storage volume; and

wherein each entry in the second searchable segment stores a reference to a parent entry in the first searchable segment such that a storage path of each entry in the second searchable segment can be obtained from the reference to the parent entry in the first searchable segment without having to store individual storage paths for the non-directory based data containers at the second searchable segment.

2. The method of claim 1, wherein the pre-processed metadata is stored in an intermediate data structure prior to being stored in the searchable data structure.

3. The method of claim 1, wherein the catalog module receives metadata from a metadata collector that interfaces with a file system for the storage system.

4. The method of claim 1, wherein for responding to the user query, the first searchable segment stores attributes for the plurality of directory entries including a unique directory identifier, a directory name, a size of a directory, a permission associated with a directory, a user identifier identifying a user that uses a directory, a group identifier identifying a group associated with a directory, an access time when a directory was accessed, a creation time when a directory was created, a modification time when a directory was modified and an indicator indicating if a directory was created, modified or deleted.

5. The method of claim 1, wherein for responding to the user query, the second searchable segment stores attributes for the plurality of data containers, including a unique identifier for identifying a data container, an identifier that associates a data container to a directory entry, a data container name, a size of a data container, a permission associated with a data container, a user identifier identifying a user for a data container, a group identifier identifying a group associated with the data container, an access time when a data container was accessed, a creation time when a data container was created, a modification time when a data container was modified and an indicator indicating if a data container was created, modified or deleted.

6. The method of claim 1, wherein in response to a user query, catalog module searches the searchable data structure and obtains information requested in the user query.

7. The method of claim 6, wherein an action is taken based on the obtained information in response to the user query.

8. The method of claim 1, wherein the searchable data structure stores metadata associated with an active file system.

9. The method of claim 1, wherein the catalog volume and the data storage volume are stored within a same volume.

10. The method of claim 1, wherein a plurality of catalog volumes associated with a plurality of data volumes are configured to store metadata for data containers stored at the plurality of data volumes.

11. The method of claim 1, wherein the metadata is collected based on an event, a schedule or a user request.

12. The method of claim 1, wherein the searchable data structure includes a snapshot identifier for identifying a snapshot when the plurality of data containers were replicated;

a time stamp when the snapshot was taken;

an indicator indicating if a data container was added, deleted or modified and a snapshot version indicator indicating the snapshot when a change in status for the data container was discovered.

13. The method of claim 12, wherein a state of a file system is established for a given snapshot by providing an indicator indicating if a data container was added, deleted or modified since a baseline image.

14. A machine implemented method for a storage system for storing a plurality of data containers at one or more storage volumes, comprising:

pre-processing metadata associated with the plurality of data containers, the metadata including an attribute that is associated with the plurality of data containers; and

generating a searchable data structure by indexing the pre-processed metadata such that information related to the plurality of data containers is obtained regardless of a storage volume location;

wherein the search data structure includes a first searchable segment for storing metadata for a plurality of directory entries and a second searchable segment for storing metadata for non-directory based data containers; and

wherein each entry in the second searchable segment stores a reference to a parent entry in the first searchable segment such that a storage path of each entry in the second searchable segment can be obtained from the reference to the parent entry in the first searchable segment without having to store individual storage paths for the non-directory based data containers at the second searchable segment.

15. The method of claim 14, further comprising;

retrieving information by searching the searchable data structure based on a request for information related to the plurality of data containers; and

presenting the information in response to the request.

16. The method of claim 15, wherein the request defines a duration pertaining to a state of the plurality of data containers and at least another attribute for searching the metadata.

17. The method of claim 16, wherein the other attribute describes one or more properties of the plurality of data containers.

18. The method of claim 14, wherein the indexed metadata is stored at a catalog volume that is managed by a catalog module.

19. A machine implemented method for a storage system for storing a plurality of data containers at one or more storage volumes, comprising:

indexing metadata associated with the plurality of data containers, the metadata including an attribute that is associated with the plurality of data containers and the metadata is collected from at least one storage volume; and

storing the indexed metadata associated with the plurality of data containers in a searchable data structure used for obtaining information regarding the plurality of data containers;

wherein the searchable data structure is stored at a catalog volume by a catalog module executed by a plurality of nodes and includes a first searchable segment for storing metadata for a plurality of directory entries of the data storage volume and a second searchable segment for storing metadata for non-directory based data containers stored at the storage volume; and

wherein each entry in the second searchable segment stores a reference to a parent entry in the first searchable segment such that a storage path of each entry in the second searchable segment can be obtained from the reference to the parent entry in the first searchable segment without having to store individual storage paths for the non-directory based data containers at the second searchable segment; and

wherein the searchable data structure stores a snapshot table identifier for identifying a snapshot when the plurality of data containers were replicated and a time stamp when the snapshot was taken.

20. The method of claim 19, wherein metadata is collected to capture a baseline state of a file system that includes the plurality of data containers.

21. The method of claim 19, wherein after the baseline image is created, metadata is collected only for data containers that have been created or modified since the baseline image.

22. The method of claim 19, wherein a state of a file system is established for a given snapshot by providing an indicator indicating if a data container was added, deleted or modified since the baseline image.

23. The method of claim 19, wherein the searchable data structure stores an indicator indicating if a data container was added, deleted or modified.

24. The method of claim 19, wherein the search data structure stores a snapshot version indicator indicating the snapshot when a change in status for the data container was discovered.

25. A machine implemented method for a storage system for storing a plurality of data containers, comprising:

preprocessing metadata associated with the plurality of data containers stored at one or more storage volumes, the metadata including an attribute that is associated with the plurality of data containers;

generating a searchable data structure for storing the pre-processed metadata for gathering information regarding any data container stored at any storage volume;

wherein the search data structure includes a first searchable segment for storing metadata for a plurality of directory entries and a second searchable segment for storing metadata for non-directory based data containers; and

wherein each entry in the second searchable segment stores a reference to a parent entry in the first searchable segment such that a storage path of each entry in the second searchable segment can be obtained from the reference to the parent entry in the first searchable segment without having to store individual storage paths for the non-directory based data containers at the second searchable segment;

searching the searchable data structure in response to a request for information related to the plurality of data containers;

retrieving the requested information from the searchable data structure; and

presenting the information in response to the request.

26. The method of claim 25, further comprising:

taking an action based on a search result generated by searching the searchable data structure.

27. The method of claim 25, wherein a report is presented in response to the request.

28. The method of claim 27, wherein the report is specific to a user having access to the storage system.

29. The method of claim 27, wherein the report is for a group of users using the storage system.

30. The method of claim 25, wherein the metadata is stored at a catalog volume that is managed by a catalog module.