System and method for automatically redistributing metadata across managers

• US 7,827,147 B1
• Filed: 03/30/2007
• Issued: 11/02/2010
• Est. Priority Date: 03/30/2007
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Data Speed Technology LLC (Empire IP LLC)
• Inventors: Custers, Joris, D'Hoye, Koen, Wispelaere, Wim De
• Primary Examiner(s): Robinson; Greta L
• Assistant Examiner(s): Wilcox; James J
• Application Number:
  US11/694,631
• Time in Patent Office: 1,313 Days
• Field of Search: None
• US Class Current: 707/652
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/134 Distributed indices

Abstract

System and method for managing metadata. A computer system includes one or more hosts, each host having an associated backup agent. Each backup agent conveys data from its associated host to a storage pool and stores metadata associated with the data in a metabase. In response to detecting that a first metabase meets a first condition, the system identifies a backup agent that has stored metadata in the first metabase and redistributes at least a portion of the metadata to a second metabase. In one embodiment, the first condition comprises detecting the amount of metadata stored in the first metabase reaching a threshold value. In a further embodiment, the metadata comprises entries, each of which corresponds to a data file. Each entry includes a fingerprint that is derived from and identifies the corresponding data file. The fingerprints identifying the data files may be encrypted.

Claims

1. A computer system comprising:

one or more storage pools;

a plurality of hosts coupled to said one or more storage pools via a network; and

at least a first metabase server comprising a metabase coupled to at least one storage pool of said one or more storage pools via a network and coupled to a first host and a second host of said plurality of hosts via a network, wherein said metabase comprises metadata for both the first host and the second host of the plurality of hosts; and

wherein each of the first host and the second host is configured to;

convey data to one or more of said storage pools via said network; and

store metadata associated with said data in said metabase;

wherein in response to detecting an amount of metadata stored in the first metabase has reached a threshold value, the system is configured to;

generate a query to discover whether there are metabases other than the first metabase coupled to the system that have capacity available for storing additional metadata;

identify a second metabase of a second metabase server as being a metabase that has capacity available for storing additional metadata, responsive to said query;

identify one or more hosts of the plurality of hosts that have metadata stored in said first metabase;

select a particular host of the one or more hosts for redistribution;

add a connection link between the second metabase server and the particular host responsive to selecting said particular host for redistribution, wherein the second metabase server is different from the first metabase server;

redistribute at least a portion of the metadata associated with the particular host from the first metabase server to the second metabase of the second metabase server; and

configure the particular host to utilize the second metabase server for accesses to said at least a portion of the metadata instead of the first metabase server.

2. The system as recited in claim 1,

wherein the metadata comprises a plurality of entries, each entry corresponding to a data file;

and

wherein each entry includes a file fingerprint that unambiguously identifies the corresponding data file.

3. The system as recited in claim 2, wherein the file fingerprints are encrypted.

4. The system as recited in claim 2, wherein a first storage pool of the storage pools is configured to:

store a first file as one or more data segments;

associate a data segment fingerprint with each of the data segments;

de-duplicate the data segments in the first storage pool; and

maintain an entry in a table which includes;

a file fingerprint of the first file; and

one or more data segment fingerprints corresponding to the one or more data segments.

5. The system as recited in claim 1,

wherein the first host is configured to store data in a plurality of folders;

wherein a first portion of metadata associated with data from a first folder of the folders is included in said at least a portion of the metadata; and

wherein a second portion of the metadata associated with data from the first folder of the folders is not redistributed from the first metabase.

15. The system as recited in claim 1, wherein data of the particular host that is associated with given metadata redistributed from the first metabase to the second metabase is stored in a first storage pool of the one or more storage pools both before and after the given metadata is redistributed from the first metabase to the second metabase.

6. A method for redistributing metadata across metabases, the method comprising:

conveying data from a plurality of hosts to one or more storage pools, wherein said plurality of hosts are coupled to said one or more storage pools via a network;

storing metadata associated with said data in a metabase included in a first metadata server, wherein said metabase is coupled to at least one storage pool of said one or more storage pools via a network and coupled to a first host and a second host of said plurality of hosts via a network, wherein said metabase comprises metadata for both a first host and a second host of the plurality of hosts;

detecting that an amount of data stored in said metabase has reached a threshold value;

wherein in response to said detecting;

generating a query to discover whether there are metabases other than the first metabase coupled to the system that have capacity available for storing additional metadata;

identifying a second metabase of a second metabase server as being a metabase that has capacity available for storing additional metadata, responsive to said query;

identifying one or more hosts of the plurality of hosts that have metadata stored in said first metabase;

selecting a particular host of the one or more hosts for redistribution;

adding a connection link between the second metabase server and the particular host responsive to selecting said particular host for redistribution, wherein the second metabase server is different from the first metabase server;

redistributing at least a portion of the metadata associated with the particular host from the first metabase server to the second metabase of the second metabase server; and

configuring the particular host to utilize the second metabase server for accesses to said at least a portion of the metadata instead of the first metabase server.

7. The method as recited in claim 6,

wherein the metadata comprises a plurality of entries, each entry corresponding to a data file;

and

wherein each entry includes a file fingerprint that unambiguously identifies the corresponding data file.

8. The method as recited in claim 7, wherein the file fingerprints identifying the data files are encrypted.

9. The method as recited in claim 7, further comprising:

storing a first file as one or more data segments;

associating a data segment fingerprint with each of the data segments;

de-duplicating the data segments in the first storage pool; and

maintaining an entry in a table which includes;

a file fingerprint of the first file; and

one or more data segment fingerprints corresponding to the one or more data segments.

10. The method as recited in claim 6, further comprising redistributing metadata to balance a load of metadata stored in two or more metabases.

11. A computer readable medium storing computer instructions that are executable by a processor to:

convey data from a plurality of hosts to one or more storage pools, wherein said plurality of hosts are coupled to said one or more storage pools via a network;

store metadata associated with said data in a metabase included in a first metadata server, wherein said metabase is coupled to at least one storage pool of said one or more storage pools via a network and coupled to a first host and a second host of said plurality of hosts via a network, wherein said metabase comprises metadata for both a first host and a second host of the plurality of hosts;

detect that an amount of data stored in said metabase has reached a threshold value;

wherein in response to said detection;

generate a query to discover whether there are metabases other than the first metabase coupled to the system that have capacity available for storing additional metadata;

identify a second metabase of a second metabase server as being a metabase that has capacity available for storing additional metadata, responsive to said query;

identify one or more hosts of the plurality of hosts that have metadata stored in said first metabase;

select a particular host of the one or more hosts for redistribution;

add a connection link between the second metabase server and the particular host, wherein the second metabase server is different from the first metabase server;

redistribute at least a portion of the metadata associated with the particular host from the first metabase server to the second metabase of the second metabase server; and

configure the particular host to utilize the second metabase server for accesses to said at least a portion of the metadata instead of the first metabase server.

12. The computer readable medium as recited in claim 11,

wherein the metadata comprises a plurality of entries, each entry corresponding to a data file;

and

wherein each entry includes a file fingerprint that is derived from and unambiguously identifies the corresponding data file.

13. The computer readable medium as recited in claim 12, wherein the file fingerprints identifying the data files are encrypted.

14. The computer readable medium as recited in claim 12, wherein the instructions are further executable to:

store a first file as one or more data segments;

associate a data segment fingerprint with each of the data segments;

de-duplicate the data segments in the first storage pool; and

maintain an entry in a table which includes;

a file fingerprint of the first file; and

one or more data segment fingerprints corresponding to the one or more data segments.