METHOD FOR DIRECTORY ENTRIES SPLIT AND MERGE IN DISTRIBUTED FILE SYSTEM

• US 20130218934A1
• Filed: 02/17/2012
• Published: 08/22/2013
• Est. Priority Date: 02/17/2012
• Status: Abandoned Application

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: SHIGA, Kenta, LIN, Wujuan
• Application Number:
  US13/399,657
• Publication Number:
  US 20130218934A1
• Time in Patent Office: Days
• US Class Current: 707/828
• CPC Class Codes: G06F 16/182 Distributed file systems

Abstract

A distributed storage system has MDSs (metadata servers). Directories of file system namespace are distributed to the MDSs based on hash value of inode number of each directory. Each directory is managed by a master MDS. When a directory grows with a file creation rate greater than a preset split threshold, the master MDS constructs a consistent hashing overlay with one or more slave MDSs and splits directory entries of the directory to the consistent hashing overlay based on hash values of file names under the directory. The number of MDSs in the consistent hashing overlay is calculated based on the file creation rate. When the directory continues growing with a file creation rate that is greater than the preset split threshold, the master MDS adds a slave MDS into the consistent hashing overlay and splits directory entries to the consistent hashing overlay based on hash values of file names.

Claims

1. A plurality of MDSs (metadata servers) in a distributed storage system which includes data servers storing file contents to be accessed by clients, each MDS having a processor and a memory and storing file system metadata to be accessed by the clients,

wherein directories of a file system namespace are distributed to the MDSs based on a hash value of inode number of each directory, each directory is managed by a MDS as a master MDS of the directory, and a master MDS may manage one or more directories;

wherein when a directory grows with a high file creation rate that is greater than a preset split threshold, the master MDS of the directory constructs a consistent hashing overlay with one or more MDSs as slave MDSs and splits directory entries of the directory to the consistent hashing overlay based on hash values of file names under the directory;

wherein the consistent hashing overlay has a number of MDSs including the master MDS and the one or more slave MDSs, the number being calculated based on the file creation rate;

wherein when the directory continues growing with a file creation rate that is greater than the preset split threshold, the master MDS adds a slave MDS into the consistent hashing overlay and splits directory entries of the directory to the consistent hashing overlay with the added slave MDS based on hash values of file names under the directory.

2. The plurality of MDSs according to claim 1,

wherein when the file creation rate of the directory drops below a preset merge threshold, the master MDS removes a slave MDS from the consistent hashing overlay and merges the directory entries of the slave MDS to be removed to a successor MDS remaining in the consistent hashing overlay.

3. The plurality of MDSs according to claim 2, wherein each MDS includes a directory entry split module configured to:

calculate a file creation rate of a directory;

check whether a status of the directory is split or normal which means not split;

if the status is normal and if the file creation rate is greater than the preset split threshold, then split the directory entries for the directory to the consistent hashing overlay based on hash values of file names under the directory; and

if the status is split and if the file creation rate is greater than the preset split threshold, then send a request to the master MDS of the directory to add a slave MDS to the consistent hashing overlay if the MDS of the directory entry split module is not the master MDS, and add a slave MDS to the consistent hashing overlay if the MDS of the directory entry split module is the master MDS.

4. The plurality of MDSs according to claim 3, wherein each MDS maintains a global consistent hashing table which stores information of all the MDSs, and wherein splitting the directory entries by the directory entry split module of the master MDS for the directory comprises:

selecting one or more other MDSs as slave MDSs from the global consistent hashing table, wherein the number of slave MDSs to be selected is calculated by rounding up a value of a ratio of (the file creation rate/the preset split threshold) to a next integer value and subtracting 1;

creating the same directory to each of the selected slave MDSs; and

splitting the directory entries to the selected slave MDSs based on the hash values of file names under the directory, by first sending only the hash values of the file names to the slave MDSs and migrating files corresponding to the file names later;

wherein files can be created in parallel to the master MDS and the slave MDSs as soon as hash values have been sent to the slave MDSs.

5. The plurality of MDSs according to claim 4, wherein each MDS comprises a file migration module configured to:

as a source MDS for file migration, send a directory name of the directory and a file to be migrated to a destination MDS; and

as a destination MDS for file migration, receive the directory name of the directory and the file to be migrated from the source MDS, and create the file to the directory.

6. The plurality of MDSs according to claim 3, wherein each MDS maintains a global consistent hashing table which stores information of all the MDSs, and wherein the directory entry split module which sends a request to the master MDS of the directory to add a slave MDS to the consistent hashing overlay is configured to:

find the master MDS of the directory by looking up the global consistent hashing table;

send an “

Add”

request to the master MDS of the directory;

receive information of a new slave MDS to be added; and

send a directory name of the directory and hash values of file names of files to be migrated to the new slave MDS.

7. The plurality of MDSs according to claim 3,

wherein each MDS maintains a global consistent hashing table which stores information of all the MDSs;

wherein each MDS includes a consistent hashing module;

wherein for adding a new slave MDS into the consistent hashing overlay, the consistent hashing module in the master MDS is configured to select the new slave MDS from the global consistent hashing table, assign to the new slave MDS a unique ID representing an ID range of hash values in the consistent hashing overlay to be managed by the new slave MDS, add the new slave MDS to the consistent hashing overlay, and, if the new slave MDS is added in response to a request from another MDS, then send a reply with the unique ID and an IP address of the new slave MDS to the MDS which sent the request for adding the new slave MDS; and

wherein for merging directory entries of a MDS which is to be removed to a successor MDS, the consistent hashing module of the master MDS is configured to send the IP address of the successor MDS to the MDS which is to be removed, and remove the MDS from the consistent hashing overlay.

8. The plurality of MDSs according to claim 7,

wherein the unique ID assigned to the new slave MDS represents an ID range of hash values equal to half of the ID range of hash values, which is managed by the MDS that sent the request for adding the new slave MDS, prior to adding the new slave MDS.

9. The plurality of MDSs according to claim 3,

wherein each MDS maintains a global consistent hashing table which stores information of all the MDSs;

wherein each MDS includes a directory entry merge module;

wherein the directory entry split module is configured to, if the status is split and if the file creation rate is smaller than the preset merge threshold, then invoke the directory entry merge module to merge the directory entries of the MDS to be removed; and

wherein the invoked directory entry merge module is configured to;

find the master MDS of the directory by looking up the global consistent hashing table; and

if the MDS of the invoked directory merge module is not the master MDS of the directory, then send a “

Merge”

request to the master MDS of the directory to merge the directory entries of the MDS to a successor MDS in the consistent hashing overlay, receive information of the successor MDS, send a directory name of the directory and hash values of file names of files to be migrated to the successor MDS and migrate files corresponding to the file names later, and delete the directory from the MDS to be removed.

10. The plurality of MDSs according to claim 2,

wherein the master MDS of a directory includes a directory inode comprising an inode number column of a unique identifier assigned for the directory and for each file in the directory, a type column indicating “

File”

or “

Directory,”

an ACL column indicating access permission of the file or directory, a status column for the directory indicating split or normal which means not split, a local consistent hashing table column, a count column indicating a number of directory entries for the directory, and a checkpoint column;

wherein the master MDS constructs a local consistent hashing table associated with the consistent hashing overlay, which is stored in the local consistent hashing table column if the status is split; and

wherein a checkpoint under the checkpoint column is initially set to 0 and can be changed when the directory entries are split or merged.

11. The plurality of MDSs according to claim 1,

wherein the master MDS of the directory has a quota equal to 1 and wherein each slave MDS has a quota equal to a ratio between capability of the slave MDS to capability of the master MDS;

and

wherein each MDS includes a directory entry split module configured to;

calculate a file creation rate of a directory;

check whether a status of the directory is split or normal which means not split;

if the status is normal and if the file creation rate is greater than the preset split threshold multiplied by the quota of the MDS of the directory entry split module, then split the directory entries for the directory to the consistent hashing overlay based on hash values of file names under the directory; and

if the status is split and if the file creation rate is greater than the preset split threshold multiplied by the quota of the MDS of the directory entry split module, then send a request to the master MDS of the directory to add a slave MDS to the consistent hashing overlay.

12. The plurality of MDSs according to claim 11,

wherein each MDS maintains a global consistent hashing table which stores information of all the MDSs;

wherein each MDS includes a consistent hashing module;

wherein for adding a new slave MDS into the consistent hashing overlay, the consistent hashing module in the master MDS is configured to select the new slave MDS from the global consistent hashing table, assign to the new slave MDS a unique ID representing an ID range of hash values in the consistent hashing overlay to be managed by the new slave MDS, add the new slave MDS to the consistent hashing overlay, and, if the new slave MDS is added in response to a request from another MDS, then send a reply with the unique ID and an IP address of the new slave MDS to the MDS which sent the request for adding the new slave MDS;

wherein for merging directory entries of a MDS which is to be removed to a successor MDS, the consistent hashing module of the master MDS is configured to send the IP address of the successor MDS to the MDS which is to be removed, and remove the MDS from the consistent hashing overlay; and

wherein the unique ID assigned to the new slave MDS represents an ID range of hash values equal to a portion of the ID range of hash values, which is managed by the MDS that sent the request for adding the new slave MDS, prior to adding the new slave MDS, such that a ratio of the portion of the ID range to be managed by the new slave MDS and a remaining portion of the ID range to be managed by the MDS that sent the request is equal to a ratio of the quota of the new slave MDS and the quota of the MDS that sent the request.

13. A distributed storage system which includes one or more clients, one or more data servers storing file contents to be accessed by the clients, and the plurality of MDSs of claim 1,

wherein each MDS maintains and each client maintains a global consistent hashing table which stores information of all the MDSs;

and

wherein each client has a processor and a memory, and is configured to find the master MDS of the directory by looking up the global consistent hashing table and to send a directory access request of the directory directly to the master MDS of the directory.

14. A distributed storage system comprising:

the plurality of MDSs of claim 1;

one or more clients;

one or more data servers storing file contents to be accessed by the clients;

a first network coupled between the one or more clients and the MDSs; and

a second network coupled between the MDSs and the one or more data servers;

wherein the MDSs serve both metadata access from the clients and file content access from the clients via the MDSs to the data servers.

15. A method of distributing directory entries to a plurality of MDSs (metadata servers) in a distributed storage system which includes clients and data servers storing file contents to be accessed by the clients, each MDS storing file system metadata to be accessed by the clients, the method comprising:

distributing directories of a file system namespace to the MDSs based on a hash value of inode number of each directory, each directory being managed by a MDS as a master MDS of the directory, wherein a master MDS may manage one or more directories;

when a directory grows with a high file creation rate that is greater than a preset split threshold, constructing a consistent hashing overlay with one or more MDSs as slave MDSs and splits directory entries of the directory to the consistent hashing overlay based on hash values of file names under the directory, wherein the consistent hashing overlay has a number of MDSs including the master MDS and the one or more slave MDSs, the number being calculated based on the file creation rate; and

when the directory continues growing with a file creation rate that is greater than the preset split threshold, adding a slave MDS into the consistent hashing overlay and splits directory entries of the directory to the consistent hashing overlay with the added slave MDS based on hash values of file names under the directory.

16. The method according to claim 15, further comprising:

when the file creation rate of the directory drops below a preset merge threshold, removing a slave MDS from the consistent hashing overlay and merging the directory entries of the slave MDS to be removed to a successor MDS remaining in the consistent hashing overlay.

17. The method according to claim 15, further comprising:

calculating a file creation rate of a directory;

checking whether a status of the directory is split or normal which means not split;

if the status is normal and if the file creation rate is greater than the preset split threshold, then splitting the directory entries for the directory to the consistent hashing overlay based on hash values of file names under the directory; and

if the status is split and if the file creation rate is greater than the preset split threshold, then sending a request to the master MDS of the directory to add a slave MDS to the consistent hashing overlay if the MDS of the directory entry split module is not the master MDS, and adding a slave MDS to the consistent hashing overlay if the MDS of the directory entry split module is the master MDS.

18. The method according to claim 17, further comprising:

maintaining in each MDS a global consistent hashing table which stores information of all the MDSs,

wherein splitting the directory entries comprises;

selecting one or more other MDSs as slave MDSs from the global consistent hashing table, wherein the number of slave MDSs to be selected is calculated by rounding up a value of a ratio of (the file creation rate/the preset split threshold) to a next integer value and subtracting 1;

creating the same directory to each of the selected slave MDSs; and

splitting the directory entries to the selected slave MDSs based on the hash values of file names under the directory, by first sending only the hash values of the file names to the slave MDSs and migrating files corresponding to the file names later;

wherein files can be created in parallel to the master MDS and the slave MDSs as soon as hash values have been sent to the slave MDSs.

19. The method according to claim 15, wherein the master MDS of the directory has a quota equal to 1 and wherein each slave MDS has a quota equal to a ratio between capability of the slave MDS to capability of the master MDS, the method further comprising:

calculating a file creation rate of a directory;

checking whether a status of the directory is split or normal which means not split;

if the status is normal and if the file creation rate is greater than the preset split threshold multiplied by the quota of the MDS of the directory entry split module, then splitting the directory entries for the directory to the consistent hashing overlay based on hash values of file names under the directory; and

if the status is split and if the file creation rate is greater than the preset split threshold multiplied by the quota of the MDS of the directory entry split module, then sending a request to the master MDS of the directory to add a slave MDS to the consistent hashing overlay.

20. The method according to claim 19, further comprising:

maintaining in each MDS a global consistent hashing table which stores information of all the MDSs;

wherein adding a new slave MDS into the consistent hashing overlay comprises selecting the new slave MDS from the global consistent hashing table, assigning to the new slave MDS a unique ID representing an ID range of hash values in the consistent hashing overlay to be managed by the new slave MDS, adding the new slave MDS to the consistent hashing overlay, and, if the new slave MDS is added in response to a request from another MDS, then sending a reply with the unique ID and an IP address of the new slave MDS to the MDS which sent the request for adding the new slave MDS;

wherein merging directory entries of a MDS which is to be removed to a successor MDS comprises sending the IP address of the successor MDS to the MDS which is to be removed, and removing the MDS from the consistent hashing overlay; and

wherein the unique ID assigned to the new slave MDS represents an ID range of hash values equal to a portion of the ID range of hash values, which is managed by the MDS that sent the request for adding the new slave MDS, prior to adding the new slave MDS, such that a ratio of the portion of the ID range to be managed by the new slave MDS and a remaining portion of the ID range to be managed by the MDS that sent the request is equal to a ratio of the quota of the new slave MDS and the quota of the MDS that sent the request.