Directory-level referral method for parallel NFS with multiple metadata servers

• US 9,342,529 B2
• Filed: 12/28/2012
• Issued: 05/17/2016
• Est. Priority Date: 12/28/2012
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Shiga, Kenta, Lin, Wujuan, Mysur, Kaushik, Ikeda, Hirokazu
• Primary Examiner(s): VU, BAI DUC
• Application Number:
  US13/729,256
• Publication Number:
  US 20140188953A1
• Time in Patent Office: 1,236 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097

Abstract

An aspect of the invention is directed to a plurality of MDSs (metadata servers) in a distributed storage system which includes data servers storing file contents, each MDS having a processor and a memory and storing file system metadata. Directories of a file system namespace are distributed to the MDSs through referral directories referring to real directories using hash value of inode number of each of the referral directories. During a process to create a directory in the file system namespace, a first MDS dynamically creates a referral directory in the first MDS, and creates a real directory in a second MDS, the referral directory in the first MDS referring to the real directory in the second MDS by maintaining location information of the real directory in the second MDS, the real directory in the second MDS containing the file system metadata for the directory in the file system namespace.

Claims

1. A plurality of MDSs (metadata servers) in a distributed storage system which includes data servers storing file contents, each MDS:

having a processor and a memory and storing file system metadata,

wherein directories of a file system namespace are distributed to the MDSs through referral directories referring to real directories using hash value of inode number of each of the referral directories;

wherein during a process to create a directory in the file system namespace, a first MDS dynamically creates a referral directory in the first MDS, and creates a real directory in a second MDS, the referral directory in the first MDS referring to the real directory in the second MDS by maintaining location information of the real directory in the second MDS, the real directory in the second MDS containing the file system metadata for the directory in the file system namespace;

wherein each directory in the MDSs has an inode, the inode including an inode number, a mode of “

referral”

for a referral directory and “

normal”

for a directory which is not a referral directory, a file system identifier, and a location entry;

wherein the file system identifier uniquely identifies the referral directory if the mode is referral;

wherein the location entry has an IP address of the MDS in which the real directory is located and a hash value of the inode number if the mode is referral;

wherein the inode of a directory further includes a count which is a number of MDSs to at least some of which sub-directories of the directory are to be distributed, and a MDS-List which is a list of IP addresses of MDSs to at least some of which the sub-directories are to be distributed; and

wherein the count and MDS-List are determined based on depth and width of the directory.

2. The plurality of MDSs according to claim 1,

wherein the location information of the referral directory includes a hash value for the real directory and an IP address of the second MDS;

and

wherein the second MDS maps the hash value for the real directory to a local directory path in a local namespace of the second MDS to the real directory.

3. The plurality of MDSs according to claim 1,

wherein the first MDS has a parent directory and creates the referral directory under the parent directory to refer to the real directory in the second MDS.

4. The plurality of MDSs according to claim 3,

wherein the first MDS constructs an inode for the referral directory which refers to the real directory, the inode for the referral directory including an inode number, a file system identifier, and a location entry which has an IP address of the second MDS in which the real directory is located and a hash value of the inode number.

5. The plurality of MDSs according to claim 1,

wherein each MDS includes a referral-to-path mapping table to store entries of a hash value of any real directory in the MDS, a corresponding local path where the real directory is stored in a local namespace of the MDS, and a corresponding IP address of a source MDS where the referral directory referring to the real directory is located.

6. The plurality of MDSs according to claim 5,

wherein each MDS maintains workload information to monitor access workload for each directory in the referral-to-path mapping table of the MDS.

7. The plurality of MDSs according to claim 1,

wherein the first MDS in the distributed storage system periodically collects utilization information from one or more of other MDSs and broadcasts consolidated utilization information to all the other MDSs.

8. The plurality of MDSs according to claim 1,

wherein any MDS, which has a real directory being referred to from a referral directory and has utilization higher than a preset threshold, migrates the real directory including sub-tree of the real directory to another MDS which has utilization sufficiently low to receive migration of the real directory with the sub-tree, and informs a source MDS where the referral directory referring to the real directory is located of the updated location information.

9. The plurality of MDSs according to claim 8,

wherein each MDS includes a referral-to-path mapping table to store entries of a hash value of any real directory in the MDS, a corresponding local path where the real directory is stored in a local namespace of the MDS, and a corresponding IP address of a source MDS where the referral directory referring to the real directory is located;

and

wherein, after migration of a real directory from one MDS to another MDS, the one MDS and said another MDS update respective referral-to-path mapping tables and update respective workload information for the respective MDSs based on the migration.

10. The plurality of MDSs according to claim 8,

wherein if the sub-tree of the real directory to be migrated includes another real directory being referred to from another referral directory, the said another real directory is not migrated but remains in the same MDS before migration of the sub-tree.

11. The plurality of MDSs according to claim 1,

wherein the inode in each directory in the MDSs has a unique inode number.

12. The plurality of MDSs according to claim 1,

wherein each directory is assigned a unique inode number.

13. A method of managing directories in a distributed storage system which includes a plurality of MDSs (metadata servers) storing file system metadata and data servers storing file contents, the method comprising:

distributing directories of a file system namespace to the MDSs through referral directories referring to real directories using hash value of inode number of each of the referral directories;

wherein during a process to create a directory in the file system namespace, a first MDS dynamically creates a referral directory in the first MDS, and creates a real directory in a second MDS, the referral directory in the first MDS referring to the real directory in the second MDS by maintaining location information of the real directory in the second MDS, the real directory in the second MDS containing the file system metadata for the directory in the file system namespace;

wherein each directory in the MDSs has an inode, the inode including an inode number, a mode of “

referral”

for a referral directory and “

normal”

for a directory which is not a referral directory, a file system identifier, and a location entry;

wherein the file system identifier uniquely identifies the referral directory if the mode is referral;

wherein the location entry has an IP address of the MDS in which the real directory is located and a hash value of the inode number if the mode is referral;

wherein the inode of a directory further includes a count which is a number of MDSs to at least some of which sub-directories of the directory are to be distributed, and a MDS-List which is a list of IP addresses of MDSs to at least some of which the sub-directories are to be distributed; and

wherein the count and MDS-List are determined based on depth and width of the directory.

14. The method according to claim 13, wherein the location information of the referral directory includes a hash value for the real directory and an IP address of the second MDS, the method further comprising:

mapping, by the second MDS, the hash value for the real directory to a local directory path in a local namespace of the second MDS to the real directory.

15. The method according to claim 13, wherein the first MDS has a parent directory and creates the referral directory under the parent directory to refer to the real directory in the second MDS, the method further comprising:

constructing, by the first MDS, an inode for the referral directory which refers to the real directory, the inode for the referral directory including an inode number, a file system identifier, and a location entry which has an IP address of the second MDS in which the real directory is located and a hash value of the inode number.

16. The method according to claim 13, wherein each MDS includes a referral-to-path mapping table to store entries of a hash value of any real directory in the MDS, a corresponding local path where the real directory is stored in a local namespace of the MDS, and a corresponding IP address of a source MDS where the referral directory referring to the real directory is located, the method further comprising:

maintaining, by each MDS, workload information to monitor access workload for each directory in the referral-to-path mapping table of the MDS.

17. The method according to claim 13, further comprising:

periodically collecting, by the first MDS in the distributed storage system, utilization information from one or more of other MDSs, and broadcasting consolidated utilization information to all the other MDSs.

18. The method according to claim 13, further comprising:

migrating, by any MDS which has a real directory being referred to from a referral directory and has utilization higher than a preset threshold, the real directory including sub-tree of the real directory to another MDS which has utilization sufficiently low to receive migration of the real directory with the sub-tree, and informing a source MDS where the referral directory referring to the real directory is located of the updated location information.

19. The method according to claim 13,

wherein the inode in each directory in the MDSs has a unique inode number.

20. The method according to claim 13, wherein each directory is assigned a unique inode number.