Distributed file system supporting data block dispatching and file processing method thereof

• US 20110307534A1
• Filed: 11/26/2009
• Published: 12/15/2011
• Est. Priority Date: 03/25/2009
• Status: Abandoned Application

• Current Assignee: ZTE Corporation
• Original Assignee: ZTE Corporation
• Inventors: Peng, Jie, Cheng, Ning, Zhang, Bo, Wang, Chong, Xia, Jianbo
• Application Number:
  US13/202,966
• Publication Number:
  US 20110307534A1
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 16/1834 implemented based on peer-to-peer networks, e.g. gnutella p2p communication protocols H04L67/104
• CPC Class Codes: H04N 21/23109 by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/23116 involving data replication, e.g. over plural servers synchronization of replicated data G06F11/1658; error detection or correction by means of data replication G06F11/2053; replication in distributed file systems G06F16/10; replication in distributed file systems G06F16/27; replication or mirroring of data in data networks H04L67/1095
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content

Abstract

A distributed file system and a file processing method thereof are disclosed. The distributed file system comprises at least two file nodes (1, 2, . . . , n), and each file node comprises a metadata server (12, 22, . . . , n2), a file access client (11, 21, . . . , n1), a file access server (13, 23, . . . , n3) and a storage medium (14, 24, . . . , n4), and file nodes can access each other. The file processing method comprises the steps of: storing different files in each file node in a scattered way; for each file node, users of the current file node firstly accessing metadata/valid data of a required file in the file node through the file access client of the file node; if the metadata/valid data of the required file are not found in the file node, the users access other nodes through the file access clients. The present invention effectively solves the problem that a metadata server is easy to become a performance bottleneck when a great number of users access hot data concurrently, and can furthest release the storage space of some file nodes by the file processing methods, such as data block dispatching and data block aging, thereby is greatly increasing the utilization ratio.

Claims

1. A distributed file system supporting data block dispatching, comprising at least two file nodes, each file node comprising a metadata server, a file access client, a file access server and a storage medium;

wherein

the metadata server is arranged to manage metadata of a file stored in a file node to which the metadata server belongs;

the file access client is arranged to provide a calling interface for a user of the file node to which the file access client belongs, read and write metadata in the metadata server of the file node or metadata servers of other file nodes, and send a request for reading and writing relevant valid data to the file access server of the file node or file access servers of other file nodes according to the metadata;

the file access server is arranged to interact with the storage medium in the file node to which the file access server belongs, complete reading and writing of valid data, respond to the request for accessing valid data from the file access client of the file node or file access clients of other file nodes, and read relevant valid data from the storage medium of the file node, according to the metadata in the metadata server and return the valid data to the file access client(s); and

the storage medium is arranged to store the valid data of the file stored in the file node to which the storage medium belongs.

2. The distributed file system according to claim 1, further comprising a configuration unit which is arranged to configure a dependence relationship among the file nodes in the distributed file system and send the dependence relationship to each file node;

wherein, the file access client is further arranged to check the dependence relationship from the configuration unit when metadata/valid data of a required file are not found in the file node to which the file access client belongs, and determine a file node where the required file is stored.

3. The distributed file system according to claim 1, wherein the each file node further comprises a broadcast unit which is arranged to send a broadcast message to file access clients of other file nodes when the file access client of the file node fails to find the metadata/valid data of the required file in the file node, and determine the file node where the file required by the user of the file node is stored, according to responses from the file access clients, and inform the file access client of the file node.

4. The distributed file system according to claim 1, wherein the metadata server is further arranged to preset an access hotspot value, count times of accessing valid data of a file which are not stored in the file node by the user of the file node to which the metadata server belongs, copy valid data which are accessed for times more than the access hotspot value from other file nodes to the local part through the file access client of the file node, and write the valid data into the storage medium of the file node through the file access server of the file node, at the same time, create metadata relevant to the valid data in the metadata server of the file node.

5. The distributed file system according to claim 1, wherein the metadata server is further arranged to preset a minimum access value and a space utilization ratio threshold value, count times of accessing valid data of is the file which are stored in the file node by the user of the file node, and when a space utilization ratio of the storage medium of the file node exceeds the space utilization ratio threshold value, inform the storage medium of the file node to delete valid data stored therein which are accessed for times less than the minimum access value, at the same time, delete metadata, which are relevant to valid data which are accessed for times less than the minimum access value, stored in the metadata server of the file node.

11. The distributed file system according to claim 2, wherein the metadata server is further arranged to preset an access hotspot value, count times of accessing valid data of a file which are not stored in the file node by the user of the file node to which the metadata server belongs, copy valid data which are accessed for times more than the access hotspot value from other file nodes to the local part through the file access client of the file node, and write the valid data into the storage medium of the file node through the file access server of the file node, at the same time, create metadata relevant to the valid data in the metadata server of the file node.

12. The distributed file system according to claim 3, wherein the metadata server is further arranged to preset an access hotspot value, count times of accessing valid data of a file which are not stored in the file node by the user of the file node to is which the metadata server belongs, copy valid data which are accessed for times more than the access hotspot value from other file nodes to the local part through the file access client of the file node, and write the valid data into the storage medium of the file node through the file access server of the file node, at the same time, create metadata relevant to the valid data in the metadata server of the file node.

13. The distributed file system according to claim 2, wherein the metadata server is further arranged to preset a minimum access value and a space utilization ratio threshold value, count times of accessing valid data of the file which are stored in the file node by the user of the file node, and when a space utilization ratio of the storage medium of the file node exceeds the space utilization ratio threshold value, inform the storage medium of the file node to delete valid data stored therein which are accessed for times less than the minimum access value, at the same time, delete metadata, which are relevant to valid data which are accessed for times less than the minimum access value, stored in the metadata server of the file node.

14. The distributed file system according to claim 3, wherein the metadata server is further arranged to preset a minimum access value and a space utilization ratio threshold value, count times of accessing valid data of the file which are stored in the file node by the user of the file node, and when a space utilization ratio of the storage medium of the file node exceeds the space utilization ratio threshold value, inform the storage medium of the file node to delete valid data stored therein which are accessed for times less than the minimum access value, at the same time, delete metadata, which are relevant to valid data which are accessed for times less than the minimum access value, stored in the metadata server of the file node.

6. A file processing method of a distributed file system, the distributed file system comprising at least two file nodes, each file node comprising a metadata server, a file access client, a file access server and a storage medium;

the file processing method comprising;

storing different files in each file node in a scattered way;

for each file node, firstly accessing metadata/valid data of a required file in the file node by a user through the file access client of the file node;

if the metadata/valid data of the required file are not found in the file node, accessing other nodes by the user through file access clients.

7. The file processing method of a distributed file system according to claim 6, before the step of accessing the metadata/valid data of the required file, the file processing method further comprising:

configuring a dependence relationship table among the file nodes in the distributed file system, and sending the dependence relationship table to each file node;

the other file nodes are ones having a dependence relationship with the currently accessed file node.

8. The file processing method of a distributed file system according to claim 6, wherein the step of accessing other file nodes through file access clients comprises:

for each file node, if the user of the file node fails to find the metadata/valid data of the required file in the file node, sending a broadcast message to all other file nodes to inquire if they store the required file, determining a file node where the file required by the user of the file node is stored according to responses from the other file nodes, and then directly accessing the metadata/valid data of the required file in the determined file node through the file access client.

9. The file processing method of a distributed file system according to claim 6, further comprising:

presetting an access hotspot value;

for each file node, counting times of accessing valid data of the file which are not stored in the file node by the user of the file node through the file access client;

copying valid data which are accessed for times more than the access hotspot value from other file nodes to the local part through the file access client of the file node, and writing the valid data into the storage medium of the file node, at the same time, creating metadata relevant to the valid data in the metadata server of the file node.

10. The file processing method of a distributed file system according to claim 6, further comprising:

presetting a minimum access value and a space utilization ratio threshold value;

for each file node, counting times of accessing valid data of the file which are stored in the file node by the user of the file node, and when a space utilization ratio of the storage medium of the file node exceeds the space utilization ratio threshold value, deleting valid data, which are accessed for times less than the minimum access value, stored in the storage medium of the file node;

at the same time, deleting metadata, which are relevant to valid data which are accessed for times less than the minimum access value, stored in the metadata server of the file node.

15. The file processing method of a distributed file system according to claim 7, further comprising:

presetting an access hotspot value;

for each file node, counting times of accessing valid data of the file which are not stored in the file node by the user of the file node through the file access client;

copying valid data which are accessed for times more than the access hotspot value from other file nodes to the local part through the file access client of the file node, and writing the is valid data into the storage medium of the file node, at the same time, creating metadata relevant to the valid data in the metadata server of the file node.

16. The file processing method of a distributed file system according to claim 8, further comprising:

presetting an access hotspot value;

for each file node, counting times of accessing valid data of the file which are not stored in the file node by the user of the file node through the file access client;

copying valid data which are accessed for times more than the access hotspot value from other file nodes to the local part through the file access client of the file node, and writing the valid data into the storage medium of the file node, at the same time, creating metadata relevant to the valid data in the metadata server of the file node.

17. The file processing method of a distributed file system according to claim 7, further comprising:

presetting a minimum access value and a space utilization ratio threshold value;

for each file node, counting times of accessing valid data of the file which are stored in the file node by the user of the file node, and when a space utilization ratio of the storage medium of the file node exceeds the space utilization ratio threshold value, deleting valid data, which are accessed for times less than the minimum access value, stored in the storage medium of the file node;

at the same time, deleting metadata, which are relevant to valid data which are accessed for times less than the minimum access value, stored in the metadata server of the file node.

18. The file processing method of a distributed file system according to claim 8, further comprising:

presetting a minimum access value and a space utilization ratio threshold value;

for each file node, counting times of accessing valid data of the file which are stored in the file node by the user of the file node, and when a space utilization ratio of the storage medium of the file node exceeds the space utilization ratio threshold value, deleting valid data, which are accessed for times less than the minimum access value, stored in the storage medium of the file node;

at the same time, deleting metadata, which are relevant to valid data which are accessed for times less than the minimum is access value, stored in the metadata server of the file node.