DISK FAULT TOLERANCE METHOD, DEVICE AND SYSTEM

• US 20140229763A1
• Filed: 04/22/2014
• Published: 08/14/2014
• Est. Priority Date: 01/22/2013
• Status: Active Grant

• Current Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Inventors: LOU, Jibing
• Granted Patent: US 9,477,565 B2
• Time in Patent Office: Days
• US Class Current: 714/6.23
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups

Abstract

The present invention relates to disk fault tolerance method, device and system which relates to a field of computer. The method comprises: receiving a file access request; acquiring one or more node lists corresponding to the file, wherein each node list comprises at least two nodes, and the mutually corresponding disks between the respective nodes store the same contents; accessing data chunks included in the file from the respective nodes selected from each of the node lists; and when an accessing from a certain node fails, selecting another node from the node list that comprises the certain node, and accessing a respective data chunk included in the file from the selected node according to the identifier of the file. The present invention proposes a multilevel pairing storage technology in combination with advantages of the dual machine duplexing mechanism, wherein a plurality of completely independent machines are mutually backed up from a node level to a disk level. This solves problems of data read-write fault, a disk fault, single node fault and the like in a large-scale distributed storage system, ensures a data reliability in any scene, and improves a disk availability and read-write performance.

Claims

1. A disk fault tolerance method, comprising:

receiving a file access request, in which an identifier of a file is carried;

according to a pre-stored correspondence between identifiers of files and node lists as well as the identifier of the file, acquiring one or more node lists corresponding to the file, wherein each node list comprises at least two nodes, each node comprises a plurality of disks, there is a one-to-one correspondence between the disks included in the respective nodes in a single node list, and the mutually corresponding disks between the respective nodes store the same contents;

selecting a node from each of the acquired node lists respectively, and accessing data chunks included in the file from respective nodes selected from each of the node lists according to the identifier of the file; and

when an accessing from a certain node fails, selecting another node from the node list that comprises the certain node, and accessing a respective data chunk included in the file from the selected node according to the identifier of the file.

2. The method according to claim 1, characterized in that, the step of accessing data chunks included in the file from the respective nodes selected from each of the node lists according to the identifier of the file, comprises:

acquiring a correspondence between disk identifiers and the identifiers of the files corresponding to the node lists;

acquiring identifiers of disks for storing the file from the correspondence between the disk identifiers and the identifiers of the files corresponding to the node lists, according to the identifier of the file; and

according to the identifier of the file and the disk identifiers for storing the file, accessing data chunks included in the file from the respective nodes selected from the node lists.

3. The method according to claim 1, characterized in that, wherein the same data chunks are stored in the mutually corresponding disks between the respective nodes, and the same data chunks included in the mutually corresponding disks between the respective nodes have a same offset

4. The method according to claim 3, characterized in that, the step of accessing data chunks included in the file from the respective nodes selected from each of the node lists according to the identifier of the file, comprises:

acquiring a correspondence of the identifiers of the files corresponding to the node lists, disk identifiers and the offsets of the data chunks;

according to the identifier of the file, acquiring identifiers of disks for storing the file and the offsets for storing the data chunks included in the file in the disk from the correspondence of the identifiers of the files corresponding to the node lists, the disk identifiers and the offsets of the data chunks; and

according to the identifiers of disks for storing the file and the offsets for storing the data chunks included in the file in the disk, accessing data chunks included in the file from the respective nodes selected from the node lists.

5. The method according to claim 1, characterized in that, the method further comprises the steps of:

receiving a file writing request, in which a file to be written is carried, wherein the file to be written can include the identifier, size, content, type information of the file;

according to the identifier of the file to be written, acquiring one or more node lists corresponding to the file to be written from the stored correspondence between the identifiers of the files and the node lists;

according to the one or more acquired node lists, dividing the file to be written into one or more data chunks, wherein the divided data chunks are in a one-to-one correspondence with the node lists; and

in each node included in every node list, determining identifiers of the disks for storing the data chunks included in the file, and writing the data chunks included in the file into disks corresponding to the disk identifiers.

6. The method according to claim 5, characterized in that, after determining identifiers of the disks for storing the data chunks included in the file, the method further comprises steps of:

according to the identifier of the file to be written and identifiers of the disks for storing the data chunks included in the file, generating a correspondence between the identifier of the file and the disk identifiers; and

storing the correspondence between the identifier of the file and the disk identifiers.

7. The method according to claim 5, characterized in that, after determining identifiers of the disks for storing the data chunks included in the file, the method further comprises steps of:

according to identifiers of the disks for storing the data chunks included in the file, determining an offset in the disks for storing the data chunks included in the file; and

writing the data chunks included in the file into disks corresponding to the disk identifiers, which further comprises;

according to the offset for storing the data chunks included in the file, storing the data chunks included in the file into data chucks of the disk corresponding to the disk identifier.

8. The method according to claim 7, characterized in that, after determining an offset in the disk for storing the data chunks included in the file, the method further comprises steps of:

according to the identifier of the file to be written, the identifiers of disks for storing the data chunks included in the file and the offsets for storing the data chunks included in the file, generating a correspondence of the identifier of the file, the disk identifiers and the offsets of the data chunks; and

storing the correspondence of the identifier of the file, the disk identifiers and the offsets of the data chunks.

9. A disk fault tolerance device, comprising:

a receiving module, for receiving a file access request, in which an identifier of a file is carried;

a first acquiring module, configured to acquire one or more node lists corresponding to the file according to a pre-stored correspondence between identifiers of the files and node lists as well as the identifier of the file, wherein each node list comprises at least two nodes, each node comprises a plurality of disks, there is a one-to-one correspondence between the disks included in respective nodes, and the mutually corresponding disks between the respective nodes store same contents;

an accessing module, configured to select a node from each of the acquired node lists respectively, and access data chunks included in the file from the respective nodes selected from each of the node lists according to the identifier of the file; and

a fault tolerance module, configured to select another node from the node list that comprises the certain node, and access a respective data chunk included in the file from the selected node according to the identifier of the file, when an accessing from a certain node fails.

10. The device according to claim 9, characterized in that, the accessing module comprises:

a selecting unit, configured to select the respective nodes from each of the acquired node lists; and

an accessing unit, which comprises;

a first acquiring subunit, configured to acquire a correspondence between disk identifiers and the identifiers of the files corresponding to the node lists;

a second acquiring subunit, configured to acquire identifiers of disks for storing the file from the correspondence between the disk identifiers and the identifiers of the files corresponding to the node lists, according to the identifier of the file; and

an accessing subunit, configured to access data chunks included in the file from the respective nodes selected by the selecting unit, according to the identifier of the file and the disk identifiers for storing the file.

11. The device according to claim 9, characterized in that, the mutually corresponding disks between the respective nodes store the same data chunk, and the same data chunk included in the mutually corresponding disks between the respective nodes has the same offset.

12. The device according to claim 11, characterized in that, the accessing module comprises:

a selecting unit, configured to select a node from each of the acquired node lists respectively;

an accessing unit, which comprises;

a first acquiring subunit, configured to acquire a correspondence of the identifiers of the files corresponding to the node lists, disk identifiers and the offsets of the data chunks;

a second acquiring subunit, configured to acquire identifiers of disks for storing the file and the offsets for storing the data chunks included in the file in the disk from the correspondence of the identifiers of the files corresponding to the node lists, the disk identifiers and the offsets of the data chunks, according to the identifier of the file; and

an accessing subunit, configured to access data chunks included in the file from the respective nodes selected by the selecting unit, according to the identifiers of disks for storing the file and the offsets for storing the data chunks included in the file in the disk.

13. The device according to claim 10, characterized in that, the device further comprises:

a receiving module, configured to receive a file writing request, in which a file to be written is carried, wherein the file to be written includes the identifier, size, contents, and type information of the file;

a second acquiring module, configured to acquire one or more node lists corresponding to the file to be written from the stored correspondence between the identifiers of the files and the node lists, according to an identifier of the file to be written;

a dividing module, configured to divide the file to be written into one or more data chunks, according to the one or more acquired node lists, wherein the divided data chunks are in a one-to-one correspondence with the node lists; and

a writing module, configured to determine an identifier of disk for storing the data chunks included in the file, and write the data chunks included in the file into a disk corresponding to the disk identifier, in each node included in every node list.

14. The device according to claim 13, characterized in that the device further comprises a storage module, which is configured to generate a correspondence between the identifier of the file and the disk identifiers, and store the correspondence between the identifier of the file and the disk identifiers, according to the identifier of the file to be written and identifiers of the disks for storing the data chunks included in the file.

15. The device according to claim 13, characterized in that, the writing module comprises:

a determining unit, configured to determine an identifier of disk for storing the data chunks included in the file in each node included in every node list, and determine an offset in the disk for storing the data chunks included in the file according to the identifier of disk for storing the data chunks included in the file; and

a writing unit, configured to store the data chunks included in the file into data chucks of the disk corresponding to the disk identifier, according to the offset for storing the data chunks included in the file.

16. The device according to claim 15, characterized in that, the device further comprises a storage module, the storage module is configured to:

according to an identifier of the file to be written, identifiers of disks for storing the data chunks included in the file and offsets for storing the data chunks included in the file, generate a correspondence of the identifier of the file, the disk identifiers and the offsets of the data chunks; and

store the correspondence of the identifier of the file, the disk identifiers and the offsets of the data chunks.

17. A disk fault tolerance system, characterized in that the system comprises a client, a storage server cluster and a management server;

the client is configured to send a file access request to the management server, wherein an identifier of a file is carried in the file access request;

the storage server cluster comprises a plurality of storage server groups, wherein each storage server group comprises at least two storage servers, each storage server comprises a plurality of disks, there is a one-to-one correspondence between the disks included in the respective storage servers, and the mutually corresponding disks between the respective storage servers store the same contents; and

the management server is configured to;

receive the file access request sent from the client;

acquire one or more storage server groups corresponding to the file according to a pre-stored correspondence between identifiers of the files and the storage server groups included in the storage server cluster as well as the identifier of the file;

select a storage server from each of the acquired storage server groups respectively, and access data chunks included in the file from the respective storage servers selected from each of the storage server groups according to the identifier of the file; and

select another storage server from the storage server group that comprises a certain storage server, and access a respective data chunk included in the file from the selected storage server according to the identifier of the file, when an accessing from the certain storage server fails.

18. The system according to claim 17, characterized in that, the management server is also configured to:

acquire a correspondence between disk identifiers and the identifiers of the files corresponding to the storage server groups;

acquire identifiers of disks for storing the file from the correspondence between the disk identifiers and the identifiers of the files corresponding to the storage server groups, according to the identifier of the file; and

access data chunks included in the file from the respective storage servers selected from the storage server groups according to the identifier of the file and the disk identifiers for storing the file.

19. The system according to claim 17, characterized in that, the storage server cluster is also configured to store the same data chunks in the mutually corresponding disks between the respective storage servers, wherein the same data chunks included in the mutually corresponding disks between the respective storage servers have a same offset.

20. The system according to claim 19, characterized in that, the management server is also configured to:

acquire a correspondence of the identifiers of the files corresponding to the storage server groups, disk identifiers and the offsets of the data chunks;

according to the identifier of the file, acquire identifiers of disks for storing the file and the offsets for storing the data chunks included in the file in the disk from the correspondence of the identifiers of the files corresponding to the storage server groups, the disk identifiers and the offsets of the data chunks; and

access data chunks included in the file from the respective storage servers selected from each of the storage server groups, according to the identifiers of disks for storing the file and the offsets for storing the data chunks included in the file in the disk.

21. The system according to claim 17, characterized in that, the client is also configured to send a file writing request to the management server, wherein a file to be written is carried in the file writing request, the file to be written includes the identifier, size, contents, type information of the file;

accordingly, the management server is also configured to;

receive the file writing request sent from the client;

according to an identifier of the file to be written, acquire one or more storage server groups corresponding to the file to be written, wherein the management server stores a correspondence between identifiers of the files and the storage server groups;

according to one or more acquired storage server groups, dividing the file to be written into one or more data chunks, wherein the divided data chunks are in a one-to-one correspondence with the storage server groups; and

in each storage server included in every storage server group, determine an identifier of disk for storing the data chunks included in the file, and write the data chunks included in the file into a disk corresponding to the disk identifier.

22. The system according to claim 21, characterized in that, the management server is further configured to generate and store a correspondence between the identifier of the file and the disk identifiers, according to the identifier of the file to be written and the identifiers of the disks for storing the data chunks included in the file.

23. The system according to claim 21, characterized in that, the management server is further configured to determine an identifier of disk for storing the data chunks included in the file in each storage server included in every storage server group, and to determine an offset in the disk for storing the data chunks included in the file, according to the identifier of disk for storing the data chunks included in the file;

and to store the data chunks included in the file into data chucks of the disk corresponding to the disk identifier, according to the offset for storing the data chunks included in the file.

24. The system according to claim 23, characterized in that, the management server is further configured to generate and store a correspondence of the identifier of the file, the disk identifiers and the offsets of the data chunks, according to the identifier of the file to be written, the identifiers of disks for storing the data chunks included in the file and the offsets for storing the data chunks included in the file.

25. The system according to claim 17, characterized in that the system further comprises a backup management server, wherein the backup management server is configured to replace the function of the management server when the management server malfunctions.