Data access with tolerance of disk fault

• US 9,477,565 B2
• Filed: 04/22/2014
• Issued: 10/25/2016
• Est. Priority Date: 01/22/2013
• Status: Active Grant

• Current Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Inventors: Lou, Jibing
• Primary Examiner(s): Mehrmanesh, Elmira
• Application Number:
  US14/259,106
• Publication Number:
  US 20140229763A1
• Time in Patent Office: 917 Days
• Field of Search: 714/6.23, 714/6.32
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups

Abstract

A method for data access is disclosed. The method includes: receiving a file access request; acquiring one or more node lists corresponding to the file, wherein each node list comprises at least two nodes, and the mutually corresponding disks between the respective nodes store the same contents; accessing data chunks included in the file from the respective nodes selected from each of the node lists; and when an accessing from a certain node fails, selecting another node from the node list that comprises the certain node, and accessing a respective data chunk included in the file from the selected node according to the identifier of the file. A system and a device with tolerance of disk fault are also disclosed.

Claims

1. A method, comprising:

receiving a file access request, in which an identifier of a file is carried, the file comprises a plurality of data chunks, and each of the data chunks is stored in at least two nodes included in a node list;

according to a pre-stored correspondence between identifiers of files and node lists as well as the identifier of the file, acquiring one or more node lists corresponding to the file, wherein each node list comprises at least two nodes, each node comprises a plurality of disks, there is a one-to-one correspondence between the disks included in the respective nodes in a single node list, and each of the mutually corresponding disks between the respective nodes stores a same divided chunk of a data chunk stored in the respective nodes, and the same divided chunk of the data chunk included in each of the mutually corresponding disks between the respective nodes has a same offset;

selecting a node from each of the acquired node lists respectively, and accessing data chunks included in the file from respective nodes selected from each of the node lists according to the identifier of the file; and

when an accessing from a certain node fails, selecting another node from the node list that comprises the certain node, and accessing a respective data chunk included in the file from the selected node according to the identifier of the file.

2. The method according to claim 1, characterized in that, accessing data chunks included in the file from the respective nodes selected from each of the node lists according to the identifier of the file, comprises:

acquiring a correspondence between disk identifiers and the identifiers of the files corresponding to the node lists;

acquiring identifiers of disks for storing the file from the correspondence between the disk identifiers and the identifiers of the files corresponding to the node lists, according to the identifier of the file; and

according to the identifier of the file and the disk identifiers for storing the file, accessing data chunks included in the file from the respective nodes selected from the node lists.

3. The method according to claim 1, characterized in that, accessing data chunks included in the file from the respective nodes selected from each of the node lists according to the identifier of the file, comprises:

acquiring a correspondence of the identifiers of the files corresponding to the node lists, disk identifiers and offsets of divided chunks;

according to the identifier of the file, acquiring identifiers of disks for storing the file and the offsets for storing the divided chunks included in the file in the disk from the correspondence of the identifiers of the files corresponding to the node lists, the disk identifiers and the offsets of the divided chunk; and

according to the identifiers of disks for storing the file and the offsets for storing the divided chunks included in the file in the disk, accessing data chunks included in the file from the respective nodes selected from the node lists.

4. The method according to claim 1, characterized in that, the method further comprises:

receiving a file writing request, in which a file to be written is carried, wherein the file to be written can include the identifier, size, content, type information of the file;

according to the identifier of the file to be written, acquiring one or more node lists corresponding to the file to be written from the stored correspondence between the identifiers of the files and the node lists;

according to the one or more acquired node lists, dividing the file to be written into one or more data chunks, wherein the divided data chunks are in a one-to-one correspondence with the node lists; and

in each node included in every node list, determining identifiers of the disks for storing divided chunks of each of the data chunks included in the file, and writing the divided chunks into disks corresponding to the disk identifiers.

5. The method according to claim 4, characterized in that after determining identifiers of the disks for storing the data chunks included in the file, the method further comprises steps of:

according to the identifier of the file to be written and identifiers of the disks for storing divided chunks of each of the data chunks included in the file, generating a correspondence between the identifier of the file and the disk identifiers; and

storing the correspondence between the identifier of the file and the disk identifiers.

6. The method according to claim 4, characterized in that, after determining identifiers of the disks for storing the divided chunks of each of the data chunks included in the file, the method further comprises steps of:

according to identifiers of the disks for storing the divided chunks of each of the data chunks included in the file, determining offsets in the disks for storing the divided chunks of each of data chunks included in the file; and

writing the divided chunks of each of the data chunks included in the file into disks corresponding to the disk identifiers, which further comprises;

according to the offsets for storing the divided chunks of each of the data chunks included in the file, storing the divided chunks of each of data chunks included in the file into the disks corresponding to the disk identifiers.

7. The method according to claim 6, characterized in that, after determining an offset in the disk for storing the divided chunks of each of the data chunks included in the file, the method further comprises steps of:

according to the identifier of the file to be written, the identifiers of disks for storing the divided chunks of each of the data chunks included in the file and the offsets for storing the divided chunks of each of the data chunks included in the file, generating a correspondence of the identifier of the file, the disk identifiers and the offsets of the divided chunks; and

storing the correspondence of the identifier of the file, the disk identifiers and the offsets of the divided chunks.

8. A device, comprising:

a receiving module, for receiving a file access request, in which an identifier of a file is carried, the file comprises a plurality of data chunks, and each of the data chunks is stored in at least two nodes included in a node list;

a first acquiring module, configured to acquire one or more node lists corresponding to the file according to a pre-stored correspondence between identifiers of the files and node lists as well as the identifier of the file, wherein each node list comprises at least two nodes, each node comprises a plurality of disks, there is a one-to-one correspondence between the disks included in respective nodes, and each of the mutually corresponding disks between the respective nodes stores a same divided chunk of a data chunk stored in the respective nodes, and the same divided chunk of the data chunk included in each of the mutually corresponding disks between the respective nodes has a same offset;

an accessing module, configured to select a node from each of the acquired node lists respectively, and access data chunks included in the file from the respective nodes selected from each of the node lists according to the identifier of the file; and

a fault tolerance module, configured to select another node from the node list that comprises a certain node, and access a respective data chunk included in the file from the selected node according to the identifier of the file, when an accessing from the certain node fails.

9. The device according to claim 8, characterized in that, the accessing module comprises:

a selecting unit, configured to select the respective nodes from each of the acquired node lists; and

an accessing unit, which comprises;

a first acquiring subunit, configured to acquire a correspondence between disk identifiers and the identifiers of the files corresponding to the node lists;

a second acquiring subunit, configured to acquire identifiers of disks for storing the file from the correspondence between the disk identifiers and the identifiers of the files corresponding to the node lists, according to the identifier of the file; and

an accessing subunit, configured to access data chunks included in the file from the respective nodes selected by the selecting unit, according to the identifier of the file and the disk identifiers for storing the file.

10. The device according to claim 9, characterized in that, the device further comprises:

a receiving module, configured to receive a file writing request, in which a file to be written is carried, wherein the file to be written includes the identifier, size, contents, and type information of the file;

a second acquiring module, configured to acquire one or more node lists corresponding to the file to be written from the stored correspondence between the identifiers of the files and the node lists, according to an identifier of the file to be written;

a dividing module, configured to divide the file to be written into one or more data chunks, according to the one or more acquired node lists, wherein the divided data chunks are in a one-to-one correspondence with the node lists; and

a writing module, configured to determine an identifier of disk for storing divided chunks of each of the data chunks included in the file, and write the divided chunks into a disk corresponding to the disk identifier, in each node included in every node list.

11. The device according to claim 10, characterized in that the device further comprises a storage module, which is configured to generate a correspondence between the identifier of the file and the disk identifiers, and store the correspondence between the identifier of the file and the disk identifiers, according to the identifier of the file to be written and identifiers of the disks for storing the divided chunks of each of the data chunks included in the file.

12. The device according to claim 8, characterized in that, the accessing module comprises:

a selecting unit, configured to select a node from each of the acquired node lists respectively;

an accessing unit, which comprises;

a first acquiring subunit, configured to acquire a correspondence of the identifiers of the files corresponding to the node lists, disk identifiers and offsets of divided chunks;

a second acquiring subunit, configured to acquire identifiers of disks for storing the file and the offsets for storing the divided chunks included in the file in the disk from the correspondence of the identifiers of the files corresponding to the node lists, the disk identifiers and the offsets of the divided chunks, according to the identifier of the file; and

an accessing subunit, configured to access data chunks included in the file from the respective nodes selected by the selecting unit, according to the identifiers of disks for storing the file and the offsets for storing the divided chunks included in the file in the disk.

13. The device according to claim 10, characterized in that, the writing module comprises:

a determining unit, configured to determine identifiers of disks for storing the divided chunks of each of the data chunks included in the file in each node included in every node list, and determine offsets in the disks for storing the divided chunks of each of the data chunks included in the file according to the identifiers of the disks for storing the divided chunks of each of the data chunks included in the file; and

a writing unit, configured to store the divided chunks of each of the data chunks included in the file into disks corresponding to the disk identifiers, according to the offsets for storing the divided chunks of each of the data chunks included in the file.

14. The device according to claim 13, characterized in that, the device further comprises a storage module, the storage module is configured to:

according to an identifier of the file to be written, identifiers of disks for storing the divided chunks of each of the data chunks included in the file and offsets for storing the divided chunks of each of the data chunks included in the file, generate a correspondence of the identifier of the file, the disk identifiers and the offsets of the divided chunks of each of the data chunks; and

store the correspondence of the identifier of the file, the disk identifiers and the offsets of the divided chunks.

15. A system, characterized in that the system comprises a client, a storage server cluster and a management server;

the client is configured to send a file access request to the management server, wherein an identifier of a file is carried in the file access request, the file comprises a plurality of data chunks, and each of the data chunks is stored in at least two nodes included in a node list;

the storage server cluster comprises a plurality of storage server groups, wherein each storage server group comprises at least two storage servers, each storage server comprises a plurality of disks, there is a one-to-one correspondence between the disks included in the respective storage servers, and each of the mutually corresponding disks between the respective storage servers stores a same divided chunk of a data chunk stored in the respective storage servers, and the same divided chunk of the data chunk included in each of the mutually corresponding disks between the respective storage servers has a same offset; and

the management server is configured to;

receive the file access request sent from the client;

acquire one or more storage server groups corresponding to the file according to a pre-stored correspondence between identifiers of the files and the storage server groups included in the storage server cluster as well as the identifier of the file;

select a storage server from each of the acquired storage server groups respectively, and access data chunks included in the file from the respective storage servers selected from each of the storage server groups according to the identifier of the file; and

select another storage server from the storage server group that comprises a certain storage server, and access a respective data chunk included in the file from the selected storage server according to the identifier of the file, when an accessing from the certain storage server fails.

16. The system according to claim 15, characterized in that, the management server is also configured to:

acquire a correspondence between disk identifiers and the identifiers of the files corresponding to the storage server groups;

acquire identifiers of disks for storing the file from the correspondence between the disk identifiers and the identifiers of the files corresponding to the storage server groups, according to the identifier of the file; and

access data chunks included in the file from the respective storage servers selected from the storage server groups according to the identifier of the file and the disk identifiers for storing the file.

17. The system according to claim 15, characterized in that, the management server is also configured to:

acquire a correspondence of the identifiers of the files corresponding to the storage server groups, disk identifiers and the offsets of divided chunks;

according to the identifier of the file, acquire identifiers of disks for storing the file and the offsets for storing the divided chunks included in the file in the disk from the correspondence of the identifiers of the files corresponding to the storage server groups, the disk identifiers and the offsets of the divided chunks; and

access data chunks included in the file from the respective storage servers selected from each of the storage server groups, according to the identifiers of disks for storing the file and the offsets for storing the divided chunks included in the file in the disk.

18. The system according to claim 15, characterized in that, the client is also configured to send a file writing request to the management server, wherein a file to be written is carried in the file writing request, the file to be written includes the identifier, size, contents, type information of the file;

accordingly, the management server is also configured to;

receive the file writing request sent from the client;

according to an identifier of the file to be written, acquire one or more storage server groups corresponding to the file to be written, wherein the management server stores a correspondence between identifiers of the files and the storage server groups;

according to one or more acquired storage server groups, dividing the file to be written into one or more data chunks, wherein the divided data chunks are in a one-to-one correspondence with the storage server groups; and

in each storage server included in every storage server group, determine an identifier of disk for storing divided chunks of each of the data chunks included in the file, and write the divided chunks into disks corresponding to the disk identifiers.

19. The system according to claim 18, characterized in that, the management server is further configured to generate and store a correspondence between the identifier of the file and the disk identifiers, according to the identifier of the file to be written and the identifiers of the disks for storing the divided chunks of each of the data chunks included in the file.

20. The system according to claim 18, characterized in that, the management server is further configured to determine identifiers of disks for storing the divided chunks of each of the data chunks included in the file in each storage server included in every storage server group, and to determine offsets in the disks for storing the divided chunks of each of the data chunks included in the file, according to the identifiers of disks for storing the divided chunks of each of the data chunks included in the file;

and to store the divided chunks of each of the data chunks included in the file into disks corresponding to the disk identifier, according to the offsets for storing the divided chunks of each of the data chunks included in the file.

21. The system according to claim 20, characterized in that, the management server is further configured to generate and store a correspondence of the identifier of the file, the disk identifiers and the offsets of the divided chunks, according to the identifier of the file to be written, the identifiers of disks for storing the divided chunks of each of the data chunks included in the file and the offsets for storing the divided chunks of each of the data chunks included in the file.

22. The system according to claim 15, characterized in that the system further comprises a backup management server, wherein the backup management server is configured to replace the function of the management server when the management server malfunctions.