Data processing method and device in distributed file storage system

• US 10,127,233 B2
• Filed: 07/22/2015
• Issued: 11/13/2018
• Est. Priority Date: 12/31/2013
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Guo, Hongxing
• Primary Examiner(s): Willis, Amanda L
• Assistant Examiner(s): Shah, Vaishali
• Application Number:
  US14/806,064
• Publication Number:
  US 20150324371A1
• Time in Patent Office: 1,210 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 16/116 Details of conversion of file system types or formats
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers

Abstract

A data processing method and a device in a distributed file storage system, where the method includes receiving, by a client agent, a data processing request which carries a file identifier, an offset address, a file length, and other information of a target file; obtaining, by the client agent, redundancy information according to the file identifier carried in the data processing request, where the redundancy information includes a quantity of data strips, N, of the distributed file storage system and a quantity of parity strips, M, of the distributed file storage system; determining a quantity of valid strips, DSC, of the target file according to the offset address and the length information; determining a quantity of actual strips, N′, of the target file according to the DSC and the M; and determining corresponding strips according to the N′ and processing the corresponding strips.

Claims

1. A data writing method, comprising:

receiving a data processing request from a host, the data processing request carrying a data identifier and a target data;

obtaining a redundancy information according to the data identifier, the redundancy information indicating a N+M redundancy protection for the target data, N being a default quantity of data strips, M being a default quantity of parity strips, N being a positive integer, and M being a positive integer;

dividing the target data into DSC data strips, DSC being less than N, and DSC being a positive integer;

combining at least M-DSC+1 empty data strips with the DSC data strips to obtain N′

actual data strips when DSC is less than M, M being less than N′

, and N′

being less than N;

using the DSC data strips as the N′

actual data strips when DSC greater than M;

generating M parity strips from the N′

actual data strips; and

writing the N′

actual data strips and the M parity strips to a distributed data storage system.

2. The method of claim 1, wherein M and N are separately stored in a corresponding directory information table.

3. The method of claim 1, further comprising adding consistency label information and information about a quantity of the DSC data strips to the N′

actual data strips and the M parity strips.

8. The method of claim 1, wherein the target data is a file, and the data identifier is an identifier of the file.

9. The method of claim 1, wherein at least one of the DSC data strips is partially filled with data from the target data.

4. A server for data processing, comprising:

a memory comprising instructions; and

a processor coupled to the memory, the processor being configured to execute the instructions to;

receive a data processing request from a host, the data processing request carrying a data identifier and a target data;

obtain a redundancy information according to the data identifier, the redundancy information indicating a N+M redundancy protection for the target data, N being a default quantity of data strips, M being a default quantity of parity strips, N being a positive integer, and M being a positive integer;

divide the target data into DSC data strips, DSC being less than N, and DSC being a positive integer;

combine at least M-DSC+1 empty data strips with the DSC data strips to obtain N′

actual data strips when DSC is less than M, M being less than N′

, and N′

being less than N;

use the DSC data strips as the N′

actual data strips when DSC greater than M;

generate M parity strips from the N′

actual data strips; and

send the N′

actual data strips and the M parity strips to a distributed data storage system.

5. The server of claim 4, wherein the server is an application server coupled to the distributed data storage system, and the distributed data storage system comprises a plurality of storage server nodes.

6. The server of claim 5, wherein the processor is further configured to add consistency label information to the N′

actual data strips and the M parity strips, and the consistency label information is a timestamp.

7. The server of claim 4, wherein M and N are separately stored in a corresponding directory information table.

10. The server of claim 4, wherein the target data is a file, and the data identifier is an identifier of the file.

11. The server of claim 4, wherein the server is an application server coupled to the distributed data storage system, and the distributed data storage system comprises a plurality of storage server nodes that implement a version of redundant array of independent disks (RAID).

12. The server of claim 5, wherein the processor is further configured to add consistency label information to the N′

actual data strips and the M parity strips, and the consistency label information is a version number.

13. The server of claim 5, wherein at least one of the DSC data strips is partially filled with data from the target data.