Data processing method and apparatus

• US 9,952,778 B2
• Filed: 05/04/2017
• Issued: 04/24/2018
• Est. Priority Date: 11/05/2014
• Status: Active Grant

• Current Assignee: Xfusion Digital Technologies Company Limited
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Luo, Xiong
• Primary Examiner(s): Dennison, Jerry B
• Application Number:
  US15/587,051
• Publication Number:
  US 20170235501A1
• Time in Patent Office: 355 Days
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 65/40 Support for services or applications
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A data processing technology is provided, and is applied to a partition management device. The partition management device stores a partition view, the partition view records a correspondence between an ID of a current partition and an address of a storage disk, and a total quantity of current partitions may be less than a total quantity of final partitions. By using the technology, data forwarding may be performed on key-value data by using a current partition, thereby reducing complexity of a partition view.

Claims

1. A data processing method, which is applied to a partition management device that manages storage disks in a cluster by current partitions, the method comprising:

storing, by the partition management device, a partition view that maps an ID of a current partition to an address of a storage disk;

obtaining a key from key-value data that comprises a value and the key corresponding to the value, and calculating, according to the key, an ID of a final partition that corresponds to the key-value data, wherein the key-value data comprises a value and the key corresponding to the value;

calculating an ID of the current partition that corresponds with the IDs of the final partition, wherein the ID of current partition corresponds with a plurality of IDs of final partitions, wherein a total quantity of current partitions is less than a total quantity of final partitions;

searching the partition view to obtain an address of a storage disk that is corresponding to the ID of the current partition; and

generating a key-value packet by using the address of the storage disk as a destination address, and sending the key-value packet to the storage disk, wherein the key-value packet carries the key-value data.

2. The data processing method according to claim 1, further comprising:

recording a total quantity L of final partitions;

calculating, according to the key, the ID of the final partition that is corresponding to the key-value data by;

performing a hash operation on the key to obtain a hash value of the key, and

performing a modulo operation on the hash value according to the total quantity L of final partitions, wherein a remainder is used as the ID of the final partition.

3. The data processing method according to claim 1, further comprising:

recording a total quantity T of current partitions, and the calculating the ID of the current partition by;

performing a modulo operation on the ID of the final partition according to the total quantity T of current partitions, wherein a remainder is used as the ID of the current partition, and the ID of the final partition is an integer greater than or equal to 0.

4. The data processing method according to claim 1, wherein:

the final partition is a sub-partition of the current partition, and is obtained through splitting of the current partition.

5. The data processing method according to claim 1, wherein the data processing method is specifically a data write method, and before the obtaining the key in key-value data, the method further comprises:

dividing to-be-written data, to obtain a value set comprising the value, and generating the key of the value, to form the key-value data.

11. The data processing method according to claim 1, comprising:

splitting at least one current partition into more current partitions when total quantity of current partitions is not enough.

12. The data processing method according to claim 1, comprising:

acquiring a quantity M of current storage disks in the cluster and a total quantity T of current partitions in the cluster when it is detected that N new storage disks are to join the cluster, wherein M, N, and T are all natural numbers;

determining whether a mathematical relationship between the total quantity T of current partitions and a total quantity M+N of storage disks meets a first condition; and

splitting at least one of the current partition into more current partitions, when average current partitions per storage disk is less than a first threshold, the total quantity of partitions obtained after the splitting is not greater than a total quantity of final partitions supported by the cluster; and

allocating the partitions obtained after the splitting to the M+N storage disks in the cluster.

13. The data processing method according to claim 12, wherein:

after splitting, the average current partitions per storage disk is larger than the first threshold and less than second threshold.

6. A data processing device, wherein the data processing device comprises:

a memory, configured to store a partition view, wherein the partition view records a correspondence between the ID of the current partition and the address of the storage disk;

an interface, configured to provide an external connection;

a computer-readable medium, configured to store a computer program; and

a processor, connected to the memory, the interface, and the computer-readable medium, and configured to execute the following steps by running the program;

obtaining a key in key-value data, and calculating, according to the key, the ID of the final partition that is corresponding to the key-value data, wherein the key-value data comprises a value and the key corresponding to the value;

calculating the ID of the current partition that is corresponding to the ID of the final partition, wherein an ID of each current partition is corresponding to IDs of a plurality of final partitions;

searching the partition view to obtain an address of a storage disk that is corresponding to the ID of the current partition; and

generating a key-value packet by using the address of the storage disk as a destination address, and sending the key-value packet to the storage disk through the interface, wherein the key-value packet carries the key-value data.

7. The data processing device according to claim 6, wherein the memory is further configured to record a total quantity L of final partitions, and the calculating, according to the key, the ID of a final partition that is corresponding to the key-value data specifically comprises:

performing a hash operation on the key to obtain a hash value of the key, and performing a modulo operation on the hash value according to the total quantity L of final partitions, wherein a remainder is used as the ID of the final partition.

8. The data processing device according to claim 6, wherein the memory is further configured to record a total quantity T of current partitions, and the calculating the ID of the current partition that is corresponding to the ID of the final partition specifically comprises:

performing a modulo operation on the ID of the final partition according to the total quantity T of current partitions, wherein a remainder is used as the ID of the current partition, and the ID of the final partition is an integer greater than or equal to 0.

9. The data processing device according to claim 6, wherein:

the final partition is a sub-partition of the current partition, and is obtained through splitting of the current partition.

10. The data processing device according to claim 7, wherein before the obtaining the key in key-value data, the processor is further configured to execute the following step:

dividing to-be-written data, to obtain a value set comprising the value, and generating the key of the value, to form the key-value data.

14. The data processing device according to claim 6, the processor is configured to execute step of:

splitting at least one current partition into more current partitions when total quantity of current partitions is not enough.

15. The data processing device according to claim 6, the processor is configured to execute steps of:

acquiring a quantity M of current storage disks in the cluster and a total quantity T of current partitions in the cluster when it is detected that N new storage disks are to join the cluster, wherein M, N, and T are all natural numbers;

determining whether a mathematical relationship between the total quantity T of current partitions and a total quantity M+N of storage disks meets a first condition; and

splitting at least one of the current partition into more current partitions, when average current partitions per storage disk is less than a firist threshold, the total quantity of partitions obtained after the splitting is not greater than a total quantity of final partitions supported by the cluster; and

allocating the partitions obtained after the splitting to the M+N storage disks in the cluster.

16. The data processing device according to claim 15, wherein:

after splitting, the average current partitions per storage disk is larger than the first threshold and less than second threshold.