Method and Apparatus for Data Transmission

• US 20160150001A1
• Filed: 01/28/2016
• Published: 05/26/2016
• Est. Priority Date: 12/03/2013
• Status: Active Grant

• Current Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Inventors: ZOU, Fangming, LIU, Yang
• Granted Patent: US 10,212,217 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/08 Key distribution or management, e.g. generation, sharing or updating, of cryptographic keys or passwords network architectures or network communication protocols for supporting key management in a packet data network H04L63/06

Abstract

A data transmission method and apparatus avoids uneven loading of stored data, supports allocation of storage spaces with load balancing, and supports fast access to data. The method may include receiving, by a distributed storage system, data and a corresponding data key sent by a client. Nodes are allocated to the data according to load conditions of nodes in the distributed storage system. The data is stored in the corresponding nodes. A constructed key is generated for the data, which includes the data key and node routing information corresponding to a location where the data is stored. The constructed key is returned to the client. The client uses the constructed key to acquire the stored data. The distributed storage system receives the constructed key, retrieves the stored data based on the routing information in the constructed key and sends the data to the client.

Claims

1. A method for data transmission, the method comprising:

receiving, by a distributed storage system, data sent by a client and a key corresponding to the data;

allocating one or more storage nodes of the distributed storage system to the data according to load conditions of the storage nodes;

storing the data in one or more allocated storage nodes;

generating a constructed key for the data, comprising the key corresponding to the data and node routing information relative to a location where the data is stored; and

returning the constructed key for the data to the client.

2. The method according to claim 1, wherein the step of allocating the one or more storage nodes comprises:

acquiring, in real-time, storage space remaining capacities and system computing resource consumptions for the storage nodes of the distributed storage system; and

allocating the one or more storage nodes to the data according to a rule that is based on the real-time storage space remaining capacities and the real-time system computing resource consumptions of the storage nodes of the distributed storage system.

3. The method according to claim 2, wherein the rule comprises:

in instances when the real-time storage space remaining capacities are substantially the same in available storage nodes, selecting one or more storage nodes with smaller real-time system computing resource consumptions relative to other of the multiple storage nodes, to be utilized as the one or more storage nodes allocated to the data;

in instances when the real-time system computing resource consumptions are substantially the same in available storage nodes, selecting one or more storage nodes with larger real-time storage space remaining capacities relative to other of the available storage nodes, to be utilized as the one or more storage nodes allocated to the data;

or

in instances when the real-time storage space remaining capacities and the real-time system computing resource consumptions are substantially different in available storage nodes, selecting one or more storage nodes with larger real-time storage space remaining capacities relative to other of the available storage nodes, to be utilized as the one or more storage nodes allocated to the data.

4. The method according to claim 1, wherein the step of generating a constructed key for the data comprises:

utilizing the key corresponding to the data and the node routing information relative to a location where the data is stored to populate a constructed key structure, and encoding the constructed key structure to determine the constructed key.

5. The method according to claim 1, wherein the method further comprises:

receiving a data acquisition request sent by the client, wherein the data acquisition request comprises the constructed key for the data;

reading the data from the one or more storage nodes according to the node routing information in the constructed key for the data; and

returning the read data to the client.

6. The method according to claim 5, wherein the step of reading data from the one or more storage nodes according to the node routing information in the constructed key for the data comprises:

decoding and authenticating the constructed key;

extracting the node routing information from the decoded constructed key; and

reading data from the one or more storage nodes according to the node routing information.

7. The method according to claim 6, wherein the step of authenticating the constructed key comprises:

checking whether the decoded constructed key comprises a constructed key symbol, wherein the constructed key symbol is used for identifying that a data structure having the symbol is the constructed key.

8. The method according to claim 1, wherein the constructed key further comprises:

a constructed key symbol, a constructed key check field, a Cyclic Redundancy Code (CRC) check value or an identity of the client.

9. The method according to claim 1, wherein the client performs:

acquiring the data and the key corresponding to the data;

sending the data and the key corresponding to the data to the distributed storage system; and

receiving the constructed key for the data returned by the distributed storage system wherein the constructed key comprises the key corresponding to the data and the node routing information relative to a location where the data is stored.

10. The method according to claim 9, wherein the client performs:

sending the data acquisition request to the distributed storage system, wherein the data acquisition request comprises the constructed key for the data that comprises the key corresponding to the data and the node routing information relative to the location where the data is stored, and wherein the constructed key is used by the distributed storage system to read the data from the one or more allocated storage nodes according to the node routing information in the constructed key, and return the read data to the client.

11. The method according to claim 10, wherein the client performs:

receiving the data returned by the distributed storage system, wherein the data is read by the distributed storage system from one or more storage nodes based on the node routing information in the constructed key.

12. An apparatus for data transmission in a distributed storage system, wherein said apparatus comprises one or more hardware processors and/or circuits that are operable to:

in a data receiving module, receive data sent by a client and a key corresponding to the data;

in a node management module, allocate one or more storage nodes to the data according to load conditions of the storage nodes, and store the data in one or more allocated storage nodes;

in a constructed key generation module, generate a constructed key for the data, comprising the key corresponding to the data and node routing information relative to a location where the data is stored; and

in a data returning module, return the constructed key for the data to the client.

13. The apparatus according to claim 12, wherein the one or more hardware processors and/or circuits are operable to:

in the node management module, acquire, in real-time, storage space remaining capacities and system computing resource consumptions of the storage nodes of the distributed storage system, wherein the allocation of the one or more storage nodes to the data is based on a rule according to the real-time storage space remaining capacities and the real-time system computing resource consumptions of the storage nodes.

14. The apparatus according to claim 13, wherein the rule comprises:

in instances when the real-time storage space remaining capacities are substantially the same in available storage nodes of the distributed storage system, select one or more storage nodes with smaller real-time system computing resource consumptions relative to other of the available storage nodes, to be utilized as the one or more storage nodes allocated to the data;

in instances when the real-time system computing resource consumptions are substantially the same in available storage nodes of the distributed storage system, select one or more storage nodes with larger real-time storage space remaining capacities relative to other of the available storage nodes, to be utilized as the one or more storage nodes allocated to the data;

or

in instances when the real-time storage space remaining capacities and the real-time system computing resource consumptions are substantially different in available storage nodes of the distributed storage system, select one or more storage nodes with larger real-time storage space remaining capacities relative to other of the available storage nodes, to be utilized as the one or more storage nodes allocated to the data.

15. The apparatus according to claim 12, wherein the one or more hardware processors and/or circuits are operable to:

in the constructed key generation module, utilize the key corresponding to the data and the node routing information relative to the location where the data is stored to populate a constructed key structure, and encode the constructed key structure to determine the constructed key.

16. The apparatus according to claim 12, wherein the one or more hardware processors and/or circuits are operable to:

in a request receiving module, receive a data acquisition request sent by the client, wherein the data acquisition request comprises the constructed key for the data;

in a data reading module, read the data from the one or more storage nodes according to the node routing information in the constructed key; and

in the data returning module, return the read data to the client.

17. The apparatus according to claim 16, wherein the one or more hardware processors and/or circuits are operable to:

in an authenticating and decoding unit of the data reading module, decode and authenticate the constructed key;

in a routing extraction unit of the data reading module, extract the node routing information from the decoded constructed key; and

in a data reading unit of the data reading module, read the data from the one or more storage nodes according to the node routing information.

18. The apparatus according to claim 17, wherein the one or more processors and/or circuits are operable to:

in the authenticating and decoding unit, check whether the decoded constructed key comprises a constructed key symbol, wherein the constructed key symbol is used for identifying that a data structure having the symbol is the constructed key.

19. The apparatus according to claim 13, wherein the constructed key further comprises:

a constructed key symbol, a constructed key check field, a Cyclic Redundancy Check (CRC) check value, or an identity of the client.

20. An apparatus for data transmission in a client, wherein said apparatus comprises one or more hardware processors and/or circuits that are operable to:

in a data acquisition module, acquire data to be transmitted and a key corresponding to the data;

in a data sending module, send the data and the key corresponding to the data to a distributed storage system, wherein the distributed storage system allocates one or more storage nodes to the data according to load conditions of the storage nodes, stores the data in one or more allocated storage nodes, and generates a constructed key for the data, wherein the constructed key comprises the key corresponding to the data and node routing information relative to a location where the data is stored; and

in a data receiving module, receive the constructed key for the data returned by the distributed storage system.

21. The apparatus according to claim 20, wherein the one or more hardware processors and/or circuits are operable to:

in a request sending module, send a data acquisition request to the distributed storage system, wherein the data acquisition request comprises the constructed key for the data, and wherein the distributed storage system reads the data from the one or more storage nodes according to the node routing information in the constructed key, and returns the read data to the client.

22. The apparatus according to claim 21, wherein the one or more hardware processors and/or circuits are operable to:

in a data receiving module, receive the data returned by the distributed storage system.