Distributed storage method and system

• US 10,447,763 B2
• Filed: 12/08/2016
• Issued: 10/15/2019
• Est. Priority Date: 12/08/2016
• Status: Expired due to Fees

• Current Assignee: Nanning Fugui Precision Industrial Co., Ltd. (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Nanning Fugui Precision Industrial Co., Ltd. (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Huang, Cheng-Yi, Su, Chih-Wei
• Primary Examiner(s): Jacobs, Lashonda T
• Application Number:
  US15/372,405
• Publication Number:
  US 20180167439A1
• Time in Patent Office: 1,041 Days
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

The present disclosure recites a distributed storage method applied in a distributed file system, and the method including steps: splitting the original file into data blocks and storing in a data block queue; assigning multiple data nodes to form a first pipe to transmit a first data block from the data block queue, and when the first one of the data nodes in the first pipe have been written in the data block, assigning another multiple data nodes to form a second pipe to transmit a second data block from the data block queue; transmitting the pipes when the overall data nodes of the first and second pipes been written in the first and second data blocks. The present disclosure also provides a distributed storage system, and all of them can improve the transmission speed of the distributed file system.

Claims

1. A distributed storage system, applied in a distributed file system, wherein the distributed file system comprises data nodes, the distributed storage system comprises:

at least one processor;

a non-transitory storage system coupled to the at least one processor and configured to store one or more programs, wherein the one or more programs are executed by the at least one processor, the one or more programs including instructions for;

splitting an original file into data blocks;

storing the data blocks in a data block queue;

assigning multiple data nodes to form a first pipe for transmitting a first data block from the data block queue, and when the first one of the data nodes in the first pipe have been written in the data block, assigning another multiple data node to form a second pipe for transmitting a second data block from the data block queue;

transmitting the pipes when the overall data nodes of the first and second pipes have been written in the first and second data blocks; and

reading the first and second data block from the first and second pipes, and when the first and second pipes are transmitted to a target area of the distributed file system, dissolving the first and second pipes to release the overall data nodes.

2. The system as claimed in claim 1, wherein the distributed file system comprises a name node, wherein the one or more programs further include instructions for:

applying the data nodes to the name node;

storing the data nodes in a data node queue; and

sorting the data nodes of the data node queue as a range according to the response time, and assigning the data nodes sorted in the front of a certain interval of the range to each of the pipes as the first one of the data nodes in each of the pipes sequentially.

3. The system as claimed in claim 2, wherein the one or more programs further include instructions for:

assigning the data node sorted outside the front of a certain interval of the range to each of the pipes as the one and except the first one of the data nodes in each of the pipes randomly.

4. The system as claimed in claim 2, wherein the number of the data nodes to be applied is determined by the number of the data nodes in a pipe, the number of pipes transmitted simultaneously, the number of data blocks and the probability of invalid data nodes.

5. The system as claimed in claim 1, wherein the number of the data nodes in a pipe is at least 3, and the maximum number of the pipes transmitting simultaneously is set according to the transmission bandwidth.

6. A distributed storage method applied in a distributed file system, wherein the distributed file system comprises data node, the method comprises:

splitting an original file into data blocks;

storing the data blocks in a data block queue;

assigning multiple data nodes to form a first pipe for transmitting a first data block from the data block queue, and when the first one of the data nodes in the first pipe have been written in the data block, assigning another multiple data node to form a second pipe for transmitting a second data block from the data block queue;

transmitting the pipes when the overall data nodes of the first and second pipes have been written in the first and second data blocks; and

reading the first and second data block from the first and second pipes, and when the first and second pipes are transmitted to a target area of the distributed file system, dissolving the first and second pipes to release the overall data nodes.

7. The method as claimed in claim 6, the distributed file system comprises a name node, wherein assigning multiple data nodes comprises:

applying the data nodes to the name node;

storing the data nodes in a data node queue; and

sorting the data nodes of the data node queue as a range according to the response time, and assigning the data nodes sorted in the front of a certain interval of the range to each of the pipes as the first one of the data nodes in each of the pipes sequentially.

8. The method as claimed in claim 7, wherein assigning multiple data nodes further comprises:

assigning the data nodes sorted outside the front of a certain interval of the range to each of the pipes as the one and except the first one of the data nodes in each of the pipes randomly.

9. The method as claimed in claim 7, wherein the number of the data nodes to be applied is determined by the number of the data nodes in a pipe, the number of pipes transmitted simultaneously, the number of data blocks and the probability of invalid data nodes.

10. The method as claimed in claim 6, wherein the number of the data nodes in a pipe is at least 3, and the maximum number of the pipes transmitting simultaneously is set according to the transmission bandwidth.