METHOD AND SYSTEM FOR DATA TRANSFER BETWEEN COMPUTE CLUSTERS AND FILE SYSTEM

• US 20140351300A1
• Filed: 10/03/2013
• Published: 11/27/2014
• Est. Priority Date: 05/24/2013
• Status: Active Grant

• Current Assignee: DataDirect Networks Incorporated
• Original Assignee: DataDirect Networks Incorporated
• Inventors: PISZCZEK, MICHAEL, UPPU, PAVAN KUMAR, COPE, JASON MICAH, NOWOCZYNSKI, PAUL
• Granted Patent: US 9,628,299 B2
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: H04L 12/6418 Hybrid transport
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]

Abstract

A data migrating system and method are provided in which a Burst Buffer Network Aggregator (BBNA) process is configured either on the File Servers or on the File System'"'"'s dedicated I/O nodes to coalesce data fragments stored in participating Burst Buffer nodes under the direction of a primary BB node appointed by a data generating entity prior to transfer of the full data stripe into the File System. The “write” request in the form of a full data stripe is distributed into a plurality of data fragments among participating BB nodes along with corresponding metadata. The primary BB node gathers the metadata from the participating BB nodes, sends the metadata list to the BBNA unit, responsive to which the BBNA unit allocates a buffer sufficient to store the full data stripe, and transfers data fragments from participating BB nodes into the full data stripe buffer, thereby coalescing the data fragments into the full data stripe, which is subsequently transferred from the buffer in the BBNA unit into the File System.

Claims

1. A method for data migration between data generating entities and a File System, comprising the steps of:

(a) operatively coupling a Burst Buffer (BB) tier between at least one data generating entity and a File System, and configuring said BB tier with a plurality of Burst Buffer (BB) nodes, each operating with data input speed exceeding the data input speed of said File System;

(b) configuring said File System with at least one Burst Buffer Network Aggregator (BBNA) unit;

(c) sending a first data “

write”

request from said at least one data generating entity to said BB tier;

(d) distributing said data into data fragments, and storing each data fragment and metadata associated therewith in a respective participating BB node from said plurality of BB nodes in said BB tier;

(e) assigning a primary BB node from said participating BB nodes for servicing said first data “

write”

request;

(f) gathering, by said primary BB node, metadata for said data fragments residing in said participating BB nodes, and generating a metadata list;

(g) sending a command along with said metadata list from said primary BB node to said at least one BBNA unit;

(h) subsequently, allocating, by said at least one BBNA unit, a full data stripe buffer sufficient to accommodate said first data “

write”

request;

(i) transferring said data fragments from said participating BB nodes into said full data stripe buffer in accordance with said metadata list, thereby coalescing said data fragments into a full data stripe in said full data stripe buffer; and

(j) transferring said full data stripe from said full data stripe buffer of said at least one BBNA unit into said File System.

2. The method of claim 1, further comprising the step of:

prior to said step (j), issuing a second data “

write”

request by said at least one BBNA unit to said File System.

3. The method of claim 1, further comprising the step of:

upon completion of said data transfer in said step (j), reporting, by said at least one BBNA unit to said primary BB node.

4. The method of claim 1, wherein each of said BB nodes allocates a byte addressable buffer, further comprising the steps of:

servicing said first data “

write”

request in a BBNA Target Mode, and

prior to said step (i), issuing a remote memory “

READ”

request by said at least one BBNA unit to read contents of said byte addressable buffer of each said BB node.

5. The method of claim 4, further comprising the steps of:

prior to said step (i), signaling, by said primary BB node to said plurality of participating BB nodes, to move data fragments residing thereat to said byte addressable buffers.

6. The method of claim 5, further comprising the step of:

in said step (f), gathering scattered information about said byte addressable buffers by said primary BB node.

7. The method of claim 6, further comprising the step of:

after said step (i), creating, by said primary BB node, a command including BB nodes ID, buffer descriptor, address, size, and a RDMA (Remote Direct Memory Access) key on said byte addressable buffers.

8. The method of claim 1, wherein said File System includes a plurality of File Servers, further comprising the steps of:

in said step (b), configuring said at least one BBNA unit on at least one of said plurality of File Servers.

9. The method of claim 1, further comprising the steps of:

operatively coupling at least one File System Input/Output node between said at least one data generating entity and said File System,

dedicating said at least one File System Input/Output node for interaction with said File System, and

in said step (b), configuring said at least one BBNA unit on said at least one File System Input/Output node.

10. The method of claim 1, further comprising the steps of:

servicing said first data “

write”

request in a BBNA Host Mode,

prior to said step (i), issuing a “

READ”

request by said at least one BBNA unit to said participating BB nodes,

responsive to said “

READ”

request, pushing data fragments residing at said participating BB nodes to said full data stripe buffer allocated by said at least one BBNA unit.

11. A method for data migration between data generating entities and a file system, comprising the steps of:

(a) operatively coupling a Burst Buffer (BB) tier between at least one data generating entity and a File System, and configuring said BB tier with a plurality of BB nodes, each operating with data input speed exceeding the data input speed of said File System;

(b) configuring said File System with at least one Burst Buffer Network Aggregator (BBNA) unit;

(c) sending a first data “

READ”

request from said at least one data generating entity;

(d) allocating, by said at least one BBNA unit, a full data stripe buffer sufficient to service said first data “

READ”

request, and issuing a second “

READ”

request to said File System to read said data therefrom to said allocated full data stripe buffer;

(e) upon completion of said second “

READ”

request by said File System, issuing, by said at least one BBNA unit, a Remote Memory “

write”

request, and

(f) transferring said data from said allocated full data stripe buffer to a destination BB node selected from said plurality of BB nodes.

12. The method of claim 11, further comprising the step of:

prior to said step (f), issuing, by a primary BB node from said plurality thereof, a “

read”

command indicating said destination BB node.

13. The method of claim 12, further comprising the step of:

upon completion of said data transfer to said destination BB node in said step (f), reporting, by said at least one BBNA unit, to said primary BB node.

14. A system for data migration between data generating entities and a File System, comprising:

a Burst Buffer (BB) tier operatively coupled between at least one data generating entity and the File System, wherein said Burst Buffer tier includes a plurality of BB nodes having data input speed exceeding the data input speed of said File System;

a plurality of File Servers operatively coupled to said BB tier;

at least one Burst Buffer Network Aggregator (BBNA) unit configured at said File System in operative coupling to said plurality of BB nodes, said at least one BBNA unit being configured to coalesce data fragments stored in participating BB nodes from said plurality thereof into a full data stripe under the direction of at least one of said participating BB nodes prior to transfer said full data stripe into said File System.

15. The system of claim 14, wherein said at least one BBNA unit resides on a respective one of said plurality of File Servers.

16. The system of claim 14, wherein said File System further comprises a plurality of File System Input/Output (FSIO) nodes operatively coupled to said plurality of data generating entities and dedicated for said File System interaction, wherein said at least one BBNA unit resides on a respective one of said plurality of FSIO nodes.

17. The system of claim 14, wherein each of said plurality of BB nodes is configured to store at least one data fragment of said full data stripe and metadata associated therewith,

wherein said at least one BB node is configured to gather said metadata from said plurality of participating BB nodes, and to generate a metadata list on data fragments residing in said participating BB nodes, and

wherein said at least one BBNA unit is configured to allocate a full data stripe buffer sufficient to store said full data stripe formed of said data fragments upon receipt of said metadata list from said at least one BB node, and to subsequently transfer said data fragments from said participating BB nodes to said full data stripe buffer prior to transferring said full data stripe to said File System.

18. The system of claim 17, further comprising an array of data storage units operatively coupled to said plurality of File Servers.

19. The system of claim 14, wherein each of said BB nodes is configured with a byte addressable buffer, and

wherein said at least one BBNA unit is configured to read contents of said byte addressable buffer.

20. The system of claim 14, wherein, for servicing a “

READ”

request from at least one data generating entity, said at least one BBNA unit is configured to;

(a) allocate a buffer sufficient to accommodate said “

READ”

request;

(b) transfer said requested data from said File System to said buffer; and

(c) subsequently transfer said data from said buffer to at least one destination BB node indicated by said at least one BB node.

21. The system of claim 14, further comprising a High Speed Network intercoupled between said data generating entities and said BB tier.

22. The system of claim 14, further comprising a Storage Fabric intercoupled between said BB tier and said File System.

23. The system of claim 14, further comprising a plurality of I/O nodes between said data generating entities and said File System, wherein each of said plurality of I/O nodes is augmented with a respective Burst Buffer unit.