Shuffle embedded distributed storage system supporting virtual merge and method thereof

• US 10,135,926 B2
• Filed: 10/21/2015
• Issued: 11/20/2018
• Est. Priority Date: 06/09/2015
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Jin, Ki Sung, Kim, Young Chang, Kim, Young Kyun, Kim, Hong Yeon, Choi, Wan
• Primary Examiner(s): Le, Hung
• Application Number:
  US14/919,135
• Publication Number:
  US 20160366225A1
• Time in Patent Office: 1,126 Days
• Field of Search: None
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Provided herein is a shuffle embedded distributed storage system and method supporting virtual merge, the system and method including a distributed shared storage configured to store a virtual merged file; a plurality of map servers connected to the distributed shared storage via a network, and configured to perform a map function and record a map result data computed as a result of the map function in the distributed shared storage by means of a map result file; and a plurality of reduce servers connected to the distributed shared storage and the map servers via the network, wherein the virtual merged file includes a list of the map result files recorded by the plurality of map servers, and an identifier of a reduce server to which the virtual merged file is to be transmitted.

Claims

1. A shuffle embedded distributed storage system supporting virtual merge, the system comprising:

a distributed shared storage configured to store a virtual merged file;

a plurality of map servers connected to the distributed shared storage via a network, and configured to perform a map function and record a map result data computed as a result of the map function in an aligned format in the distributed shared storage by means of a map result file; and

a plurality of reduce servers connected to the distributed shared storage and the map servers via the network for performing a reduce function on the map result files received from the virtual merged file of the distributed shared storage,

wherein the virtual merged file comprises a list of the map result files generated by the plurality of map servers, and an identifier of one of the plurality of reduce servers to which the virtual merged file is to be transmitted, wherein the map result files are registered in the virtual merged file,

wherein one or more of the plurality of map servers transmits an identifier of the virtual merged file to one or more of the plurality of the reduce servers, and

wherein, in response to receiving a request for data reading from a selected one of the plurality of reduce servers, the distributed shared storage searches the virtual merged file having an identifier that is identical to the selected reduce server, and reads and aligns data of the map result files included in the searched virtual merged file consecutively, and transmits the aligned data to the selected reduce server without merging the map result files.

2. A shuffle embedded distributed storage method supporting virtual merge, the method comprising:

reading, by a plurality of map servers, a map input file from a distributed shared storage, and performing a map function;

performing the map function, and recording a computed map result data in an aligned format in the distributed shared storage by means of a map result file;

registering information on map result files recorded by the plurality of map servers in a virtual merged file, wherein the virtual merged file comprises a list of the map result files generated by the plurality of map servers, and an identifier of one of the plurality of reduce servers to which the virtual merged file is to be transmitted,

transmitting by one or more of the plurality of map servers an identifier of the virtual merged file to one or more of the plurality of the reduce servers,

in response to receiving a request for data reading from a selected one of the plurality of reduce servers, searching the virtual merged file having an identifier that is identical to the selected reduce server, and reading and aligning data of the map result files included in the searched virtual merged file consecutively, and

transmitting the aligned data to the selected reduce server without merging the map result files.

3. The shuffle embedded distributed storage method supporting virtual merge according to claim 2, further comprising:

requesting, by the reduce server, the distributed shared storage to read data;

determining whether or not there is data remaining in map result files registered in the virtual merged file;

in response to there being data in the map result files, circulating the map result files and reading data of a certain area and accumulating the read data in a memory;

aligning the data accumulated in the memory; and

transmitting the aligned data to the reduce server.

4. The shuffle embedded distributed storage method supporting virtual merge according to claim 3, wherein, in response to there being no data remaining in the map result files, the reduce function of the reduce server is ended.

5. The shuffle embedded distributed storage method supporting virtual merge according to claim 3, wherein the map servers, distributed shared storage and reduce server are connected to one another via a network.

6. The shuffle embedded distributed storage method supporting virtual merge according to claim 3, wherein, after the information on the map result files is registered in the virtual merged file, the map servers transmit the identifier of the virtual merged file to the reduce server.