Apparatus and method for controlling distributed memory cluster

• US 8,984,085 B2
• Filed: 02/14/2012
• Issued: 03/17/2015
• Est. Priority Date: 02/14/2011
• Status: Active Grant

• Current Assignee: KT Corporation
• Original Assignee: KT Corporation
• Inventors: Jung, Ki-Young
• Primary Examiner(s): Gillis, Brian J
• Assistant Examiner(s): Lin, Steve
• Application Number:
  US13/396,329
• Publication Number:
  US 20120209943A1
• Time in Patent Office: 1,127 Days
• Field of Search: 709/213, 707/752, 707/822
• US Class Current: 709/213
• CPC Class Codes: G06F 15/161 Computing infrastructure, e.g. computer clusters, blade chassis or hardware partitioning casings, cabinets, racks or drawers for data centers H05K5/00
• CPC Class Codes: G06F 15/167 using a common memory, e.g. mailbox

Abstract

Provided are an apparatus and method for controlling a distributed memory cluster. A distributed computing system may include a computing node cluster, a distributed memory cluster, and a controlling node. The computing node cluster may include a plurality of computing nodes including first computing nodes that each generates associated data. The distributed memory cluster may be configured to store the associated data of the first computing nodes. The controlling node may be configured to select memory blocks of the associated data for distribution on the distributed memory cluster based on a node selection rule and memory cluster structure information, and to select second computing nodes from the computing node cluster based on a location selection rule and the memory cluster structure information.

Claims

1. A distributed computing system comprising:

a computing node cluster including a plurality of computing nodes comprising first computing nodes that each generates associated data;

a distributed memory cluster configured to store the associated data of the first computing nodes;

a controlling node configured to select memory blocks of the associated data for distribution on the distributed memory cluster based on a node selection rule and memory cluster structure information, and to select second computing nodes from the computing node cluster based on a location selection rule and the memory cluster structure information,

wherein the controller node is configured to select, as one of the selected second computing nodes, a computing node included in a server rack storing the associated data greater than a given data amount, and to control the selected computing node to process the data stored in the same server rack.

2. The distributed computing system of claim 1, wherein according to the node selection rule, the controller node is configured to:

assign the associated data to server racks included in the distributed memory cluster; and

select the memory blocks from each of the server racks up to a number of the generated data assigned to each server rack.

3. The distributed computing system of claim 2, wherein the controller node is configured to use a round robin scheme to assign the associated data to the server racks.

4. The distributed computing system of claim 2, wherein the controlling node is configured to select, from memory blocks included in each of the server racks, a memory block having an available capacity greater than a given capacity.

5. The distributed computing system of claim 2, wherein the memory blocks are included in at least one storage node included in each of the server racks.

6. The distributed computing system of claim 1, wherein the controlling node is further configured to:

control the selected computing node, as one of the second computing nodes, to process the data stored in another one of the server racks storing the associated data less than the given data amount.

7. The distributed computing system of claim 1, wherein:

the distributed computing system performs a MapReduce operation;

the first computing nodes perform a map function, and the associated data of the first computing nodes are intermediate data; and

the second computing nodes perform a reduce function.

8. The distributed computing system of claim 7, wherein:

a number of memory blocks required to store the generated data is determined based on partition information generated by a partition function of the MapReduce operation;

a number of second computing nodes is determined to be equivalent to a number of storing nodes including the memory blocks required to store the generated data based on the partition information.

9. The distributed computing system of claim 1, wherein the memory cluster structure information includes identifiers of server racks included in the distributed memory cluster, identifiers of storage nodes included in the server racks, identifiers of memory blocks included in the storage nodes, identifiers of the first computing nodes generating the data, identifiers of the second computing node processing the stored data, and server rack switch information.

10. A method for controlling a distributed memory cluster having a plurality of server racks on a distributed computing system, the method comprising:

generating associated data from first computing nodes;

selecting memory blocks distributed on the distributed memory cluster based on a node selection rule and memory cluster structure information;

storing the generated, associated data at the selected memory blocks;

selecting second computing nodes based on a location selection rule and the memory cluster structure information; and

controlling the selected second computing node to process the data stored in the selected memory block,

wherein the selecting second computing nodes includes;

selecting, as the one of the second computing nodes, a computing node included in a server rack storing the generated, associated data greater than a given data amount; and

controlling the selected computing node, as one of the second computing nodes, to process the data stored in the given server rack.

11. The method of claim 10, wherein the node selection rule of the selecting memory blocks includes:

assigning the generated, associated data to server racks included in the distributed memory cluster; and

selecting the memory blocks from each of the server racks up to a number of the generated, associated data assigned to each of the server racks.

12. The method of claim 11, wherein in the assigning the generated, associated data to server racks, a round robin scheme is employed.

13. The method of claim 11, wherein in the selecting the memory blocks from each of the server racks, a memory block having an available capacity of at least a given capacity is selected from memory blocks included in each server rack.

14. The method of claim 11, wherein the memory blocks are included in at least one storage node included in each of the server racks.

15. The method of claim 10, wherein the location selection rule of the selecting second computing nodes further includes:

controlling the selected computing node, as one of the second computing nodes, to process the data stored in another one of the server racks storing the generated data less than the given data amount.

16. The method of claim 10, wherein:

the distributed computing system performs a MapReduce operation;

the first computing nodes perform a map function and the generated, associated data of the first computing nodes are intermediate data; and

the second computing nodes perform a reduce function.

17. The method of claim 16, wherein:

a number of memory blocks required to store the generated, associated data is determined based on partition information generated by a partition function of the MapReduce operation;

a number of second computing nodes is determined to be equivalent to a number of storing nodes including the memory blocks required to store the generated data based on the partition information.

18. The method of claim 10, wherein the memory cluster structure information includes identifiers of server racks included in the distributed memory cluster, identifiers of storage nodes included in the server racks, identifiers of memory blocks included in the storage nodes, identifiers of the first computing nodes generating the data, identifiers of the second computing node processing the stored data, and server rack switch information.