WORKLOAD-AWARE DISTRIBUTED DATA PROCESSING APPARATUS AND METHOD FOR PROCESSING LARGE DATA BASED ON HARDWARE ACCELERATION

• US 20130227244A1
• Filed: 08/27/2012
• Published: 08/29/2013
• Est. Priority Date: 02/27/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jung, Myung-June, Lee, Ju-Pyung
• Granted Patent: US 8,898,422 B2
• Time in Patent Office: Days
• US Class Current: 711/173
• CPC Class Codes: G06F 2209/501 Performance criteria
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

A workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration are provided. The data processing apparatus includes a memory buffer including partitions. The data processing apparatus further includes a partition unit configured to distribute a mapping result to the partitions based on a partition proportion scheme. The data processing apparatus further includes a reduce node configured to receive content of a corresponding one of the partitions, and perform a reduction operation on the content to generate a reduce result.

Claims

1. A data processing apparatus comprising:

a memory buffer comprising partitions;

a partition unit configured to distribute a mapping result to the partitions based on a partition proportion scheme; and

a reduce node configured to receive content of a corresponding one of the partitions, and perform a reduction operation on the content to generate a reduce result.

2. The data processing apparatus of claim 1, further comprising:

a job controller configured to measure a processing performance of the reduce node, and establish the partition proportion scheme based on the processing performance.

3. The data processing apparatus of claim 1, wherein:

the mapping result comprises a key; and

the partition unit comprises

a main processor configured to generate an intermediate code with a fixed size based on the key, and

a proportion controller configured to determine the partitions based on the intermediate code and the partition proportion scheme, generate partition codes corresponding to the respective partitions.

4. The data processing apparatus of claim 3, further comprising:

a data transfer controller configured to write the mapping result in the partitions based on the partition codes.

5. The data processing apparatus of claim 1, further comprising:

a sorter configured to sort the content, and output the sorted content to the reduce node.

6. The data processing apparatus of claim 1, further comprising:

a memory buffer management table configured to store memory buffer management information.

7. The data processing apparatus of claim 6, wherein the memory buffer management information comprises a start address of the memory buffer, or a number of partitions of the memory buffer, or a number of records remaining to be processed, or a memory section size, or information of a memory section header, or a partition lookup table, or any combination thereof.

8. The data processing apparatus of claim 1, further comprising:

a mapping node configured to process input data in parallel to generate the mapping result.

9. The data processing apparatus of claim 8, wherein:

the mapping result comprises a key; and

the mapping node is further configured to

discard a bit other than a least significant bit of the key to update the key if a length of the key is greater than the least significant bit, and

add a padding bit to the least significant bit to update the key if the length of the key is less than the least significant bit.

10. The data processing apparatus of claim 1, wherein the data processing apparatus is implemented through hardware acceleration on a field programmable gate array (FPGA).

11. A data processing method comprising:

distributing a mapping result to partitions of a memory buffer based on a partition proportion scheme;

determining content of a corresponding one of the partitions; and

performing a reduction operation on the content to generate a reduce result.

12. The data processing method of claim 11, further comprising:

measuring a processing performance of a reduce node; and

establishing the partition proportion scheme based on the processing performance.

13. The data processing method of claim 12, further comprising:

extracting a job environment;

acquiring a list of reduce nodes from the job environment; and

determining the reduce node to be measured for the processing performance based on the list of the reduce nodes.

14. The data processing method of claim 12, wherein the measuring of the processing performance comprises:

extracting first information about a number of records being processed by the reduce node;

extracting second information about the number of records being processed by the reduce node after a predetermined period of time; and

determining the processing performance based on the first information, the second information, and the predetermined period of time.

15. The data processing method of claim 12, wherein the establishing of the partition proportion scheme comprises:

determining a proportion for each of the partitions based on the processing performance.

16. The data processing method of claim 11, wherein:

the mapping result comprises a key; and

the distributing of the mapping result comprises

generating an intermediate code with a fixed size based on the key,

determining the partitions based on the intermediate code and the partition proportion scheme, and

generating partition codes corresponding to the respective partitions.

17. The data processing method of claim 16, wherein the distributing of the mapping result further comprises:

writing the mapping result in the partitions based on the partition codes.

18. The data processing method of claim 11, wherein the determining of the content comprises:

sorting the content.

19. The data processing method of claim 11, further comprising:

processing input data in parallel to generate the mapping result.

20. The data processing method of claim 11, wherein the data processing method is implemented through hardware acceleration on a field programmable gate array (FPGA).