ELECTRONIC SYSTEM WITH DATA EXCHANGE MECHANISM AND METHOD OF OPERATION THEREOF

• US 20170212783A1
• Filed: 05/06/2016
• Published: 07/27/2017
• Est. Priority Date: 01/22/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Choi, Inseok Stephen, Ki, Yang Seok
• Granted Patent: US 10,268,521 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

A electronic system includes: a cluster manager configured to: divide a user program into a group of parallel execution tasks, and generate shuffling metadata to map intermediate data and processed data from the parallel execution tasks; a shuffling cluster node, coupled to the cluster manager, configured to: store the shuffling metadata by an in-storage computer (ISC), and incrementally shuffle each of the sub-packets of the intermediate data and the processed data, by the ISC, based on the shuffling metadata when the parallel execution task is in process; and a local storage, coupled to the shuffling cluster node and mapped through the shuffling metadata, for receiving the sub-packets of the processed data from the shuffling cluster node.

Claims

1. An electronic system comprising:

a cluster manager configured to;

divide a user program into a group of parallel execution tasks, and generate shuffling metadata to map intermediate data and processed data from the parallel execution tasks;

a shuffling cluster node, coupled to the cluster manager, configured to;

store the shuffling metadata by an in-storage computer (ISC), and incrementally shuffle each of the sub-packets of the intermediate data and the processed data, by the ISC, based on the shuffling metadata when the parallel execution task is in process; and

a local storage, coupled to the shuffling cluster node and mapped through the shuffling metadata, for receiving the sub-packets of the processed data from the shuffling cluster node.

2. The system as claimed in claim 1 wherein the cluster manager is configured to transmit the shuffling metadata to the ISC including a destination address provided for each of the intermediate data sub-packets generated by the sub-tasks of the parallel execution task.

3. The system as claimed in claim 1 wherein the shuffling cluster node is configured as a pre-processing client to generate the intermediate data.

4. The system as claimed in claim 1 wherein the shuffling cluster node is configured as a post-processing client to generate the processed data.

5. The system as claimed in claim 1 wherein the cluster manager configured to divide the user program into the group of the parallel execution tasks includes a plurality of sub-tasks defined for each of the parallel execution tasks.

6. The system as claimed in claim 1 wherein the shuffling cluster node includes an in-storage compute (ISC), coupled to a node storage engine, configured to monitor sub-tasks of the parallel execution tasks and transfer a data sub-packet of the intermediate data based on the shuffling metadata.

7. The system as claimed in claim 1 wherein the shuffling cluster node includes a node central processing unit (CPU), coupled to a node storage engine, configured to execute the parallel execution tasks.

8. The system as claimed in claim 1 wherein the shuffling cluster node includes the in-storage computer (ISC) configured to receive the shuffling metadata from a shuffling control unit.

9. The system as claimed in claim 1 wherein the shuffling cluster node includes an in-storage computer (ISC) configured to:

monitor sub-tasks of the parallel execution tasks;

transfer a data sub-packet of the intermediate data, based on a destination address from the shuffling metadata, at the completion of each of the sub-tasks; and

receive the data sub-packet as an input for an initial sub-task of a subsequent task without a data shuffle interval.

10. The system as claimed in claim 1 wherein the shuffling cluster node is configured as a post-processing client and includes the in-storage computer (ISC) configured to transfer processed data to an output file in the local storage.

11. A method of operation of an electronic system comprising:

configuring a cluster manager for;

dividing a user program into a group of parallel execution tasks, and generating shuffling metadata for mapping intermediate data and processed data from the parallel execution tasks;

configuring a shuffling cluster node for;

storing the shuffling metadata, and

incrementally shuffling each of the sub-packets of the intermediate data and the processed data based on the shuffling metadata when processing the parallel execution task; and

mapping a local storage, through the shuffling metadata, for receiving the sub-packets of the processed data from the shuffling cluster node.

12. The method as claimed in claim 11 wherein dividing the user program into the group of the parallel execution tasks includes defining a group of sub-tasks within each of the parallel execution tasks.

13. The method as claimed in claim 11 wherein coupling the shuffling cluster node includes configuring the shuffling cluster node as a pre-processing client for generating the intermediate data.

14. The method as claimed in claim 11 wherein coupling the shuffling cluster node includes configuring the shuffling cluster node as a post-processing client for generating the processed data.

15. The method as claimed in claim 11 wherein dividing the user program into the group of the parallel execution tasks includes defining a plurality of sub-tasks for each of the parallel execution tasks.

16. The method as claimed in claim 11 wherein monitoring sub-tasks of the parallel execution tasks and transferring a data sub-packet of the intermediate data based on the shuffling metadata.

17. The method as claimed in claim 11 wherein configuring the shuffling cluster node for executing the parallel execution tasks including accessing a node storage engine by a node central processing unit (CPU).

18. The method as claimed in claim 11 wherein configuring the shuffling cluster node for storing the shuffling metadata includes receiving the shuffling metadata by an in-storage computer (ISC).

19. The method as claimed in claim 11 further comprising:

monitoring sub-tasks of the parallel execution tasks;

transferring a data sub-packet of the intermediate data based on a destination address from the shuffling metadata at the completion of each of the sub-tasks; and

receiving the data sub-packet as an input for an initial sub-task of a subsequent task without a data shuffle interval.

20. The method as claimed in claim 11 wherein transferring processed data to an output file in the local storage includes configuring an in-storage computer (ISC) for transferring the processed data.