Enabling dynamic job configuration in mapreduce

• US 9,766,940 B2
• Filed: 02/10/2014
• Issued: 09/19/2017
• Est. Priority Date: 02/10/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Fuller, Nicholas C., Kim, Minkyong, Li, Min, Meng, Shicong, Tan, Jian, Zeng, Liangzhao, Zhang, Li
• Primary Examiner(s): WU, BENJAMIN C
• Application Number:
  US14/176,679
• Publication Number:
  US 20150227392A1
• Time in Patent Office: 1,317 Days
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 9/466 Transaction processing
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

Methods, systems, and articles of manufacture for enabling dynamic task-level configuration in MapReduce are provided herein. A method includes generating a first set of configurations for a currently executing MapReduce job, wherein said set of configurations comprises job-level configurations and task-level configurations; dynamically modifying configurations associated with a mapper component and/or a reducer component associated with at least one ongoing map task and/or ongoing reduce task of the MapReduce job based on the generated first set of configurations; and deploying said first set of configurations to the mapper component and/or the reducer component associated with the MapReduce job.

Claims

1. A method comprising:

generating a first set of configurations for a currently executing MapReduce job, wherein said set of configurations comprises job-level configurations and task-level configurations;

dynamically modifying;

(i) a distinct set of task-level configurations of a mapper component associated with at least one ongoing map task of the MapReduce job, based on the generated first set of configurations, wherein said task-level configurations of the mapper component comprise a size of a map task input, resource allocation for a mapper component, central processing unit, memory, a size of a sorting area, and a number of threads when writing a map output to a local disk, and (ii) a distinct set of task-level configurations of a reducer component associated with at least one ongoing reduce task of the MapReduce job, based on the generated first set of configurations, wherein said task-level configurations of the reducer component comprise a number of threads when copying a map output to a reducer component, a size of a reduce task input, resource allocation of a reduce task, central processing unit, memory, and a size of a sorting area; and

deploying said modified configurations to the mapper component and the reducer component associated with the MapReduce job in accordance with one of multiple temporal deployment schedules, wherein the temporal deployment schedule is based on the contents of the modified configurations;

wherein said generating, said modifying, and said deploying are carried out by one or more computing devices.

2. The method of claim 1, comprising:

generating a set of job and/or task configuration parameter values that can be changed and rendered effective under a run-time status.

3. The method of claim 1, wherein said generating comprises periodically generating a new set of configurations for the MapReduce job.

4. The method of claim 1, wherein said dynamically modifying comprises dynamically modifying an existing set of configurations associated with the mapper component based on the generated set of configurations.

5. The method of claim 1, wherein said dynamically modifying comprises initiating a future mapper component associated with the MapReduce job based on the generated set of configurations.

6. The method of claim 1, wherein said dynamically modifying comprises dynamically modifying an existing set of configurations associated with the reducer component based on the generated set of configurations.

7. The method of claim 1, wherein said dynamically modifying comprises initiating a future reducer component associated with the MapReduce job based on the generated set of configurations.

8. The method of claim 1, wherein said dynamically modifying comprises dynamically modifying configurations associated with multiple mapper components and/or multiple reducer components associated with the MapReduce job based on the generated set of configurations.

9. The method of claim 8, wherein said multiple mapper components and/or multiple reducer components comprise multiple heterogeneous mapper components and/or multiple heterogeneous reducer components.

10. The method of claim 1, wherein said deploying comprises deploying said modified configurations to the mapper component and/or the reducer component without interrupting task execution by the mapper component and/or the reducer component.

11. The method of claim 1, wherein said deploying comprises deploying said modified configurations based on type of parameters encompassed by the modified configurations.

12. The method of 1, wherein said deploying comprises deploying said modified configurations based on a task execution status associated with the mapper component and/or the reducer component.

13. An article of manufacture comprising a non-transitory computer readable storage medium having computer readable instructions tangibly embodied thereon which, when implemented, cause a computer to carry out a plurality of method steps comprising:

generating a first set of configurations for a currently executing MapReduce job, wherein said set of configurations comprises job-level configurations and task-level configurations;

dynamically modifying;

(i) a distinct set of task-level configurations of a mapper component associated with at least one ongoing map task of the MapReduce job, based on the generated first set of configurations, wherein said task-level configurations of the mapper component comprise a size of a map task input, resource allocation for a mapper component, central processing unit, memory, a size of a sorting area, and a number of threads when writing a map output to a local disk, and (ii) a distinct set of task-level configurations of a reducer component associated with at least one ongoing reduce task of the MapReduce job, based on the generated first set of configurations, wherein said task-level configurations of the reducer component comprise a number of threads when copying a map output to a reducer component, a size of a reduce task input, resource allocation of a reduce task, central processing unit, memory, and a size of a sorting area; and

deploying said modified configurations to the mapper component and the reducer component associated with the MapReduce job in accordance with one of multiple temporal deployment schedules, wherein the temporal deployment schedule is based on the contents of the modified configurations.

14. The article of manufacture of claim 13, wherein said dynamically modifying comprises dynamically modifying an existing set of configurations associated with the mapper component based on the generated set of configurations.

15. The article of manufacture of claim 13, wherein said dynamically modifying comprises initiating a future mapper component associated with the MapReduce job based on the generated set of configurations.

16. The article of manufacture of claim 13, wherein said dynamically modifying comprises dynamically modifying an existing set of configurations associated with the reducer component based on the generated set of configurations.

17. The article of manufacture of claim 13, wherein said dynamically modifying comprises initiating a future reducer component associated with the MapReduce job based on the generated set of configurations.

18. A system comprising:

at least one memory; and

at least one processor coupled to the at least one memory and configured for;

generating a first set of configurations for a currently executing MapReduce job, wherein said set of configurations comprises job-level configurations and task-level configurations;

dynamically modifying;

(i) a distinct set of task-level configurations of a mapper component associated with at least one ongoing map task of the MapReduce job, based on the generated first set of configurations, wherein said task-level configurations of the mapper component comprise a size of a map task input, resource allocation for a mapper component, central processing unit, memory, a size of a sorting area, and a number of threads when writing a map output to a local disk, and (ii) a distinct set of task-level configurations of a reducer component associated with at least one ongoing reduce task of the MapReduce job, based on the generated first set of configurations, wherein said task-level configurations of the reducer component comprise a number of threads when copying a map output to a reducer component, a size of a reduce task input, resource allocation of a reduce task, central processing unit, memory, and a size of a sorting area; and

deploying said modified configurations to the mapper component and the reducer component associated with the MapReduce job in accordance with one of multiple temporal deployment schedules, wherein the temporal deployment schedule is based on the contents of the modified configurations.