COST-BASED OPTIMIZATION OF CONFIGURATION PARAMETERS AND CLUSTER SIZING FOR HADOOP

• US 20130254196A1
• Filed: 03/15/2013
• Published: 09/26/2013
• Est. Priority Date: 03/26/2012
• Status: Active Grant

• Current Assignee: Duke University
• Original Assignee: Duke University
• Inventors: Babu, Shivnath, Herodotou, Herodotos
• Granted Patent: US 9,367,601 B2
• Time in Patent Office: Days
• US Class Current: 707/736
• CPC Class Codes: G06F 11/3404 for parallel or distributed programming
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3457 Performance evaluation by simulation
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

Cost-based optimization of configuration parameters and cluster sizing for distributed data processing systems are disclosed. According to an aspect, a method includes receiving at least one job profile of a MapReduce job. The method also includes using the at least one job profile to predict execution of the MapReduce job within a plurality of different predetermined settings of a distributed data processing system. Further, the method includes determining one of the predetermined settings that optimizes performance of the MapReduce job. The method may also include automatically adjusting the distributed data processing system to the determined predetermined setting.

Claims

1. A method comprising:

at a processor and memory;

receiving at least one measure of performance of a MapReduce job;

determining a job profile based on the at least one measure of performance; and

providing the job profile for one of user interface and prediction processes.

2. The method of claim 1, wherein receiving at least one measure of performance comprises receiving one of a dataflow measure, an execution time measure, and a resource usage measure.

3. The method of claim 1, wherein receiving at least one measure of performance comprises receiving run-time monitoring information.

4. The method of claim 1, wherein the job profile comprises dataflow information associated with the MapReduce job.

5. The method of claim 4, wherein the dataflow information comprises one of a size of data and I/O transfer processed during execution of the MapReduce job.

6. The method of claim 1, wherein the job profile comprises cost information associated with the MapReduce job.

7. The method of claim 6, wherein the cost information comprises one of resource usage and execution time associated with execution of the MapReduce job.

8. The method of claim 1, wherein determining a job profile comprises generating profile fields based on the at least one measure of performance of the MapReduce job.

9. The method of claim 8, wherein profile fields comprises one of a dataflow field, a cost field, a dataflow statistics field, and a cost statistic field.

10. The method of claim 1, further comprising using the job profile to predict behavior of the MapReduce job.

11. The method of claim 10, wherein using the job profile comprises using the job profile to simulate execution of the MapReduce job.

12. The method of claim 1, wherein the MapReduce job executes as map tasks and reduce tasks.

13. A method comprising:

at a processor and memory;

receiving at least one job profile of a MapReduce job;

using the at least one job profile to predict execution of the MapReduce job within a predetermined setting of a distributed data processing system; and

determining performance of the MapReduce job based on the prediction.

14. The method of claim 13, further comprising presenting the determined performance via a user interface.

15. The method of claim 13, wherein the job profile comprises dataflow information associated with the MapReduce job.

16. The method of claim 15, wherein the dataflow information comprises one of a size of data and I/O transfer processed during execution of the MapReduce job.

17. The method of claim 13, wherein the job profile comprises cost information associated with the MapReduce job.

18. The method of claim 17, wherein the cost information comprises one of resource usage and execution time associated with execution of the MapReduce job.

19. The method of claim 13, wherein the predetermined setting includes functions in the MapReduce program, layout of input data, sizing of cluster resources, configuration of the MapReduce job, and configuration of the distributed data processing system.

20. The method of claim 13, wherein using the job profile comprises simulating scheduling and execution of map and reduce tasks in the MapReduce job.

21. A method comprising:

at a processor and memory;

receiving at least one job profile of a MapReduce job;

using the at least one job profile to predict execution of the MapReduce job within a plurality of different predetermined settings of a distributed data processing system; and

determining one of the predetermined settings that optimizes performance of the MapReduce job.

22. The method of claim 21, further comprising presenting the determined predetermined setting via a user interface.

23. The method of claim 21, further comprising automatically adjusting the distributed data processing system to the determined predetermined setting.

24. The method of claim 21, wherein the job profile comprises dataflow information associated with the MapReduce job.

25. The method of claim 24, wherein the dataflow information comprises one of a size of data and I/O transfer processed during execution of the MapReduce job.

26. The method of claim 24, wherein the job profile comprises cost information associated with the MapReduce job.

27. The method of claim 26, wherein the cost information comprises one of resource usage and execution time associated with execution of the MapReduce job.

28. The method of claim 21, further comprising using input data and cluster resources for the simulation.

29. The method of claim 21, wherein the predetermined setting includes functions in the MapReduce program, layout of input data, sizing of cluster resources, configuration of the MapReduce job, and configuration of the distributed data processing system.

30. The method of claim 21, wherein using the job profile comprises simulating scheduling and execution of map and reduce tasks in the MapReduce job.

31. The method of claim 21, wherein determining one of the predetermined settings comprises determining the one of the predetermined settings that optimizes performance of the MapReduce job on a cluster.

32. A method comprising:

at a processor and memory;

receiving at least one job profile of a MapReduce job; and

using the at least one job profile to determine one of the predetermined settings that optimizes performance of the MapReduce job within a plurality of different predetermined settings of a distributed data processing system

33. The method of claim 32, wherein the predetermined setting includes functions in the MapReduce program, layout of input data, sizing of cluster resources, configuration of the MapReduce job, and configuration of the distributed data processing system.

34. A computing device comprising:

a processor and memory configured to;

receive at least one measure of performance of a MapReduce job;

determine a job profile based on the at least one measure of performance; and

provide the job profile for one of user interface and prediction processes.

35. A computing device comprising:

a processor and memory configured to;

receive at least one job profile of a MapReduce job;

use the at least one job profile to predict execution of the MapReduce job within a predetermined setting of a distributed data processing system; and

determine performance of the MapReduce job based on the prediction.

36. A computing device comprising:

a processor and memory configured to;

receive at least one job profile of a MapReduce job;

use the at least one job profile to predict execution of the MapReduce job within a plurality of different predetermined settings of a distributed data processing system; and

determine one of the predetermined settings that optimizes performance of the MapReduce job.