CREATING A MODEL RELATING TO EXECUTION OF A JOB ON PLATFORMS

• US 20140215471A1
• Filed: 01/28/2013
• Published: 07/31/2014
• Est. Priority Date: 01/28/2013
• Status: Abandoned Application

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Cherkasova, Ludmila, Verma, Abhishek
• Application Number:
  US13/751,262
• Publication Number:
  US 20140215471A1
• Time in Patent Office: Days
• US Class Current: 718/102
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

At least one benchmark is determined. The at least one benchmark is run on first and second computing platforms to generate platform profiles. Based on the generated platform profiles, a model is generated that characterizes a relationship between a MapReduce job executing on the first platform and the MapReduce job executing on the second platform, wherein the MapReduce job includes map tasks and reduce tasks.

Claims

1. A method comprising:

determining, by a system having a processor, at least one benchmark that includes a set of parameters and values assigned to the respective parameters;

generating, by the system, platform profiles based on running the at least one benchmark on respective first and second computing platforms; and

creating, by the system based on the generated platform profiles, a model that characterizes a relationship between a MapReduce job executing on the first computing platform and the MapReduce job executing on the second computing platform, wherein the MapReduce job includes map tasks and reduce tasks.

2. The method of claim 1, wherein the map tasks produce intermediate results based on segments of input data, and the reduce tasks produce an output based on the intermediate results.

3. The method of claim 1, wherein each of the platform profiles includes values of a performance metric for respective phases of the map tasks and respective phases of the reduce tasks.

4. The method of claim 3, wherein the performance metric includes a time duration.

5. The method of claim 1, wherein generating the platform profiles comprises collecting measurements relating to phases of the map tasks and reduce tasks during running of the at least one benchmark on the first and second computing platforms.

6. The method of claim 5, wherein the phases of the map tasks include a read phase, a map phase, and a collect phase.

7. The method of claim 6, wherein the phases of each map task further include a spill phase and a merge phase.

8. The method of claim 5, wherein the phases of each reduce task include a shuffle phase, a reduce phase, and a write phase.

9. A system comprising:

at least one processor to;

produce a plurality of benchmarks that describe respective characteristics of MapReduce jobs that include map tasks and reduce tasks;

run the benchmarks on different computing platforms;

collect measurements relating to map tasks and reduce tasks during running the benchmarks; and

create a model based on the collected measurements, wherein the model characterizes a relationship between MapReduce job execution on a first one of the computing platforms with MapReduce job execution on a second one of the computing platforms.

10. The system of claim 9, wherein the first computing platform is an existing computing platform on which production MapReduce jobs are executed, and the second computing platform is a new computing platform for replacing the existing computing platform.

11. The system of claim 9, wherein the first and second computing platforms are alternative computing platforms considered for selection.

12. The system of claim 9, wherein the model is created based on using linear regression based on the measurements.

13. The system of claim 9, wherein the model includes sub-models that make up the model, wherein each of the sub-models relates a phase of a map task or reduce task on the first computing platform to a corresponding phase of a map task or reduce task on the second computing platform.

14. The system of claim 9, wherein the benchmarks are produced using a benchmark specification that includes parameters and collections of candidate values of the corresponding parameters, wherein each of the parameters relates to a characteristic of a map task or reduce task.

15. The system of claim 14, wherein the benchmarks produced using the benchmark specification are based on using different ones of the candidate values of the collection of values associated with at least one of the parameters in the benchmark specification.

16. The system of claim 9, wherein each of the benchmarks includes a map selectivity parameter that represents a ratio of a size of a map task output to a size of map task input.

17. The system of claim 16, wherein each of the benchmarks further includes a reduce selectivity parameter that represents a ratio of a size of a reduce task output to a size of a reduce task input.

18. The system of claim 17, wherein each of the benchmarks further includes a map computation parameter that represents computation performed by a map task, and a reduce computation parameter that represents computation performed by a reduce task.

19. The system of claim 9, wherein the measurements include durations of respective phases of map tasks and respective phases of reduce tasks.

20. An article comprising at least one machine-readable storage medium storing instructions that upon execution cause a system having a processor to:

determine at least one benchmark that represents characteristics of map and reduce tasks;

generate platform profiles based on running the at least one benchmark on respective first and second computing platforms, wherein the platform profiles includes values of at least one performance metric for respective phases of map tasks and respective phases of reduce tasks; and

create, based on the generated platform profiles, a model that characterizes a relationship between a MapReduce job executing on the first computing platform and the MapReduce job executing on the second computing platform, wherein the MapReduce job includes map tasks and reduce tasks.