DEVICE OF MANAGING DISTRIBUTED PROCESSING AND METHOD OF MANAGING DISTRIBUTED PROCESSING

• US 20140201114A1
• Filed: 08/15/2012
• Published: 07/17/2014
• Est. Priority Date: 08/15/2011
• Status: Abandoned Application

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Tamano, Hiroshi
• Application Number:
  US14/238,708
• Publication Number:
  US 20140201114A1
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Provided is a device of managing distributed processing, including: a selecting unit that estimates a total execution time on the basis of each of distributed-execution patterns indicating a grouping mode for plural computers and corresponding to the number of computers that are in charge of each of processes having different parameters in plural phases, this total execution time being necessary for the plural computers to execute the plural processes in a distributed manner, thereby selecting a distributed-execution pattern that makes the total execution time minimal, from among the distributed-execution patterns.

Claims

1. A device of managing distributed processing, comprising:

a selecting unit that estimates a total execution time on the basis of each of distributed-execution patterns indicating a grouping mode for plural computers and corresponding to the number of computers that are in charge of each of processes having different parameters in plural phases, the total execution time being necessary for the plural computers to execute the plural processes in a distributed manner, thereby selecting a distributed-execution pattern that makes the total execution time minimal, from among the distributed-execution patterns.

2. The device of managing distributed processing according to claim 1, wherein

the plural phases at least include:

a Map phase in which input data used for each of the processes is read, and data obtained by applying a predetermined process to the input data is transmitted to a later phase; and

a Reduce phase in which a predetermined process is applied to the data decomposed in the Map phase, and

the selecting unit estimates the total execution time by using an estimation expression that depends on each of the distributed-execution patterns and is obtained by combining the estimation expression for the processing time for the Map phase and an estimation expression for a processing time for the Reduce phase.

3. The device of managing distributed processing according to claim 2, further comprising:

a Map-phase measuring unit that measures a calculation time for the Map phase by causing at least one of the plural computers to execute the Map phase, in which

the selecting unit acquires a time for reading data per computer corresponding to each of the distributed-execution patterns, and the calculation time for the Map phase measured by the Map-phase measuring unit, and uses any one of the acquired times as the processing time for the Map phase included in the total execution time.

4. The device of managing distributed processing according to claim 2, further comprising:

a Reduce-phase measuring unit that measures a processing time for the Reduce phase per process corresponding to a first distributed-execution pattern of the plural distributed-execution patterns by causing one of the plural processes to be executed with the first distributed-execution pattern, wherein

the selecting unit;

acquires an estimation model expression for estimating, on the basis of an amount of data processed and each of the distributed-execution patterns, a processing time for a reference process serving as a reference for processing of the Reduce phase;

estimates, on the basis of this estimation model expression, the processing time for the reference process corresponding to the first distributed-execution pattern;

corrects this estimation model expression using a ratio between the processing time for the Reduce phase per process corresponding to the first distributed-execution pattern and the processing time for the reference process corresponding to the first distributed-execution pattern, thereby estimating the processing time for the Reduce phase per process corresponding to each of the distributed-execution patterns; and

estimates an execution time for the Reduce phase included in the total execution time using the estimated processing time for the Reduce phase.

5. The device of managing distributed processing according to claim 4, further comprising:

a reference-process measuring unit that measures an execution time for the reference process by actually executing the reference process while varying the number of computers in charge and an amount of data processed; and

a regression analyzing unit that estimates the estimation model expression by performing a regression analysis using plural combination data of the number of computers in charge, the amount of data processed, and the execution time for the reference process, each of which is acquired by the reference-process measuring unit.

6. The device of managing distributed processing according to claim 5, further comprising:

an estimation-model storage unit that stores plural estimation model expressions for estimating the processing time for the reference process; and

an estimation-model selecting unit that evaluates the plural estimation model expressions using an information criterion on the basis of a result of the regression analysis to each of the estimation model expressions by the regression analyzing unit, thereby selecting one estimation model expression from among the plural estimation model expressions, wherein

the selecting unit acquires the estimation model expression selected by the estimation-model selecting unit.

7. The device of managing distributed processing according to claim 2, wherein

the plural phases further include a Setup phase in which an initialization process for a later phase is performed,

the device of managing distributed processing further includes a Setup-phase measuring unit that measures a processing time for the Setup phase per process by causing at least one computer of the plural computers to execute one of the plural processes, and

the selecting unit:

acquires the number of processes per computer corresponding to each of the distributed-execution patterns;

multiplies the processing time for the Setup phase per process by the number of processes per computer to estimate an estimation expression for the processing time for the Setup phase; and

estimates the total execution time using an estimation expression that depends on each of the distributed-execution patterns and is obtained by further combining the estimation expression for the processing time for the Setup phase with the estimation expression for the processing time for the Map phase and the estimation expression for the processing time for the Reduce phase.

8. The device of managing distributed processing according to claim 1, further comprising

a distributed-processing execution unit that assigns a parameter to each group on the basis of the grouping mode indicated by the distributed-execution pattern selected by the selecting unit, and instructs each group to execute the plural processes in a distributed manner.

11. The device of managing distributed processing according to claim 1, wherein

the plural phases at least include:

a Map phase in which input data used for each of the processes is read, and data obtained by applying a predetermined process to the input data is transmitted to a later phase; and

a Reduce phase in which a predetermined process is applied to the data decomposed in the Map phase, and

the selecting unit estimates the total execution time by using an estimation expression that depends on each of the distributed-execution patterns and is obtained from an estimation expression for a processing time for the Map phase.

9. A method of managing distributed processing, the method being performed by a computer and including:

estimating a total execution time on the basis of each of distributed-execution patterns indicating a grouping mode for plural computers and corresponding to the number of computers that are in charge of each of processes having different parameters in plural phases, this total execution time being necessary for the plural computers to execute the plural processes in a distributed manner, thereby selecting a distributed-execution pattern that makes the total execution time minimal, from among the distributed-execution patterns.

10. A non-transitory computer-readable storage medium storing a program for causing a program that causes a computer to realize:

a selecting unit that estimates a total execution time on the basis of each of distributed-execution patterns indicating a grouping mode for plural computers and corresponding to the number of computers that are in charge of each of processes having different parameters in plural phases, this total execution time being necessary for the plural computers to execute the plural processes in a distributed manner, thereby selecting a distributed-execution pattern that makes the total execution time minimal, from among the distributed-execution patterns.