Selecting processing techniques for a data flow task

• US 9,135,071 B2
• Filed: 08/19/2011
• Issued: 09/15/2015
• Est. Priority Date: 08/19/2011
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Simitsis, Alkiviadis, Wilkinson, William Kevin
• Primary Examiner(s): Kim, Sisley
• Application Number:
  US13/213,582
• Publication Number:
  US 20130047161A1
• Time in Patent Office: 1,488 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 2209/501 Performance criteria
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

A method for data flow processing includes determining values for each of a set of parameters associated with a task within a data flow processing job, and applying a set of rules to determine one of a set of processing techniques that will be used to execute the task. The set of rules is determined through a set of benchmark tests for the task using each of the set of processing techniques while varying the set of parameters.

Claims

1. A method for selecting processing techniques for a data flow task performed by a physical computing system, the method comprising:

determining values for each of a set of parameters associated with a task within a data flow processing job; and

applying a set of rules to said values to determine one of a set of processing techniques that will be used to execute said task;

wherein said set of rules is determined through a plurality of benchmark tests for said task using each of said set of processing techniques while varying said set of parameters,

wherein said plurality of benchmark tests comprise at least one crossover point defining a transition between a first parameter range where a first processing technique completes said task in a smallest amount of time and a second parameter range where a second processing technique completes said task in the smallest amount of time.

2. The method of claim 1, wherein said set of processing techniques includes at least one of:

a map-reduce processing technique, a database management system processing flow, an Extract, Transform, Load (ETL) engine processing flow, and a custom script processing flow.

3. The method of claim 1, wherein said set of parameters comprises at least one of:

a blocking operator parameter, a number of nodes parameter, a data size parameter, a network bandwidth parameter, and a reduce job parameter.

4. The method of claim 1, further comprising, applying said set of rules to determine one of said set of processing techniques for each of an additional number of tasks within said data flow processing job based on parameters of said additional tasks.

5. The method of claim 1, wherein said set of rules considers a cost of transforming data associated with said data flow processing job to a different format to use a different one of said set of processing techniques.

6. The method of claim 5, wherein a cost of transforming said data from a local file system to a distributed file system takes into account a process of changing metadata for stored data on a local file system to indicate that said stored data is part of a distributed file system without actually copying said stored data from said local file system to said distributed file system.

7. The method of claim 6, wherein said stored data with changed metadata is reallocated across said distributed file system in response to specified conditions of a network connecting said distributed file system.

8. The method of claim 1, wherein said plurality of benchmark tests define a number of blocking operations.

17. The method of claim 1, wherein said set of benchmark tests comprise:

varying a parameter within said set of parameters over a range for a simulated task; and

measuring execution times for each of said processing techniques to perform said simulated task over said range.

18. The method of claim 1, wherein said set of benchmark tests comprise a crossover point, wherein said set of rules comprises applying a first processing technique for said task when a said value for said task falls below said crossover point and applying a second processing technique when said value falls above said crossover point.

9. A method for transferring data from a local file system to a distributed file system for a data flow process, the method comprising:

on a node of a distributed computing system, changing metadata associated with data stored on a local file system of said node without copying said data to a distributed file system, said changed metadata indicating that said data is no longer associated with said local file system, but is associated with said distributed file system; and

indicating a presence of said data to a management function of said distributed file system;

wherein, a cost of transferring said data from said local file system to said distributed file system is used in part to define a set of rules used to determine through a plurality of benchmark tests which of a set of processing techniques is to be used for a task of a data flow process based on parameters associated with said task and said distributed file system,

wherein said rules are based on based on the plurality of benchmark tests comprising at least one crossover point defining a transition between a first parameter range where a first processing technique completes said task in a smallest amount of time and a second parameter range where a second processing technique completes said task in the smallest amount of time.

10. The method of claim 9, further comprising, on each of a number of additional nodes within said distributed computing system, changing metadata associated with stored data on a local file system of that additional node to indicate that said stored data on that additional node is associated with said distributed file system without copying said stored data on that additional node to said distributed file system.

11. The method of claim 9, further comprising, rebalancing data spread across multiple nodes of said distributed filing system in response to specified conditions of a network connecting said distributed file system.

12. The method of claim 11, wherein said specified conditions comprises a load of said network falling below a predetermined threshold level.

13. The method of claim 9, wherein a cost of transferring said data from said local file system to said distributed file system is considered in a set of rules used to determine which of a set of processing techniques is to be used for a task of a data flow process based on parameters associated with said task and said distributed file system.

14. The method of claim 9, further comprising, changing metadata of distributed data stored on said node within said distributed file system to indicate that said distributed data is part of a local file system on said node.

15. A distributed computing system comprising:

a node comprising;

at least one processor; and

a memory communicatively coupled to the at least one processor, the memory comprising computer executable code that, when executed by the at least one processor, causes the at least one processor to;

change metadata associated with data stored on a local file system of said node without copying said data to a distributed file system, said changed metadata indicating that said data is no longer associated with said local file system, but is associated with said distributed file system; and

indicate a presence of said data to a management function of said distributed file system;

wherein, a cost of transferring said data from said local file system to said distributed file system is used in part to define a set of rules used to determine, through a number of benchmark tests, which of a set of processing techniques is to be used for a task of a data flow process based on parameters associated with said task and said distributed file system, wherein said rules are based on the number of benchmark tests comprising at least one crossover point defining a transition between a first parameter range where a first processing technique completes said task in a smallest amount of time and a second parameter range where a second processing technique completes said task in the smallest amount of time.

16. The system of claim 15, wherein said set of processing techniques includes at least one of:

a map-reduce processing flow, a database management system processing flow, an Extract, Transform, Load (ETL) engine processing flow, and a custom script processing flow.

19. The system of claim 15, wherein said benchmark tests define which of said set of processing techniques is most time efficient based on a parameter associated with said task, wherein said set of benchmark tests comprise varying said parameter over a range and measuring which of said processing techniques is most time efficient in execution said task.

20. The system of claim 19, wherein said parameter is a number of blocking operations in said task and said set of benchmark tests comprise measuring an execution time for simulated tasks with a range of blocking operations for each processing flow in said set of processing techniques.