SELECTING PROCESSING TECHNIQUES FOR A DATA FLOW TASK

• US 20130047161A1
• Filed: 08/19/2011
• Published: 02/21/2013
• Est. Priority Date: 08/19/2011
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Inventors: Simitsis, Alkiviadis, Wilkinson, William Kevin
• Granted Patent: US 9,135,071 B2
• Time in Patent Office: Days
• US Class Current: 718/100
• CPC Class Codes: G06F 2209/501 Performance criteria
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

A method for data flow processing includes determining values for each of a set of parameters associated with a task within a data flow processing job, and applying a set of rules to determine one of a set of processing techniques that will be used to execute the task. The set of rules is determined through a set of benchmark tests for the task using each of the set of processing techniques while varying the set of parameters.

Claims

1. A method for selecting processing techniques for a data flow task performed by a physical computing system, the method comprising:

determining values for each of a set of parameters associated with a task within a data flow processing job; and

applying a set of rules to said values to determine one of a set of processing techniques that will be used to execute said task;

wherein said set of rules is determined through a set of benchmark tests for said task using each of said set of processing techniques while varying said set of parameters.

2. The method of claim 1, wherein said set of processing techniques includes at least one of:

a map-reduce processing techniques, a database management system processing flow, an Extract, Transform, Load (ETL) engine processing flow, and a custom script processing flow.

3. The method of claim 1, wherein said set of parameters comprises at least one of:

a blocking operator parameter, a number of nodes parameter, a size parameter, a network bandwidth parameter, and a reduce job parameter.

4. The method of claim 1, further comprising, applying said set of rules to determine one of said set of processing techniques for each of an additional number of tasks within said ETL processing job based on parameters of said additional tasks.

5. The method of claim 1, wherein said set of rules considers a cost of transforming data associated with said data flow processing job to a different format to use a different one of said set of processing technique.

6. The method of claim 5, wherein a cost of transforming said data from a local file system to a distributed filing system takes into account a process of changing metadata for stored data on a local file system to indicate that said stored data is part of a distributed file system without actually copying said stored data from said local file system to said distributed file system.

7. The method of claim 6, wherein said stored data with changed metadata is reallocated across said distributed file system in response to specified conditions of a network connecting said distributed file system.

8. A method for transferring data from a local file system to a distributed file system for a data flow process, the method comprising:

on a node of a distributed computing system, changing metadata associated with data stored on a local file system of said node without copying said data to a distributed file system, said changed metadata indicating that said data is associated with said distributed file system; and

indicating a presence of said data to a management function of said distributed file system.

9. The method of claim 8, further comprising, on each of a number of additional nodes within said distributed computing system, changing metadata associated with stored data on a local file system of that additional node to indicate that said stored data on that additional node is associated with said distributed file system without copying said stored data on that additional node to said distributed file system.

10. The method of claim 8, further comprising, rebalancing data spread across multiple nodes of said distributed filing system in response to specified conditions of a network connecting said distributed file system.

11. The method of claim 10, wherein said specified conditions comprises a load of said network falling below a predetermined threshold level.

12. The method of claim 8, wherein a cost of transferring said data from said local file system to said distributed file system is considered in a set of rules used to determine which of a set of processing techniques is to be used for a task of a data flow process based on parameters associated with said task and said distributed file system.

13. The method of claim 8, further comprising, changing metadata of distributed data stored on said node within said distributed file system to indicate that said distributed data is part of a local file system on said node.

14. A distributed computing system comprising:

a node comprising;

at least one processor; and

a memory communicatively coupled to the at least one processor, the memory comprising computer executable code that, when executed by the at least one processor, causes the at least one processor to;

change metadata associated with data stored on a local file system of said node without copying said data to a distributed file system, said changed metadata indicating that said data is associated with said distributed file system; and

indicate a presence of said data to a management function of said distributed file system;

wherein, a cost of transferring said data from said local file system to said distributed file system is used in part to define a set of rules used to determine which of a set of processing flows is to be used for a task of a data flow process based on parameters associated with said task and said distributed file system.

15. The system of claim 14, wherein said set of processing techniques includes at least one of:

a map-reduce processing flow, a database management system processing flow, an Extract, Transform, Load (ETL) engine processing flow, and a custom script processing flow.