Workflow controller compatibility

• US 9,336,288 B2
• Filed: 06/03/2013
• Issued: 05/10/2016
• Est. Priority Date: 06/03/2013
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Mundlapudi, Bharath, Banala, Karthik, Koneru, Rajesh
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Tran, Loc
• Application Number:
  US13/908,681
• Publication Number:
  US 20140358844A1
• Time in Patent Office: 1,072 Days
• Field of Search: 707/601, 707/602, 707/603
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

Aspects of the disclosure relate to workflow controller compatibility for in order to format workflow data such that communication between incompatible workflow controllers is facilitated. A compatibility processing module may execute one or more processes to format a workflow such that communication between previously incompatible workflow controllers is facilitated. A first workflow controller is configured with a compatibility processing module and receives a large number of data points. The compatibility module executes one or more processes on a workflow such that the processed workflow is compatible with a second workflow controller.

Claims

1. A system for establishing compatibility between workflow controllers comprising:

a computing memory;

a first computing device configured to receive a data stream comprising a plurality of data points, the first computing device being configured to distribute individual data points of the data stream to one of a plurality of second computing devices, each second computing device being configured to process the data point received and generate a processed data point;

a data store storing a first workflow having a first data format, the first workflow comprising a plurality of action nodes, each action node comprising an instruction to be executed wherein the instruction of at least one of the action nodes comprises a map and reduce process on a set of data points of the data stream;

a first workflow controller which schedules execution of the action nodes of the first workflow;

wherein the first workflow controller is further executed one or more formatting processes on the first workflow in order to obtain a second workflow having a second data format that is compatible with a second workflow controller which schedules execution of the second workflow; and

wherein at least one of the formatting processes is configured to format a first workflow parameter of a plurality of first workflow parameters of the first workflow according to the second data format to obtain a second workflow parameter for the second workflow, the plurality of workflow parameters comprising (i) a mapper to utilize during the map and reduce process, (ii) a reducer to utilize during the map and reduce process, (iii) an input location at which the plurality of data points are stored, (iv) an output location at which to store the processed data point, and (v) a location at which a job tracker which monitors a status of the first workflow resides.

2. The system according to claim 1, wherein:

the first computing device is further configured to communicate a workflow status to the second workflow controller; and

the workflow status comprises at least one of a succeeded status, a failed status, or a killed status.

3. The system according to claim 2, wherein the first computing device communicates the workflow status to the second workflow controller before communicating the status to the first workflow controller.

4. The system according to claim 1, further comprising a distributed file system, wherein the first computing device configures the first workflow from the first workflow controller to execute one or more distributed file system processes.

5. The system according to claim 1, further comprising a data warehouse, wherein the first computing device configures the first workflow from the first workflow controller to execute one or more data warehouse processes.

6. A non-transitory computer-readable storage medium having computer executable program instructions stored thereon that when executed by a processor cause the processor to perform steps establishing compatibility between workflow controllers, the steps comprising:

receiving, a data stream comprising a plurality of data points;

distributing individual data points of the data stream to one of a plurality of second computing devices, each second computing device being configured to process the data point received and generate a processed data point;

storing a first workflow having a first data format, the first workflow comprising a plurality of action nodes, each action node comprising an instruction to be executed wherein the instruction of at least one of the action nodes comprises a map and reduce process on a data point from the data stream;

scheduling, using a first workflow controller, execution of the action nodes of the first workflow; and

executing one or more formatting processes on the first workflow workflow in order to obtain a second workflow having a second data format that is compatible with a second workflow controller which schedules execution of the second workflow;

wherein at least one of the formatting processes is configured to format a first workflow parameter of a plurality of first workflow parameters of the first workflow according to the second data format to obtain a second workflow parameter for the second workflow, the plurality of workflow parameters comprising (i) a mapper to utilize during the map and reduce process, (ii) a reducer to utilize during the map and reduce process, (iii) an input location at which the plurality of data points are stored, (iv) an output location at which to store the processed data point, and (v) a location at which a job tracker which monitors a status of the first workflow resides.

7. The non-transitory computer-readable storage medium of claim 6 wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising:

communicating, workflow status to the second workflow controller, wherein the workflow status comprises at least one of a succeeded status, a failed status, or a killed status.

8. The non-transitory computer-readable storage medium of claim 7 wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising:

communicating.

9. The non-transitory computer-readable storage medium of claim 6 wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising:

configuring the second workflow from the second workflow controller to execute one or more processes of an distributed file system.

10. The non-transitory computer-readable storage medium of claim 6 wherein the computer-executable instructions, when executed by the processor, cause the processor to perform steps further comprising:

configuring the second workflow from the second workflow controller to execute one or more processes of a data warehouse.

11. A method of establishing compatibility between workflow controllers, comprising:

receiving a data stream, comprising a plurality of data points;

distributing individual data points of the data stream to one of a plurality of second computing devices, each second computing device being configured to process the data point received and generate a processed data point;

storing a first workflow having a first data format, the first workflow comprising a plurality of action nodes, each action node comprising an instruction to be executed wherein the instruction of at least one of the action nodes comprises a map and reduce process on a data point from the data stream;

scheduling, using a first workflow controller, execution of the action nodes of the first workflow;

executing one or more formatting processes on the first workflow in order to obtain a second workflow having a second data format that is compatible with a second workflow controller which schedules execution of the second workflow; and

wherein at least one of the formatting processes is configured to format a first workflow parameter of a plurality of first workflow parameters of the first workflow according to the second data format to obtain a second workflow parameter for the second workflow, the plurality of workflow parameters comprising (i) a mapper to utilize during the map and reduce process, (ii) a reducer to utilize during the map and reduce process, (iii) an input location at which the plurality of data points are stored, (iv) an output location at which to store the processed data point, and (v) a location at which a job tracker which monitors a status of the first workflow resides.

12. The method of claim 11, further comprising:

communicating a workflow status to the second workflow controller, wherein the workflow status comprises at least one of a succeeded status, a failed status, or a killed status.

13. The method of claim 12, further comprising:

communicating the workflow status to the second workflow controller before communicating the status to the first workflow controller.

14. The method of claim 11, further comprising:

configuring the second workflow from the second workflow controller to execute one or more processes of an distributed file system.

15. The method of claim 11, further comprising:

configuring the second workflow from the second workflow controller to execute one or more processes of a data warehouse.