Operator graph changes in response to dynamic connections in stream computing applications

• US 9,146,775 B2
• Filed: 04/26/2012
• Issued: 09/29/2015
• Est. Priority Date: 04/26/2012
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Branson, Michael J., Cradick, Ryan K., Santosuosso, John M., Schulz, Brandon W.
• Primary Examiner(s): Al Kawsar, Abdullah
• Assistant Examiner(s): TEETS, BRADLEY A
• Application Number:
  US13/456,600
• Publication Number:
  US 20130290966A1
• Time in Patent Office: 1,251 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 9/48 Program initiating; Program switching, e.g. by interrupt
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

A stream computing application may permit one job to connect to a data stream of a different job. As more and more jobs dynamically connect to the data stream, the connections may have a negative impact on the performance of the job that generates the data stream. Accordingly, a variety of metrics and statistics (e.g., CPU utilization or tuple rate) may be monitored to determine if the dynamic connections are harming performance. If so, the stream computing system may be optimized to mitigate the effects of the dynamic connections. For example, particular operators may be unfused from a processing element and moved to a compute node that has available computing resources. Additionally, the stream computing application may clone the data stream in order to distribute the workload of transmitting the data stream to the connected jobs.

Claims

1. A computer program product for optimizing a stream computing application comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising computer-readable program code configured to;

execute a first job and a second job, each comprising a plurality of respective operators that process streaming data by operation of one or more computer processors, wherein the plurality of respective operators in the first and second jobs are, respectively, interconnected such that data tuples flow between the plurality of respective operators to perform the first and second jobs;

establish an operator graph comprising the plurality of respective operators of both the first and second jobs, the operator graph defining at least one respective execution path through the plurality of respective operators for the first job and for the second job;

while the first and second jobs are executing, establish a connection between the first job and the second job by transmitting a data stream from a first operator of the first job to a second operator of the second job, wherein the first and second jobs are in the operator graph both before and after the connection is established;

before establishing the connection between the first job to the second job, set the data stream as exportable, wherein the plurality of respective operators associated with the second job do not receive data from, or send data to, the plurality of respective operators associated with the first job prior to setting the data stream as exportable;

monitor a performance indicator associated with the first operator of the first job, the performance indicator measuring an effect that the connection between the first and second jobs has on a performance of the first job; and

upon determining a value of the performance indicator satisfies a predefined threshold, optimize the stream computing application to improve the value of the performance indicator.

2. The computer program product of claim 1, wherein optimizing the stream computing application comprises optimizing the first job, further comprising computer-readable program code configured to, after determining the value of the performance indicator satisfies a predefined threshold, delaying optimizing the first job until the data stream of the first job is idle.

3. The computer program product of claim 1, wherein monitoring the performance indicator further comprises:

recording at least one of;

a time when the first job connects to the second job and a number of jobs connected to the data stream of the first job.

4. The computer program product of claim 1, wherein optimizing comprises at least one of:

moving at least one of the plurality of respective operators to a compute node with available resources, fusing or unfusing at least one of the plurality of respective operators, and cloning the first operator of the first job that generates the data stream.

5. The computer program product of claim 1, wherein the stream computing application comprises the first and second jobs.

6. A system for optimizing a stream computing application, comprising:

a computer processor; and

a memory containing a program that, when executed on the computer processor, performs an operation for processing data, comprising;

executing a first job and a second job, each comprising a plurality of respective operators that process streaming data, wherein the plurality of respective operators in the first and second jobs are, respectively, interconnected such that data tuples flow between the plurality of respective operators to perform the first and second jobs;

establishing an operator graph comprising the plurality of respective operators of both the first and second jobs, the operator graph defining at least one respective execution path through the plurality of respective operators for the first job and for the second job;

while the first and second jobs are executing, establishing a connection between the first job and the second job by transmitting a data stream from a first operator of the first job to a second operator of the second job, wherein the first and second jobs are in the operator graph both before and after the connection is established;

before establishing the connection between the first job to the second job, setting the data stream as exportable, wherein the plurality of respective operators associated with the second job do not receive data from, or send data to, the plurality of respective operators associated with the first job prior to setting the data stream as exportable;

monitoring a performance indicator associated with the first operator of the first job, the performance indicator measuring an effect that the connection between the first and second jobs has on a performance of the first job; and

upon determining a value of the performance indicator satisfies a predefined threshold, optimizing the stream computing application to improve the value of the performance indicator.

7. The system of claim 6, wherein optimizing the stream computing application comprises optimizing the first job, the program further comprising, after determining the value of the performance indicator satisfies a predefined threshold, delaying optimizing the first job until the data stream of the first job is idle.

8. The system of claim 6, wherein monitoring the performance indicator further comprises:

recording at least one of;

a time when the first job connects to the second job and a number of jobs connected to the data stream of the first job.

9. The system of claim 6, wherein optimizing comprises at least one of:

moving at least one of the plurality of respective operators to a compute node with available resources, fusing or unfusing at least one of the plurality of respective operators, and cloning the first operator of the first job that generates the data stream.

10. The system of claim 6, wherein the stream computing application comprises the first and second jobs.

11. A computer program product for optimizing a stream computing application comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising computer-readable program code configured to;

execute a first job and a second job, each comprising a plurality of respective operators that process streaming data by operation of one or more computer processors, wherein the plurality of respective operators in the first and second jobs are, respectively, interconnected such that data tuples flow between the plurality of respective operators to perform the first and second jobs;

establish an operator graph comprising the plurality of respective operators of both the first and second jobs, the operator graph defining at least one respective execution path through the plurality of respective operators for the first job and for the second job;

while the first and second jobs are executing, establish a connection between the first job and the second job by transmitting a data stream from a first operator of the first job to a second operator of the second job, wherein the first and second jobs are in the operator graph both before and after the connection is established;

monitor a performance indicator associated with the first operator of the first job by recording at least one of a time when the first job connects to the second job and a number of jobs connected to the data stream of the first job, wherein the performance indicator measures an effect that the connection between the first and second jobs has on a performance of the first job; and

upon determining a value of the performance indicator satisfies a predefined threshold, optimize the stream computing application to improve the value of the performance indicator.

12. The computer program product of claim 11, wherein optimizing the stream computing application comprises optimizing the first job, further comprising computer-readable program code configured to, after determining the value of the performance indicator satisfies a predefined threshold, delaying optimizing the first job until the data stream of the first job is idle.

13. The computer program product of claim 11, wherein optimizing comprises at least one of:

moving at least one of the plurality of respective operators to a compute node with available resources, fusing or unfusing at least one of the plurality of respective operators, and cloning the first operator of the first job that generates the data stream.