Reducing redundant operations in a streaming environment

• US 9,886,485 B2
• Filed: 03/11/2015
• Issued: 02/06/2018
• Est. Priority Date: 03/11/2015
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barsness, Eric L., Branson, Michael J., Santosuosso, John M.
• Primary Examiner(s): Nguyen, Kim
• Application Number:
  US14/644,398
• Publication Number:
  US 20160267197A1
• Time in Patent Office: 1,063 Days
• Field of Search: 707790, 707798
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A stream computing application may permit one or more jobs to connect to a data stream of a different job. As more jobs connect to the data stream, the likelihood that two or more operators are performing redundant operations on a data stream may increase. A stream manager may monitor an operator graph to identify two operators performing a redundant operation. If the stream manager determines that there are redundant operators in the operator graph, the stream manager may modify the operator graph to remove the redundant operators.

Claims

1. A system for processing a stream of tuples comprising:

a plurality of processing elements to receive a stream of tuples, each processing element having one or more stream operators;

two or more processors; and

a memory containing an application that, when executed, causes a processor to;

receive a stream of tuples to be processed by a plurality of processing elements operating on one or more computer processors, each processing element having one or more stream operators, one or more of the stream operators include code configured to output tuples to one or more other stream operators, the processing elements forming an operator graph in which tuples flow between stream operators, the operator graph defining one or more execution paths for processing and routing the stream of tuples, each processing element having an associated memory space;

identify a first job, the first job including a first stream operator and a second stream operator, the first stream operator being configured to import a group of tuples from a first data stream and output tuples to the second stream operator;

identify a second job, the second job including a third stream operator and a fourth stream operator, the third stream operator being configured to import the group of tuples from the first data stream and output tuples to the fourth stream operator;

determine whether the first stream operator and the third stream operator perform one or more redundant operations on the group of tuples;

create, in response to the first stream operator and the third stream operator performing one or more redundant operations on the group of tuples, an intermediary stream operator, the intermediary stream operator being configured to perform the one or more redundant operations;

set the intermediary stream operator as exportable; and

connect the intermediary stream operator to the first and second jobs.

2. A computer program product for processing a stream of tuples, the computer program product comprising a computer readable storage medium having program code embodied therein, the program code comprising computer readable program code configured to:

receive a stream of tuples to be processed by a plurality of processing elements operating on one or more computer processors, each processing element having one or more stream operators, one or more of the stream operators include code configured to output tuples to one or more other stream operators, the processing elements forming an operator graph in which tuples flow between stream operators, the operator graph defining one or more execution paths for processing and routing the stream of tuples, each processing element having an associated memory space;

identify a first job, the first job including a first stream operator and a second stream operator, the first stream operator being configured to import a group of tuples from a first data stream and output tuples to the second stream operator;

identify a second job, the second job including a third stream operator and a fourth stream operator, the third stream operator being configured to import the group of tuples from the first data stream and output tuples to the fourth stream operator;

determine whether the first stream operator and the third stream operator perform one or more redundant operations on the group of tuples;

create, in response to the first stream operator and the third stream operator performing one or more redundant operations on the group of tuples, an intermediary stream operator, the intermediary stream operator being configured to perform the one or more redundant operations;

set the intermediary stream operator as exportable; and

connect the intermediary stream operator to the first and second jobs.