Stream operator management

• US 10,042,611 B2
• Filed: 03/21/2016
• Issued: 08/07/2018
• Est. Priority Date: 03/21/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barsness, Eric L., Cook, Alexander, Orozco, Manuel, Santosuosso, John M.
• Primary Examiner(s): Kang, Insun
• Application Number:
  US15/076,601
• Publication Number:
  US 20170269911A1
• Time in Patent Office: 869 Days
• Field of Search: None
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 8/20 Software design
• CPC Class Codes: G06F 8/36 Software reuse
• CPC Class Codes: H04L 65/60 Network streaming of media packets

Abstract

Disclosed aspects relate to stream operator management. A stream operator may be analyzed with respect to a set of usage indicators. The stream operator may operate on a computer processor and correspond to a processing element for processing a stream of tuples. Based on analyzing the stream operator, a set of profile data for the stream operator may be determined. The set of profile data for the stream operator may be established for utilization to develop a streaming application.

Claims

1. A computer-implemented method of stream operator management, the method comprising:

analyzing, with respect to a set of usage indicators, a first stream operator which operates on a computer processor and corresponds to a first processing element for processing a stream of tuples of a stream application;

determining, based on the analyzing, a first set of profile data for the first stream operator, wherein the first set of profile data includes at least a tuple size threshold value;

establishing, for utilization to develop the streaming application, the first set of profile data for the first stream operator, wherein establishing the first set of profile data for the first stream operator includes adjusting stream application configuration parameters of the streaming application for processing the stream of tuples;

responsive to identifying a potential bottleneck representing a deficiency factor in the stream of tuples of the streaming application, comparing the deficiency factor to sets of profile data for available stream operators, the sets of profile data including the first set of profile data; and

responsive to identifying a second stream operator from the available stream operators for alleviating the potential bottleneck in the stream of tuples, introducing, using a second set of profile data corresponding to the second stream operator, a split operator to the stream application to allocate a portion of the stream of tuples to the second stream operator which operates on the computer processor and corresponds to a second processing element for processing the portion of the stream of tuples, wherein introducing the split operator comprises adding or suggesting the second stream operator for inclusion in an operator graph of the streaming application.

2. The method of claim 1, wherein developing the streaming application includes constructing implementation code.

3. The method of claim 1, wherein developing the streaming application includes configuring a plurality of parameters.

4. The method of claim 1, wherein establishing, for utilization to develop the streaming application, the first set of profile data for the first stream operator includes:

providing, for utilization to develop the streaming application in an integrated development environment, the first set of profile data for the first stream operator to provide immediate feedback on a stream design in the integrated development environment.

5. The method of claim 1, wherein establishing, for utilization to develop the streaming application, the first set of profile data for the first stream operator includes:

capturing, in a stream manager for utilization to develop the streaming application, the first set of profile data for the first stream operator.

6. The method of claim 1, wherein the set of usage indicators includes a set of values which corresponds to a selection from a group consisting of at least one of:

a processor utilization factor,

a memory utilization factor,

a disk utilization factor, or

a throughput factor.

7. The method of claim 1, wherein analyzing, with respect to the set of usage indicators, the first stream operator includes:

monitoring the first stream operator;

identifying, based on the monitoring, a set of values for the set of usage indicators; and

collecting the set of values to determine the first set of profile data for the first stream operator.

8. The method of claim 1, wherein the first set of profile data for the first stream operator includes:

a set of computing resource arrangements for utilization to develop the streaming application.

9. The method of claim 8, wherein the set of computing resource arrangements includes a selection from a group consisting of at least one of:

a threshold value for a set of processing resources,

a threshold value for a set of memory resources, or

a threshold value for a set of disk resources.

10. The method of claim 1, wherein the first set of profile data for the first stream operator further includes a selection from a group consisting of at least one of:

a tuple type, a tuple quantity threshold value, a tuple processing rate threshold value, an attribute type, an attribute quantity threshold value, an attribute size threshold value, a window size threshold value, or a temporal period threshold value.

11. The method of claim 1, wherein determining the first set of profile data for the first stream operator includes:

computing a set of threshold factor values which correspond to the set of usage indicators.

12. The method of claim 1, wherein determining the first set of profile data for the first stream operator includes:

computing a set of target factor values which correspond to the set of usage indicators.

13. The method of claim 12, wherein a first throughput value using the set of target factor values exceeds a second throughput value using a set of original factor values.

14. The method of claim 1, further comprising:

introducing, using the first set of profile data, a selection from a group consisting of at least one of;

a filter operator,

a functor operator,

a custom operator,

an ordering operator,

a barrier operator, or

a sync operator.

15. The method of claim 1, further comprising:

receiving the stream of tuples to be processed by a set of processing elements which includes the first and second processing elements and operates on a set of compute nodes; and

processing, using the set of processing elements operating on the set of compute nodes, the stream of tuples.

16. A system for stream operator management, the system comprising:

a hardware memory having a set of computer readable computer instructions, and

a processor for executing the set of computer readable instructions, the set of computer readable instructions including;

analyzing, with respect to a set of usage indicators, a first stream operator which operates on a computer processor and corresponds to a first processing element for processing a stream of tuples of a stream application;

determining, based on the analyzing, a first set of profile data for the first stream operator, wherein the first set of profile data includes at least a tuple size threshold value;

establishing, for utilization to develop the streaming application, the first set of profile data for the first stream operator, wherein establishing the first set of profile data for the first stream operator includes adjusting stream application configuration parameters of the streaming application for processing the stream of tuples;

responsive to identifying a potential bottleneck representing a deficiency factor in the stream of tuples of the streaming application, comparing the deficiency factor to sets of profile data for available stream operators, the sets of profile data including the first set of profile data; and

responsive to identifying a second stream operator from the available stream operators for alleviating the potential bottleneck in the stream of tuples, introducing, using a second set of profile data corresponding to the second stream operator, a split operator to the stream application to allocate a portion of the stream of tuples to the second stream operator which operates on the computer processor and corresponds to a second processing element for processing the portion of the stream of tuples, wherein introducing the split operator comprises adding or suggesting the second stream operator for inclusion in an operator graph of the streaming application.

17. A computer program product for stream operator management, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising:

analyzing, with respect to a set of usage indicators, a first stream operator which operates on a computer processor and corresponds to a first processing element for processing a stream of tuples of a stream application;

determining, based on the analyzing, a first set of profile data for the first stream operator, wherein the first set of profile data includes at least a tuple size threshold value;

establishing, for utilization to develop the streaming application, the first set of profile data for the first stream operator, wherein establishing the first set of profile data for the first stream operator includes adjusting stream application configuration parameters of the streaming application for processing the stream of tuples;

responsive to identifying a potential bottleneck representing a deficiency factor in the stream of tuples of the streaming application, comparing the deficiency factor to sets of profile data for available stream operators, the sets of profile data including the first set of profile data; and

responsive to identifying a second stream operator from the available stream operators for alleviating the potential bottleneck in the stream of tuples, introducing, using a second set of profile data corresponding to the second stream operator, a split operator to the stream application to allocate a portion of the stream of tuples to the second stream operator which operates on the computer processor and corresponds to a second processing element for processing the portion of the stream of tuples, wherein introducing the split operator comprises adding or suggesting the second stream operator for inclusion in an operator graph of the streaming application.