Stream application performance monitoring metrics

• US 9,396,157 B2
• Filed: 08/26/2011
• Issued: 07/19/2016
• Est. Priority Date: 08/26/2011
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cradick, Ryan K., Pfeifer, Michael D., Santosuosso, John M.
• Primary Examiner(s): Patel, Chirag R
• Application Number:
  US13/218,545
• Publication Number:
  US 20130054779A1
• Time in Patent Office: 1,789 Days
• Field of Search: 709224-227, 709/223, 707713-716
• US Class Current: 1/1
• CPC Class Codes: G06F 15/173 using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 65/70 Media network packetisation

Abstract

Techniques are disclosed for calculating performance metrics associated with a data stream. A processing element configured to process data tuples flowing through a stream-based computing system receives data tuples via the data stream. Each of the one or more tuples is processed at the processing element, and each of the one or more tuples are associated with metadata that includes information related to the processing of the tuple by the processing element. Performance metrics are then calculated for the data stream based on the metadata associated with the one or more tuples.

Claims

1. A non-transitory computer-readable storage medium storing an application, which, when executed on a processor in a stream-based computing system, performs an operation comprising:

receiving, at a first processing element of a plurality of processing elements of the stream-based computing system that are configured to process data tuples flowing through the stream-based computing system, one or more tuples associated with a first data stream, each of the one or more tuples including a plurality of data attributes;

processing each of the one or more tuples at the first processing element;

for each of the one or more tuples, associating the tuple with respective metadata that includes information related to the processing of the tuple by the first processing element and includes at least a first and a second counter that each correspond to a respective one of the plurality of data attributes of the tuple, wherein the first and second counters indicate a number of times that the corresponding data attribute has changed;

updating, at one or more of the plurality of processing elements that are located downstream of the first processing element, at least one of the first and second counters based on further processing of the one or more tuples by the one or more downstream processing elements;

calculating, after the one or more tuples have flowed through the plurality of processing elements, performance metrics based on the metadata associated with the one or more tuples;

displaying the calculated performance metrics via a user interface on a display device; and

automatically adjusting, based on the calculated performance metrics, a configuration of the stream-based computing system.

2. The computer-readable storage medium of claim 1, wherein the metadata information further includes a timestamp, an identification (ID) of the first processing element, an ID of an operator graph in which the first processing element is included, and an ID of a compute node on which the first processing element is executing.

3. The computer-readable storage medium of claim 2, wherein the calculated performance metrics indicate, for each of the one or more tuples received via the first data stream, an amount of time for the first processing element and other processing elements included in the stream-based computing system to process the tuple.

4. The computer-readable storage medium of claim 2, wherein the calculated performance metrics indicate, for each of the one or more tuples received via the first data stream, an amount of time for two or more processing elements included in the stream-based computing system to process the tuple.

5. The computer-readable storage medium of claim 2, wherein the calculated performance metrics indicate an amount of time for each of the one or more tuples to travel through the stream-based computing system.

6. The computer-readable storage medium of claim 2, wherein the calculated performance metrics indicate an amount of time for each of the one or more tuples to travel between two or more compute nodes included in the stream-based computing system.

7. The computer-readable storage medium of claim 2, wherein the calculated performance metrics indicate, for each of the one or more tuples received via the first data stream, a number of times that each processing element, operator graph, and/or compute node included in the stream-based computing system processes the tuple.

8. The computer-readable storage medium of claim 2, wherein the calculated performance metrics indicate, for each of the one or more tuples received via the first data stream, a number of times that each data attribute included in the tuple has been modified.

9. A stream-based computing system, comprising:

a plurality of compute nodes collectively providing a plurality of processing elements for processing data tuples flowing through the stream-based computing system, each compute node comprising a respective processor and a respective memory, wherein the memory stores a stream performance monitor which, when executed on the compute node, is configured to perform an operation for calculating performance metrics, the operation comprising;

receiving, at a first processing element of the plurality of processing elements, one or more tuples associated with a first data stream, each of the one or more tuples including a plurality of data attributes;

processing each of the one or more tuples at the first processing element;

for each of the one or more tuples, associating the tuple with respective metadata that includes information related to the processing of the tuple by the first processing element and includes at least a first and a second counter that each correspond to a respective one of the plurality of data attributes of the tuple, wherein the first and second counters indicate indicating a number of times that the corresponding data attribute has changed;

updating, at one or more of the plurality of processing elements that are located downstream of the first processing element, at least one of the first and second counters based on further processing of the one or more tuples by the one or more downstream processing elements;

calculating, after the one or more tuples have flowed through the plurality of processing elements, performance metrics based on the metadata associated with the one or more tuples;

displaying the calculated performance metrics via a user interface on a display device; and

automatically adjusting, based on the calculated performance metrics, a configuration of the stream-based computing system.

10. The system of claim 9, wherein the metadata information further includes a timestamp, an identification (ID) of the first processing element, an ID of an operator graph in which the first processing element is included, and an ID of a compute node on which the first processing element is executing.

11. The system of claim 10, wherein the calculated performance metrics indicate, for each of the one or more tuples received via the first data stream, an amount of time for the first processing element and other processing elements included in the stream-based computing system to process the tuple.

12. The system of claim 10, wherein the calculated performance metrics indicate, for each of the one or more tuples received via the first data stream, an amount of time for two or more processing elements included in the stream-based computing system to process the tuple.

13. The system of claim 10, wherein the calculated performance metrics indicate an amount of time for each of the one or more tuples to travel through the stream-based computing system.

14. The system of claim 10, wherein the calculated performance metrics indicate an amount of time for each of the one or more tuples to travel between two or more compute nodes included in the stream-based computing system.

15. The system of claim 10, wherein the calculated performance metrics indicate, for each of the one or more tuples received via the first data stream, a number of times that each processing element, operator graph, and/or compute node included in the stream-based computing system processes the tuple.

16. The system of claim 10, wherein the calculated performance metrics indicate, for each of the one or more tuples received via the first data stream, a number of times that each data attribute included in the tuple has been modified.

17. A stream-based computing system, comprising:

a plurality of compute nodes collectively providing a plurality of processing elements for processing data tuples flowing through the stream-based computing system, each compute node comprising a respective processor and a respective memory, wherein the memory stores a stream performance monitor which, when executed on the compute node, is configured to perform an operation for calculating performance metrics, the operation comprising;

receiving, at a first processing element of the plurality of processing elements, one or more tuples associated with a first data stream, each of the one or more tuples including a plurality of data attributes;

processing each of the one or more tuples at the first processing element;

for each of the one or more tuples, associating the tuple with respective metadata that includes information related to the processing of the tuple by the first processing element and includes at least a first and a second counter that each correspond to a respective one of the plurality of data attributes of the tuple and indicate a number of times that the respective data attribute has changed;

updating, at one or more of the plurality of processing elements that are located downstream of the first processing element, at least one of the first and second counters based on further processing of the one or more tuples by the one or more downstream processing elements;

calculating, after the one or more tuples have flowed through the plurality of processing elements, performance metrics based on values of the first and second counters associated with each of the one or more tuples; and

based on the calculated performance metrics, automatically modify a configuration of a plurality of processing elements that are executing on the plurality of compute nodes.

18. The system of claim 17, wherein automatically modifying a configuration of a plurality of processing elements comprises moving one or more processing elements from a first compute node to a second compute node.