STREAMS OPTIONAL EXECUTION PATHS DEPENDING UPON DATA RATES

• US 20140136724A1
• Filed: 12/05/2012
• Published: 05/15/2014
• Est. Priority Date: 11/13/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Branson, Michael J., Santosuosso, John M.
• Granted Patent: US 9,756,099 B2
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 67/61 taking into account QoS or priority requirements

Abstract

Processing elements in a streaming application may contain one or more optional code modules—i.e., computer-executable code that is executed only if one or more conditions are met. In one embodiment, an optional code module is executed based on evaluating data flow rate between components in the streaming application. As an example, the stream computing application may monitor the incoming data rate between processing elements and select which optional code module to execute based on this rate. For example, if the data rate is high, the stream computing application may choose an optional code module that takes less time to execute. Alternatively, a high data rate may indicate that the incoming data is important; thus, the streaming application may choose an optional code module containing a more rigorous data processing algorithm, even if this algorithm takes more time to execute.

Claims

1. A method of processing data comprising:

receiving streaming data to be processed by a plurality of interconnected processing elements, each processing element comprising one or more operators that process at least a portion of the received data by operation of one or more computer processors;

measuring a data flow rate in a data path between at least two operators in the plurality of processing elements;

selecting an optional code module based on the measured flow rate; and

executing the selected code module on one of the plurality of processing elements such that data received by the one processing element is processed by the selected code module.

2. The method of claim 1, wherein each of the at least two operators are contained in respective processing elements such that the data path transmits the streaming data between the respective processing elements.

3. The method of claim 1, wherein the optional code module is an optional code element in an operator in the one processing element, wherein the optional code element is used to process the data received by the one processing element only when the measured flow rate satisfies a predefined threshold,

wherein the measured flow rate is at least one of the number of data elements flowing in the data path during a predefined time period or a ratio of ingress data elements to egress data elements.

4. The method of claim 1, further comprising, upon determining that the measured data flow rate satisfies a predefined threshold, fusing an operator to the one processing element, wherein the fused operator is the optional code module.

5. The method of claim 4, further comprising, upon determining that the measured data flow rate no longer satisfies the predefined threshold, deactivating the fused operator from the one processing element by one of:

(i) un-fusing the fused operator from the one processing element and (ii) instructing the fused operator to no longer processes the data received by the one processing element.

6. The method of claim 1, wherein the optional code module is a process routine selected from a plurality of process routines on the one processing element, wherein each data tuple received by the one processing element is processed using at least one of the process routines, and wherein only one of the process routines is selected.

7. The method of claim 1, wherein the measured data flow rate is compared to a plurality of predefined thresholds arranged in a hierarchy such that the predefined threshold satisfied by the data flow rate with the highest priority is used to select the optional code module, wherein each predefined threshold is associated with a different optional code module.

8. The method of claim 1, wherein the data flow rate is extrapolated from data flowing on at least two data paths, wherein each of the two data paths transmit data between at least two operators in the operator graph.

9. The method of claim 1, wherein the at least two operators are located on one or more of the plurality of processing elements that are either upstream or downstream, in an operator graph of the plurality of processing element, from the one processing element that executes the selected code module,

the operator graph defining at least one execution path, wherein each processing element of the operator graph is configured to receive data from at least one operator in an upstream processing element and transmit data to at least one operator in a downstream processing element.