Detection of data flow bottlenecks and disruptions based on operator timing profiles in a parallel processing environment

• US 9,424,160 B2
• Filed: 03/27/2015
• Issued: 08/23/2016
• Est. Priority Date: 03/18/2014
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Caufield, Brian K., Greene, Lawrence A., Jacobson, Eric A., Li, Yong, Mudambi, Shyam R., Pu, Xiaoyan, Wei, Dong J.
• Primary Examiner(s): WU, BENJAMIN C
• Application Number:
  US14/671,102
• Publication Number:
  US 20150269006A1
• Time in Patent Office: 515 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3404 for parallel or distributed programming
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/524 Deadlock detection or avoidance

Abstract

Data flow disruptions over a series of data processing operators can be detected by a computer system that generates a profile for data flow at an operator. The profile can include data input, processing, and output wait times. Using the profile, the system can detect potential flow disruptions. If the potential disruption satisfies a rule, it is considered a data flow disruption and a recommendation associated with the satisfied rule is identified. The recommendation and the operator identity is displayed.

Claims

1. A computer implemented method for detecting data flow disruptions over a series of data processing operators that are each configured to receive and store data in an input record block, process data from the input record block, store results of the processing in an output record block, and output data from the output record block to a next processing operator in the series, the method comprising:

generating, for a particular processing operator in the series of data processing operators, a processing operator timing profile that includes;

an input wait time based upon a period of time that a particular data set is stored in a particular input data record,

an operator processing time based upon a period of time between a start of processing of the particular data set by the particular processing operator and a completion of the processing of the particular data set by the particular processing operator, and

an output wait time based upon a period of time that the particular data set is stored in a particular output data record block;

detecting, from the processing operator timing profile, a potential flow disruption condition;

determining that the processing operator timing profile satisfies at least one rule from a set of flow disruption rules that are each associated with at least one corresponding recommendation;

identifying, based on the at least one rule, a corresponding recommendation; and

displaying, in response to identifying the corresponding recommendation, an identity of the particular processing operator and the corresponding recommendation, wherein the series of data processing operators are part of a system of operators working in a parallel processing environment, wherein the parallel processing environment comprises a conductor, a plurality of section leaders, and a plurality of players, and wherein a section leader of the plurality of section leaders is configured to create a record block I/O monitoring thread to implement the generating, for the particular processing operator in the series of processing operators, the processing operator timing profile in the parallel processing environment.

2. The computer implemented method of claim 1, wherein the displaying the identity of the particular processing operator and the corresponding recommendation occurs via a user interface, the user interface part of a job execution design recommendation of a performance management system.

3. The computer implemented method of claim 1, further comprising, in response to the identifying the corresponding recommendation, implementing the recommendation, based on settings in a performance management system.

4. A computer system for detecting data flow disruptions over a series of data processing operators that are each configured to receive and store data in an input record block, process data from the input record block, store results of the processing in an output record block, and output data from the output record block to a next processing operator in the series, the system comprising:

at least one processor circuit configured to;

generate, for a particular processing operator in the series of data processing operators, a processing operator timing profile that includes;

an input wait time based upon a period of time that a particular data set is stored in a particular input data record,

an operator processing time based upon a period of time between a start of processing of the particular data set by the particular processing operator and a completion of the processing of the particular data set by the particular processing operator, and

an output wait time based upon a period of time that the particular data set is stored in a particular output data record block;

detect, from the processing operator timing profile, a potential flow disruption condition;

determine that the processing operator timing profile satisfies at least one rule from a set of flow disruption rules that are each associated with at least one corresponding recommendation;

identify, based on the at least one rule, a corresponding recommendation; and

display, in response to identifying the corresponding recommendation, an identity of the particular processing operator and the corresponding recommendation, wherein the series of data processing operators are part of a system of operators working in a parallel processing environment, wherein the parallel processing environment comprises a conductor, a plurality of section leaders, and a plurality of players, and wherein a section leader of the plurality of section leaders is configured to create a record block I/O monitoring thread to implement the generating, for the particular processing operator in the series of processing operators, the processing operator timing profile in the parallel processing environment.

5. The computer system of claim 4, wherein the at least one processor circuit is configured to display the identity of the particular processing operator and the corresponding recommendation via a user interface, the user interface part of a job execution design recommendation of a performance management system.

6. The computer system of claim 4, wherein the at least one processor circuit is further configured implement the recommendation, in response to the identifying the corresponding recommendation, based on settings in a performance management system.

7. A computer program product for detecting data flow disruptions over a series of data processing operators that are each configured to receive and store data in an input record block, process data from the input record block, store results of the processing in an output record block, and output data from the output record block to a next processing operator in the series, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a computer processing circuit to cause the circuit to perform the method comprising:

generating, for a particular processing operator in the series of data processing operators, a processing operator timing profile that includes;

an input wait time based upon a period of time that a particular data set is stored in a particular input data record,

an operator processing time based upon a period of time between a start of processing of the particular data set by the particular processing operator and a completion of the processing of the particular data set by the particular processing operator, and

an output wait time based upon a period of time that the particular data set is stored in a particular output data record block;

detecting, from the processing operator timing profile, a potential flow disruption condition;

determining that the processing operator timing profile satisfies at least one rule from a set of flow disruption rules that are each associated with at least one corresponding recommendation;

identifying, based on the at least one rule, a corresponding recommendation; and

displaying, in response to identifying the corresponding recommendation, an identity of the particular processing operator and the corresponding recommendation, wherein the series of data processing operators are part of a system of operators working in a parallel processing environment, wherein the parallel processing environment comprises a conductor, a plurality of section leaders, and a plurality of players, and wherein a section leader of the plurality of section leaders is configured to create a record block I/O monitoring thread to implement the generating, for the particular processing operator in the series of processing operators, the processing operator timing profile in the parallel processing environment.

8. The computer program product of claim 7, wherein the displaying the identity of the particular processing operator and the corresponding recommendation occurs via a user interface, the user interface part of a job execution design recommendation of a performance management system.

9. The computer program product of claim 7, further comprising, in response to the identifying the corresponding recommendation, implementing the recommendation, based on settings in a performance management system.