×

Detection of data flow bottlenecks and disruptions based on operator timing profiles in a parallel processing environment

  • US 9,424,160 B2
  • Filed: 03/27/2015
  • Issued: 08/23/2016
  • Est. Priority Date: 03/18/2014
  • Status: Expired due to Fees
First Claim
Patent Images

1. A computer implemented method for detecting data flow disruptions over a series of data processing operators that are each configured to receive and store data in an input record block, process data from the input record block, store results of the processing in an output record block, and output data from the output record block to a next processing operator in the series, the method comprising:

  • generating, for a particular processing operator in the series of data processing operators, a processing operator timing profile that includes;

    an input wait time based upon a period of time that a particular data set is stored in a particular input data record,an operator processing time based upon a period of time between a start of processing of the particular data set by the particular processing operator and a completion of the processing of the particular data set by the particular processing operator, andan output wait time based upon a period of time that the particular data set is stored in a particular output data record block;

    detecting, from the processing operator timing profile, a potential flow disruption condition;

    determining that the processing operator timing profile satisfies at least one rule from a set of flow disruption rules that are each associated with at least one corresponding recommendation;

    identifying, based on the at least one rule, a corresponding recommendation; and

    displaying, in response to identifying the corresponding recommendation, an identity of the particular processing operator and the corresponding recommendation, wherein the series of data processing operators are part of a system of operators working in a parallel processing environment, wherein the parallel processing environment comprises a conductor, a plurality of section leaders, and a plurality of players, and wherein a section leader of the plurality of section leaders is configured to create a record block I/O monitoring thread to implement the generating, for the particular processing operator in the series of processing operators, the processing operator timing profile in the parallel processing environment.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×