Apparatus and method for identifying performance bottlenecks in pipeline parallel processing environment

• US 9,575,916 B2
• Filed: 01/06/2014
• Issued: 02/21/2017
• Est. Priority Date: 01/06/2014
• Status: Expired due to Fees

• Current Assignee: AirBNB Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Caufield, Brian K., Liu, Ron E., Wei, DongJie, Yang, Xin Ying
• Primary Examiner(s): Tseng, Cheng-Yuan
• Application Number:
  US14/148,697
• Publication Number:
  US 20150193368A1
• Time in Patent Office: 1,142 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 13/36 for access to common bus or bus system

Abstract

A system identifies a performance bottleneck status in a parallel data processing environment by examining data flow associated with the parallel data processing environment to identify at least one operator, where an operator type is associated with at least one operator, at least one buffer, and a relationship that the buffer has with the operator, where the relationship is associated with the operator type. The system monitors the buffer to determine a buffer status associated with the buffer. The system applies a set of rules to identify an operator bottleneck status associated with the operator. The set of rules is applied to the operator, based on the operator type, the buffer status, and relationship that the buffer has with the operator. The system then determines a performance bottleneck status associated with the parallel data processing environment, based on the operator bottleneck status.

Claims

1. A method for identifying performance bottleneck status in a parallel data processing environment, implemented by a computing processor, the method comprising:

examining, by the computing processor, data flow associated with the parallel data processing environment to identify;

i) at least one operator, wherein an operator type is associated with the at least one operator;

ii) at least one buffer; and

iii) a relationship that the at least one buffer has with the at least one operator, wherein the relationship is associated with the operator type; and

wherein examining the data flow associated with the parallel data processing environment comprises;

identifying a first sub-operator connected to a second sub-operator, wherein no buffer exists between the first sub-operator and the second sub-operator; and

combining the first sub-operator and the second sub-operator to create the at least one operator;

monitoring, by the computing processor, the at least one buffer to determine a buffer status associated with the at least one buffer;

applying, by the computing processor, a set of rules to identify an operator bottleneck status associated with the at least one operator, wherein the set of rules is applied to;

i) the at least one operator, based on the operator type;

ii) the at least one buffer status; and

iii) the relationship that the at least one buffer has with the at least one operator; and

determining, by the computing processor, a performance bottleneck status associated with the parallel data processing environment, based on the operator bottleneck status.

2. The method of claim 1 wherein examining data flow associated with the parallel data processing environment comprises:

determining the operator type associated with the at least one operator based on a data partition configuration associated with;

i) the at least one buffer; and

ii) the relationship that the at least one buffer has with the at least one operator.

3. The method of claim 1 wherein monitoring the at least one buffer to determine the buffer status associated with the at least one buffer comprises:

determining the buffer status based on a flow of data from the at least one operator to at least one other operator.

4. The method of claim 3 wherein determining the buffer status based on the flow of data comprises:

identifying the buffer status as at least one of;

i) a buffer full status wherein no data can be written to the at least one buffer;

ii) a buffer empty status wherein no data can be read from the at least one buffer; and

iii) a buffer unknown status wherein the buffer status is neither the buffer full status nor the buffer empty status.

5. The method of claim 1 wherein applying the set of rules comprises:

traversing the data flow associated with the parallel data processing environment to apply the rules to each operator in the data flow.

6. The method of claim 1 wherein determining the performance bottleneck status associated with the parallel data processing environment comprises:

rendering a visualization of the data flow associated with the parallel data processing environment with respect to the operator bottleneck status associated with the at least one operator.

7. The method of claim 1 wherein determining the performance bottleneck status associated with the parallel data processing environment comprises:

periodically monitoring the at least one buffer and applying the set of rules over a period of time to monitor performance of the parallel data processing environment.

8. A computer program product for identifying performance bottleneck status in a parallel data processing environment, the computer program product comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the program code executable by a processor to;

examine data flow associated with the parallel data processing environment to identify;

i) at least one operator, wherein an operator type is associated with the at least one operator;

ii) at least one buffer; and

iii) a relationship that the at least one buffer has with the at least one operator, wherein the relationship is associated with the operator type; and

wherein the computer readable program code configured to examine data flow associated with the parallel data processing environment is further configured to;

identify a first sub-operator connected to a second sub-operator, wherein no buffer exists between the first sub-operator and the second sub-operator; and

combine the first sub-operator and the second sub-operator to create the at least one operator;

monitor the at least one buffer to determine a buffer status associated with the at least one buffer;

apply a set of rules to identify an operator bottleneck status associated with the at least one operator, wherein the set of rules is applied to;

i) the at least one operator, based on the operator type;

ii) the at least one buffer status; and

iii) the relationship that the at least one buffer has with the at least one operator; and

determine a performance bottleneck status associated with the parallel data processing environment, based on the operator bottleneck status.

9. The computer program product of claim 8 wherein the computer readable program code configured to examine data flow associated with the parallel data processing environment is further configured to:

determine the operator type associated with the at least one operator based on a data partition configuration associated with;

i) the at least one buffer; and

ii) the relationship that the at least one buffer has with the at least one operator.

10. The computer program product of claim 8 wherein the computer readable program code configured to monitor the at least one buffer to determine the buffer status associated with the at least one buffer is further configured to:

determine the buffer status based on a flow of data from the at least one operator to at least one other operator.

11. The computer program product of claim 10 wherein the computer readable program code configured to determine the buffer status based on the flow of data is further configured to:

identify the buffer status as at least one of;

i) a buffer full status wherein no data can be written to the at least one buffer;

ii) a buffer empty status wherein no data can be read from the at least one buffer; and

iii) a buffer unknown status wherein the buffer status is neither the buffer full status nor the buffer empty status.

12. The computer program product of claim 8 wherein the computer readable program code configured to apply the set of rules is further configured to:

traverse the data flow associated with the parallel data processing environment to apply the rules to each operator in the data flow.

13. The computer program product of claim 8 wherein the computer readable program code configured to determine the performance bottleneck status associated with the parallel data processing environment is further configured to:

periodically monitor the at least one buffer and applying the set of rules over a period of time to monitor performance of the parallel data processing environment.

14. A system comprising:

a processor; and

a computer readable storage medium operationally coupled to the processor, the computer readable storage medium having computer readable program code embodied therewith to be executed by the processor, the computer readable program code configured to;

examine data flow associated with the parallel data processing environment to identify;

i) at least one operator, wherein an operator type is associated with the at least one operator;

ii) at least one buffer; and

iii) a relationship that the at least one buffer has with the at least one operator, wherein the relationship is associated with the operator type, and wherein the computer readable program code configured to examine data flow associated with the parallel data processing environment is further configured to;

determine the operator type associated with the at least one operator based on a data partition configuration associated with;

i) the at least one buffer; and

ii) the relationship that the at least one buffer has with the at least one operator;

monitor the at least one buffer to determine a buffer status associated with the at least one buffer;

apply a set of rules to identify an operator bottleneck status associated with the at least one operator, wherein the set of rules is applied to;

i) the at least one operator, based on the operator type;

ii) the at least one buffer status; and

iii) the relationship that the at least one buffer has with the at least one operator; and

determine a performance bottleneck status associated with the parallel data processing environment, based on the operator bottleneck status.

15. The system of claim 14 wherein the computer readable program code configured to monitor the at least one buffer to determine the buffer status associated with the at least one buffer is further configured to:

determine the buffer status based on a flow of data from the at least one operator to at least one other operator.

16. The system of claim 14 wherein the computer readable program code configured to determine the buffer status based on the flow of data is further configured to:

identify the buffer status as at least one of;

i) a buffer full status wherein no data can be written to the at least one buffer;

ii) a buffer empty status wherein no data can be read from the at least one buffer; and

iii) a buffer unknown status wherein the buffer status is neither the buffer full status nor the buffer empty status.

17. The system of claim 14 wherein the computer readable program code configured to determine the performance bottleneck status associated with the parallel data processing environment is further configured to:

periodically monitor the at least one buffer and applying the set of rules over a period of time to monitor performance of the parallel data processing environment.