PERFORMANCE REGRESSION MANAGER FOR LARGE SCALE SYSTEMS

• US 20150112641A1
• Filed: 10/18/2013
• Published: 04/23/2015
• Est. Priority Date: 10/18/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Faraj, Daniel A.
• Granted Patent: US 9,792,194 B2
• Time in Patent Office: Days
• US Class Current: 702/186
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3065 Monitoring arrangements determined by the means or processing involved in reporting the monitored data error or fault reporting or logging G06F11/0766
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 9/44526 Plug-ins; Add-ons

Abstract

System and computer program product to perform an operation comprising generating, based on a first output generated by a first execution instance of a command, a first output file specifying a value of at least one performance metric, wherein the first output file is formatted according to a predefined format, comparing the value of the at least one performance metric in the first output file to a value of the performance metric in a second output file, the second output file having been generated based on a second output generated by a second execution instance of the command, and outputting for display an indication a result of the comparison of the value of the at least one performance metric of the first output file to the value of the at least one performance metric of the second output file.

Claims

1. A system, comprising:

one or more computer processors; and

a memory containing a program, which, when executed by the one or more computer processors, performs an operation, the operation comprising;

generating, based on a first output generated by a first execution instance of a command, a first output file specifying a value of at least one performance metric, wherein the first output file is formatted according to a predefined format;

comparing, based on at least one comparison rule, the value of the at least one performance metric in the first output file to a value of the performance metric in a second output file, the second output file having been generated based on a second output generated by a second execution instance of the command; and

outputting for display an indication a result of the comparison of the value of the at least one performance metric of the first output file to the value of the at least one performance metric of the second output file.

2. The system of claim 1, wherein the at least one performance metric and the value of the at least one performance metric is specified in a user-defined plugin, the operation further comprising:

receiving the command; and

triggering the first execution instance of the command.

3. The system of claim 1, wherein each output is tagged with a respective instance identifier, wherein the instance identifier is used to generate a directory used to store each respective output and wherein the operation further comprises:

outputting for display an indication of whether the first execution instance of the command completed successfully.

4. The system of claim 1, wherein a first comparison rule is compared against each of a plurality of configurations of the first and second execution instances, wherein a second comparison rule is compared against a first configuration and not a second configuration of the plurality of configuration instances.

5. The system of claim 1, wherein a user specifies:

(i) the command, (ii) the at least one performance metric, and (iii) the at least one comparison rule.

6. The system of claim 1, wherein the at least one comparison rule specifies to compare the value of the at least one performance metric of the first output file to value of the at least one performance metric of the second output file and to a value of the at least one performance metric of a third output file, the third output file being generated based on a third output set generated by a third execution instance of the command, the operation further comprising:

outputting for display a result of the comparison of the value of the at least one performance metric of the first output file and the value of the at least one performance metric of the third output file.

7. The system of claim 1, wherein the at least one comparison rule specifies a comparison threshold, wherein the outputted comparison result indicates whether the comparison threshold was satisfied.

8. A computer program product, comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising;

computer-readable program code configured to generate, based on a first output generated by a first execution instance of a command, a first output file specifying a value of at least one performance metric, wherein the first output file is formatted according to a predefined format;

computer-readable program code configured to compare, based on at least one comparison rule, the value of the at least one performance metric in the first output file to a value of the performance metric in a second output file, the second output file having been generated based on a second output generated by a second execution instance of the command; and

computer-readable program code configured to output for display an indication a result of the comparison of the value of the at least one performance metric of the first output file to the value of the at least one performance metric of the second output file.

9. The computer program product of claim 8, wherein the at least one performance metric and the value of the at least one performance metric is specified in a user-defined plugin, the computer-readable program code further comprising:

computer-readable program code configured to receive the command; and

computer-readable program code configured to trigger the first execution instance of the command.

10. The computer program product of claim 8, wherein each output is tagged with a respective instance identifier, wherein the instance identifier is used to generate a directory used to store each respective output and wherein the computer-readable program code further comprises:

computer-readable program code configured to output for display an indication of whether the first execution instance of the command completed successfully.

11. The computer program product of claim 8, wherein a first comparison rule is compared against each of a plurality of configurations of the first and second execution instances, wherein a second comparison rule is compared against a first configuration and not a second configuration of the plurality of configuration instances.

12. The computer program product of claim 8, wherein a user specifies:

(i) the command, (ii) the at least one performance metric, and (iii) the at least one comparison rule.

13. The computer program product of claim 8, wherein the at least one comparison rule specifies to compare the value of the at least one performance metric of the first output file to value of the at least one performance metric of the second output file and to a value of the at least one performance metric of a third output file, the third output file being generated based on a third output set generated by a third execution instance of the command, the computer-readable program code further comprising:

computer-readable program code configured to output for display a result of the comparison of the value of the at least one performance metric of the first output file and the value of the at least one performance metric of the third output file.