Test self-verification with integrated transparent self-diagnose

• US 9,952,965 B2
• Filed: 08/06/2015
• Issued: 04/24/2018
• Est. Priority Date: 08/06/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Liu, Su, Paul, Priya, Su, Jun, Xu, Cheng
• Primary Examiner(s): Rutten, James D
• Application Number:
  US14/819,518
• Publication Number:
  US 20170039121A1
• Time in Patent Office: 992 Days
• Field of Search: 717124
• CPC Class Codes: G06F 11/3624 by performing operations on the source code, e.g. via a compiler
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases

Abstract

A method, program product, and computer system is provided for test case self-validating. A probe builder, instruments one or more source code modules with a test probe. The test probe placement is based on at least one criterion including: an application program interface (API), a component, a test case name, a product release, and a product feature. The probe builder registers the test probe in a probe database. The registered test probe has record in the probe database that includes a probe identifier, a probed command, a probed file name, a line number, the test case name, and a location of an executable binary containing the test probe. The probe builder compiles the instrumented source code modules into one or more binary executable modules. The test case generator creates a test case that includes at least one registered test probe. The test case validator validates the test case.

Claims

1. A method for test case self-validating comprising:

instrumenting, by a probe builder, one or more source code modules with a test probe wherein a placement for the test probe is based on at least one criterion including;

an application program interface (API), a component, a test case name, a product release, and a product feature;

registering, by the probe builder, the test probe in a probe database wherein the registered test probe has a corresponding record comprising;

a probe identifier, a probed command, a probed file name, a line number, the test case name, and a location of an executable binary containing the test probe;

compiling, by the probe builder, the instrumented source code modules into one or more binary executable modules;

creating, by a test case generator, a test case wherein the test case includes at least one registered test probe; and

validating, by a test case validator, the test case.

2. The method of claim 1, wherein compiling further comprises:

enabling at least one test probe during the compiling and storing the one or more binary executable modules in a database of probe enabled binary executables; and

compiling the instrumented source code modules without test probes enabled into one or more non-probe enabled binary executable modules and storing the one or more non-probe enabled binary executable modules in a database of non-probe enabled binary executables.

3. The method of claim 2, wherein validating the test case further comprises:

performing a first execution of the test case using the database of non-probe enabled binary executables;

performing a second execution of the test case using the database of probe enabled binary executables;

marking the test case as invalid based on the first and the second execution returning the same results; and

marking the test case as valid based on the first and the second executions returning different results.

4. The method of claim 2, wherein the database of probe enabled binary executables is packaged with a software product and provided as a library of real-time diagnostics.

5. The method of claim 1, wherein creating the test case further comprises:

comparing a first source code repository corresponding to a current source code release and a second source code repository corresponding to a previous source code release wherein the comparing is based on at least one criterion including;

the API, the component, the product release, and the product feature;

getting an expected output based on a change present in the first source code repository; and

verifying the test case; and

merging the verified test case into a test case pool.

6. The method of claim 1, further comprising:

selecting one or more probes from the probe database; and

tracing an execution of one or more fields selected from the probe database record corresponding to the selected probes.

7. The method of claim 1, wherein the registered test probe is enabled or disabled in response to one of:

a compiler directive, and a profile setting.

8. A computer program product for test case self-validating comprising:

a test case management and validation (TCMV) system embodied on a computer readable storage medium, the TCMV including program instructions executable by a processor, the program instructions comprising;

program instructions to instrument, by a probe builder, one or more source code modules with a test probe wherein a placement for the test probe is based on at least one criterion including;

an application program interface (API), a component, a test case name, a product release, and a product feature;

program instructions to register, by the probe builder, the test probe in a probe database wherein the registered test probe has a corresponding record comprising;

a probe identifier, a probed command, a probed file name, a line number, the test case name, and a location of an executable binary containing the test probe;

program instructions to compile, by the probe builder, the instrumented source code modules into one or more binary executable modules;

program instructions to create, by a test case generator, a test case wherein the test case includes at least one test probe; and

program instructions to validate, by a test case validator, the test case.

9. The computer program product of claim 8, wherein the program instructions to compile further comprise:

program instructions to enable at least one test probe during the compiling and store the one or more binary executable modules in a database of probe enabled binary executables; and

program instructions to compile the instrumented source code modules without test probes enabled into one or more non-probe enabled binary executable modules and store the one or more non-probe enabled binary executable modules in a database of non-probe enabled binary executables.

10. The computer program product of claim 9, wherein the program instructions to validate the test case further comprise:

program instructions to perform a first execution of the test case using the database of non-probe enabled binary executables;

program instructions to perform a second execution of the test case using the database of probe enabled binary executables;

program instructions to mark the test case as invalid based on the first and the second execution returning the same results; and

program instructions to mark the test case as valid based on the first and the second executions returning different results.

11. The computer program product of claim 9, wherein the database of probe enabled binary executables is packaged with a software product and provided as a library of real-time diagnostics;

and wherein the registered test probe is enabled or disabled in response to one of;

a compiler directive, and a profile setting.

12. The computer program product of claim 8, further comprising:

program instructions to select one or more probes from the probe database; and

program instructions to trace an execution of one or more fields selected from the probe database record corresponding to the selected probes.

13. The computer program product of claim 8, wherein the program instructions to create the test case further comprise:

program instructions to compare a first source code repository corresponding to a current source code release and a second source code repository corresponding to a previous source code release wherein the comparing is based on at least one criterion including;

the API, the component, the product release, and the product feature;

program instructions to get an expected output based on a change present in the first source code repository; and

program instructions to merge the verified test case into a test case pool.

14. A computer system for test case self-validating comprising:

one or more processors, one or more computer-readable storage devices, and a plurality of program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors, the plurality of program instructions comprising;

program instructions to instrument, by a probe builder, one or more source code modules with a test probe wherein a placement for the test probe is based on at least one criterion including;

an application program interface (API), a component, a test case name, a product release, and a product feature;

program instructions to register, by the probe builder, the test probe in a probe database wherein the registered test probe has a corresponding record comprising;

a probe identifier, a probed command, a probed file name, a line number, the test case name, and a location of an executable binary containing the test probe;

program instructions to compile, by the probe builder, the instrumented source code modules into one or more binary executable modules;

program instructions to create, by a test case generator, a test case wherein the test case includes at least one registered test probe; and

program instructions to validate, by a test case validator, the test case.

15. The computer system of claim 14, wherein the program instructions to compile further comprise:

program instructions to enable at least one test probe during the compiling and store the one or more binary executable modules in a database of probe enabled binary executables; and

program instructions to compile the instrumented source code modules without test probes enabled into one or more non-probe enabled binary executable modules and store the one or more non-probe enabled binary executable modules in a database of non-probe enabled binary executables.

16. The computer system of claim 15, wherein the program instructions to validate the test case further comprise:

program instructions to perform a first execution of the test case using the database of non-probe enabled binary executables;

program instructions to perform a second execution of the test case using the database of probe enabled binary executables;

program instructions to mark the test case as invalid based on the first and the second execution returning the same results; and

program instructions to mark the test case as valid based on the first and the second executions returning different results.

17. The computer system of claim 15, wherein the database of probe enabled binary executables is packaged with a software product and provided as a library of real-time diagnostics.

18. The computer system of claim 14, further comprising:

program instructions to select one or more probes from the probe database; and

program instructions to trace an execution of one or more fields selected from the probe database record corresponding to the selected probes.

19. The computer system of claim 14, wherein the program instructions to create the test case further comprise:

program instructions to compare a first source code repository corresponding to a current source code release and a second source code repository corresponding to a previous source code release wherein the comparing is based on at least one criterion including;

the API, the component, the product release, and the product feature;

program instructions to get an expected output based on a change present in the first source code repository; and

program instructions to merge the verified test case into a test case pool.

20. The computer system of claim 14, wherein the registered test probe is enabled or disabled in response to one of:

a compiler directive, and a profile setting.