Apparatus, system, and method for persistent testing with progressive environment sterilzation

• US 20070168734A1
• Filed: 11/17/2005
• Published: 07/19/2007
• Est. Priority Date: 11/17/2005
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Vasile, Phil
• Application Number:
  US11/281,646
• Publication Number:
  US 20070168734A1
• Time in Patent Office: Days
• US Class Current: 714/33
• CPC Class Codes: G06F 11/3672 Test management

Abstract

An apparatus, system, and method are disclosed for automatically testing a plurality of software test cases. The testing executes a quick test of the test cases which executes each test case in a test environment that is initialized just prior to the first test case and after subsequent test case failures. The testing further executes an adjusted test of the failing test cases in which delay parameters associated with the failing test cases are increased in accordance with a system load recorded during the quick test. Finally, the testing executes a sterilized test of the remaining failing test cases in a test environment that is initialized prior to each test case execution.

Claims

1. A method for automating execution of a plurality of test cases, the method:

comprising;

executing a quick test of a test suite comprising a plurality of test cases;

compiling a set of questionable test cases that failed the quick test;

executing an adjusted test of the questionable test cases;

compiling a set of suspect test cases that failed the adjusted test;

executing a sterilized test of the suspect test cases; and

compiling a set of broken test cases that failed the sterilized test.

2. The method of claim 1, wherein executing an adjusted test further comprises adjusting delay parameters associated with the set of questionable test cases based on the percentage of test cases that failed the quick test.

3. The method of claim 2, wherein adjusting delay parameters comprises increasing delay parameters in response to a system load of a computer system executing the quick test, adjusted test, and sterilized test.

4. A system to automate the execution of a plurality of test cases and systematically identify a set of broken test cases, the system comprising:

at least one computing device;

a test environment comprising a plurality of userids on the at least one computing device;

a test suite comprising a plurality of test cases that utilize at least one of the userids;

a quick test module configured to execute the test suite using the test environment and compile a set of questionable test cases comprising the failed test cases executed by the quick test module;

an adjusted test module configured to execute the set of questionable test cases using the test environment and compile a set of suspect test cases comprising the failed test cases executed by the adjusted test module, wherein the adjusted test module initializes the test environment prior to execution of the set of questionable test cases and subsequent to the failed execution of a questionable test case and wherein the adjusted test module increases delay parameters associated with the set of questionable test cases based on a percentage of failed test cases from the execution of the test suite by the quick test module;

a sterilized test module configured to execute the set of suspect test cases using the test environment and compile a set of broken test cases comprising the failed test cases executed by the sterilized test module, wherein the sterilized test module initializes the test environment prior to executing each suspect test case;

a control module configured to control execution of the quick test module, the adjusted test module, and the sterilized test module; and

a watch module configured to detect a testing irregularity and reinitialize the control module in response to the detected irregularity in the control module, such that the control module continues execution.

5. The system of claim 4, the control module further configured to track an execution status of each test case based on the results of the execution of the test case by the quick test module, the adjusted test, and the sterilized test module.

6. The system of claim 5, the control module further configured to notify a system operator of test case failures that exceed a threshold, wherein the notification is sent prior to the completion of the execution of the test suite.

7. The system of claim 5, the control module further configured to generate a report of broken test cases.

8. The system of claim 5, the control module further configured to continue execution of the test suite in response to the re-initialization of the control module according to the execution status of each test case.

9. The system of claim 4, wherein the at least one computing device comprises at least one International Business Machines (IBM) mainframe running the Virtual Machine (VM) operating system and the test environment comprises Multiple Virtual Storage (MVS) guest machines running under the VM operating system.

10. The system of claim 9, wherein the control module is further configured to initialize the test environment to an initial state prior to execution of test cases by the quick test module, the adjusted test module, and the sterilized test module.

11. The system of claim 10 wherein initializing the test environment comprises copying a set of initialization files to each MVS guest machine.

12. The system of claim 11, wherein initializing the test environment further comprises initializing each MVS guest machine by initializing MVS, Virtual Telecommunications Access Method (VTAM) and Information Management System (IMS) according to specifications associated with the test suite.

13. A signal bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform an operation to test a computer application the operation comprising:

executing a quick test of a test suite comprising a plurality of test cases configured to execute in a test environment comprising Multiple Virtual Storage (MVS) guest machines running on a Virtual Machine (VM) operating system on a mainframe;

compiling a set of questionable test cases that failed the quick test;

increasing delay parameters in the questionable test cases in accordance with the percentage of questionable test cases compared to the plurality of test cases

executing an adjusted test of the questionable test cases;

compiling a set of suspect test cases that failed the adjusted test;

executing a sterilized test of the suspect test cases;

compiling a set of broken test cases that failed the sterilized test;

maintaining an execution status for each test; and

monitoring the execution of the quick test, the adjusted test, and the sterilized test for a testing irregularity and restarting the execution of the quick test, the adjusted test, and the sterilized test in response to a detected testing irregularity according to the execution status of each test case.

14. The signal bearing medium of claim 13, wherein the instructions further comprise lengthening the delay parameters in accordance with a system load during the execution of the quick test.

15. The signal bearing medium of claim 13, wherein the instructions further comprise lengthening the delay parameters in accordance with a system load during the execution of the adjusted test.

16. The signal bearing medium of claim 13, wherein maintaining the execution status of each test case comprises tracking for each test case successful and failed completion of the execution of the quick test, the adjusted test, and the sterilized and wherein restarting the execution of the quick test, the adjusted test, and the sterilized test comprises completing the execution of each test case having no execution status.

17. The signal bearing medium of claim 13, wherein a service person causes the instructions to be executed to validate the integrity of a software installation.

18. The signal bearing medium of claim 13, wherein executing an adjusted test further comprises initializing the test environment to an initial state prior to executing the questionable test cases.

19. The signal bearing medium of claim 18, wherein executing an adjusted test further comprises detecting a failure of a questionable test case and initializing the test environment to the initial state prior to executing a next questionable test case.

20. The signal bearing medium of claim 19, wherein executing a sterilized test further comprises initializing the test environment to the initial state prior to executing each test case from the set of suspect test cases.