Testcase selection by the exclusion of disapproved, non-tested and defect testcases

• US 6,560,721 B1
• Filed: 04/26/2000
• Issued: 05/06/2003
• Est. Priority Date: 08/21/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Boardman, Trevor John, Steph, Kal Christian
• Primary Examiner(s): Beausoliel, Robert
• Assistant Examiner(s): CHU, GABRIEL L
• Application Number:
  US09/558,719
• Time in Patent Office: 1,105 Days
• Field of Search: 714/25, 714/26, 714/32, 714/33, 714/37, 714/38
• US Class Current: 714/33
• CPC Class Codes: G06F 11/263 Generation of test inputs, e.g. test vectors, patterns or sequences ; with adaptation of the tested hardware for testability with external testers

Abstract

A testcase management system comprises a test suite including a plurality of testcases, wherein these testcases are cooperable with a test harness for executing a test run in which at least one of the testcases is executed on a system to be tested. A method and apparatus are provided for generating a list (overall list) indicative of the testcases that are to be executed during the test run. The overall list is generated as follows: A list is generated of all testcases in the test suite. Some of these testcases will have been fully tested in all environments and be verified as approved. They are manually added to an approved list. Others are disapproved and are manually added to a disapproved list. A list is automatically generated comprising those testcases which are neither approved nor disapproved. They are not-tested. Those testcases on the disapproved and not-tested lists are excluded from the overall list. Some of the testcases correspond to defects in the system to be tested which have not yet been fixed. These are added to a defect list and the testcases in this list are also excluded from the overall list.

Claims

1. In a testcase management system comprising a test suite including a plurality of testcases cooperable with a test harness for executing a test run in which at least one of said testcases is executed on a system to be tested, a method of generating a list indicative of the testcases that are to be executed, said method comprising the steps of:

generating a list of all testcases in said test suite;

providing a list of those testcases which are approved;

providing a list of those testcases which are disapproved;

generating a defect list of those testcases corresponding to defects in said system to be tested which have not been fixed;

generating a further list of testcases, indicative of the cases to be executed on said test run, said cases excluding;

cases from the disapproved list;

cases from the all testcases list which are neither in the approved nor disapproved lists; and

cases from the defect list.

2. The method of claim 1, wherein said further list generating step generates an exclusion list comprising cases which are not to be tested.

3. The method of claim 2, wherein the exclusion list further comprises information as to why a particular testcase is excluded.

4. The method of claim 1, wherein said further list generating step generates a list comprising cases which are to be tested.

5. The method of claim 1, wherein said cases which are on neither the approved nor disapproved lists are not-tested, and wherein the method further comprises:

generating a not-tested list of cases.

6. The method of claim 5, further comprising:

repairing a disapproved testcase;

removing said disapproved testcase from the disapproved list;

re-generating the not-tested list, wherein said not-tested list includes the repaired testcase.

7. The method of claim 5, further comprising:

periodically testing the cases on the not-tested list; and

responsive to said cases executing successfully, adding the successful cases to the approved list.

8. The method claim 5, further comprising:

periodically testing the cases on the not-tested list; and

responsive to said cases executing unsuccessfully, adding said unsuccessful cases to the disapproved list.

9. The method of claim 1, further comprising:

providing at least one version list of testcases, an or each version list being associated with a respective version of the system to be tested and comprising a list of testcases corresponding to defects which have been corrected in said version of the system, wherein said further list generating step generates a list including cases from those version lists relating to versions later than the version of the system to be tested.

10. The method of claim 9 wherein the or each version list comprises testcases which correspond to defects which have been fixed in that version of the system to be tested irrespective of operating system.

11. The method of claim 9, wherein the or each version list is associated with a version and an operating system on which said system to be tested is associated.

12. The method of claim 11, wherein the further list generating step generates a list including cases from those version lists relating to versions both equal to or earlier than the version of the system to be tested, and corresponding to an operating system other than that which the version of the system to be tested runs.

13. The method of claim 1, wherein one or all of the lists comprise a text file of testcase identifiers and wherein each testcase identifier is a path name to each of said testcases.

14. The method of claim 13, wherein each testcase identifier is associated with a testcase writer identifier.

15. The method of claim 14 further comprising informing a testcase writer that one of their testcases is disapproved.

16. The method of claim 15, wherein said testcase writer is informed via electronic mail.

17. The method of claim 1, wherein the disapproved list of cases further comprises information relating to why each case is disapproved.

18. The method of claim 1, wherein a testcase may be excluded for more than one reason.

19. An apparatus cooperable with a testcase management system comprising a test suite including a plurality of testcases, said apparatus being adapted to generate a list indicative of testcases that are to be executed under the control of a test harness in a test run in which one or more of said test cases is executed on a system to be tested, said apparatus comprising:

means for generating a list of all testcases in said test suite, wherein said testcases are either approved, disapproved or not-tested;

means for generating a defect list of those testcases corresponding to defects in said system to be tested which have not been fixed;

means for generating a further list of testcases, indicative of the cases to be executed on said test run, said cases excluding;

cases which are disapproved;

cases which are not-tested; and

cases from the defect list.

20. A computer program product comprising computer program code stored on a computer readable storage medium, the program code comprising an apparatus for, when executed on a computer, cooperable with a testcase management system comprising a test suite including a plurality of testcases, said computer program product adapted to generate a list indicative of the testcases that are to be executed under the control of a test harness in a test run in which one or more of said testcases is executed on a system to be tested, said computer program product comprising:

program code for generating a list of all testcases in said test suite, wherein said testcases are either approved, disapproved or not-tested;

program code for generating a defect list of those testcases corresponding to defects in said system to be tested which have not been fixed;

program code for generating a further list of testcases, indicative of the cases to be executed on said test run, said cases excluding;

cases which are disapproved;

cases which are not-tested; and

cases from the defect list.