Software Testing Method and System

• US 20080046791A1
• Filed: 06/05/2007
• Published: 02/21/2008
• Est. Priority Date: 06/16/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Carter, Marc Stanley, Taylor, Michael George, Bicheno, Richard Michael, Dunn, Tim, Braithwaite, Kevin Anthony
• Granted Patent: US 8,056,060 B2
• Time in Patent Office: Days
• US Class Current: 714/742
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

A software product is tested by first obtaining a performance matrix for the software product, the performance matrix containing the profile results of a plurality of tests on the software product, and an expected result vector for the plurality of tests. A test sequence is then executed for the software product, the sequence comprising selecting a subset of the plurality of tests, running the test subset to obtain a new result vector for the test subset, comparing the new result vector entry with the expected result vector entry for the same test, selecting a test (which may be one of the subset or may be a new test) according to the outcome of the result vector comparison and the performance matrix, and running the selected test under profile.

Claims

1. A method for testing a software product comprising:

obtaining a performance matrix for the software product, the performance matrix containing the profile results of a plurality of tests on the software product, and an expected result vector for the plurality of tests; and

executing a test sequence for the software product comprising

selecting a subset of the plurality of tests,

running the test subset to obtain a new result vector for the test subset,

comparing the new result vector entry with the expected result vector entry for the same test,

selecting a test according to the outcome of the result vector comparison and the performance matrix, and

running the selected test under profile.

2. A method according to claim 1 wherein the profile results, for each test stored in the performance matrix, comprise weighting data for each of a plurality of functional areas.

3. A method according to claim 2 wherein the step of selecting a test according to the outcome of the result vector comparison and the performance matrix includes identifying a functional area in which performance has degraded.

4. A method according to claim 3 wherein the step of selecting a test according to the outcome of the result vector comparison and the performance matrix includes choosing a test specifically for the functional area in which a degraded performance is detected.

5. A method according to claim 4 wherein the selected test to be run under profile is not a member of the test subset.

6. A method according to claim 4 wherein the step of selecting a subset of the plurality of tests comprises recalling a predefined test subset.

7. A method according to claim 4 wherein the result vector contains numerical values.

8. A system for testing a software product comprising a storage device storing a performance matrix for the software product the performance matrix containing the profile results of a plurality of tests on the software product, and an expected result vector for the plurality of tests, and a processor arranged to execute a test sequence for the software product comprising:

selecting a subset of the plurality of tests,

running the test subset to obtain a new result vector for the test subset,

comparing the new result vector entry with the expected result vector entry for the same test,

selecting a test according to the outcome of the result vector comparison and the performance matrix, and

running the selected test under profile.

9. A system according to claim 8 wherein the profile results, for each test stored in the performance matrix, comprise weighting data for each of a plurality of functional areas.

10. A system according to claim 9 wherein the processor is arranged, when selecting a test according to the outcome of the result vector comparison and the performance matrix, to identify a functional area in which performance has degraded.

11. A system according to claim 10 wherein the processor is arranged, when selecting a test according to the outcome of the result vector comparison and the performance matrix, to choose a test specifically for the functional area in which a degraded performance is detected.

12. A system according to claim 10 wherein the selected test, to be am under profile is not a member of the test subset.

13. A system according to claim 10 wherein the processor is arranged, when selecting a subset of the plurality of tests, to recall a predefined test subset.

14. A computer program product comprising a computer usable media embodying program code for testing a software product, the program code when loaded onto and executed in a computer causing the computer to perform a method comprising the steps of:

obtaining a performance matrix for the software product, the performance matrix containing the profile results of a plurality of tests on the software product, and an expected result vector for the plurality of tests; and

executing a test sequence for the software product comprising;

selecting a subset of the plurality of tests,

running the test subset to obtain a new result vector for the test subset,

comparing the new result vector entry with the expected result vector entry for the same test and the performance matrix,

selecting a test according to the outcome of the result vector comparison and the performance matrix, and

running the selected test under profile.

15. A computer program product according, to claim 14 wherein the profile results, for each test stored in the performance matrix, comprise weighting data for each of a plurality of functional areas.

16. A computer program product according to claim 15 wherein the program instructions for causing the computer to perform the step of selecting a test according to the outcome of the result vector comparison and the performance matrix, includes further program instructions for causing the computer to identify a functional area in which performance has degraded.

17. A computer program product according to claim 16 wherein the program instructions for causing the computer to perform the step of selecting a test according to the outcome of the result vector comparison and the performance matrix, includes program instructions that cause the computer to perform the step of choosing a test specifically for the functional area in which a degraded performance is detected.

18. A computer program product according to claim 17 wherein the selected test to be run under profile is not a member of the test subset.

19. A computer program product according to claim 17 wherein the program instructions for causing the computer to perform the step of selecting a subset of the plurality of tests comprises program instructions for causing the computer to perform the step of recalling a predefined test subset.

20. A computer program product according to claim 17 wherein the result vector contains numerical values.