Method and apparatus for testing and evaluating a software component using an abstraction matrix

• US 6,986,125 B2
• Filed: 10/23/2001
• Issued: 01/10/2006
• Est. Priority Date: 08/01/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hoskins, Curtis L., Apuzzo, Joseph T., Suri, Hemant R., Marino, John P., Race, Timothy L.
• Primary Examiner(s): DAS, CHAMELI
• Application Number:
  US10/006,596
• Publication Number:
  US 20030037314A1
• Time in Patent Office: 1,540 Days
• Field of Search: 717/124, 717/101, 717/107, 717/126, 714/33, 714/38, 714/1, 714/48, 714/715, 714/723, 709/223
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3696 Methods or tools to render software testable

Abstract

A functional testing and evaluation technique is provided employing an abstraction matrix that describes a complex software component to be tested. The abstraction matrix includes at least one test case scenario and mapped expected results therefore. Test cases are derived from the at least one test case scenario and used to test the software component, thereby generating test results. The test results are automatically evaluated using the abstraction matrix. The evaluating includes comparing a test case to the at least one test case scenario of the abstraction matrix and if a match is found, comparing the test result for that test case with the mapped expected result therefore in the abstraction matrix.

Claims

1. A computer implemented method stored on at least one computer readable medium and executing on at least one computer for evaluating a software component, comprising:

deriving a conceptual layering scheme of the software component, the conceptual layering scheme dividing the software component into multiple software layers, the multiple software layers taking into account relationships that exist between software layers;

providing an abstraction matrix in mathematical abstract form that describes the multiple software layers of the software component and accounts for the relationships that exist between the software layers, the abstraction matrix further comprising at least one test case scenario for each software layer and mapped expected results therefor;

testing the multiple software layers of the software component using the test case scenarios to generate test results;

evaluating the test results using the abstraction matrix, the evaluating including for each software layer automatically comparing a test case employed in testing of the software layer to the at least one test case scenario of the abstraction matrix therefor, and if a match is found, automatically comparing the test result for that test case with the mapped expected result for that test case scenario in the abstraction matrix; and

wherein evaluation of the software component is accomplished upon completion of evaluation of the test results.

2. The method of claim 1, further comprising, prior to said testing, deriving at least some test cases for each software layer from the at least one test case scenario of the abstraction matrix.

3. The method of claim 1, wherein the software component comprises multiple states, and wherein the abstraction matrix comprises multiple test case scenarios, each test case scenario being associated with a different state of the software component.

4. The method of claim 1, further comprising generating an error log, said error log containing a list of test cases each having a test result which failed to match the mapped expected result therefor in a matching test case scenario of the abstraction matrix.

5. The method of claim 4, wherein said error log further includes the test results of failing test cases, as well as the mapped expected results from the abstraction matrix for the failing test cases.

6. The method of claim 1, further comprising creating at least one test results file and at least one abstraction matrix file, and wherein said evaluating comprises automatically reviewing each test result in the at least one test results file by comparing its test case to the at least one test case scenario of the abstraction matrix contained in the at least one abstraction matrix file.

7. The method of claim 1, wherein the providing comprises creating the abstraction matrix from a functional specification of the software component.

8. The method of claim 1, wherein the evaluating comprises automatically extracting test statistics, said test statistics including a total number of test cases executed, and at least one of a total number of test cases successfully executed or a total number of test cases unsuccessfully executed.

9. The method of claim 1, wherein the providing further includes parsing the abstraction matrix to automatically generate and factor out doable test cases and mapped expected results therefor, and separating the test cases based on the software layers of the software component, and associating data structures with the separated test cases of the software layers, the data structures allowing the test cases of the multiple software layers to be uncorrelated, and wherein the abstraction matrix comprises an abstraction file for each software layer, and the parsing comprises generating mapped expected result for each line of each abstraction file based on associated state information.

10. The method of claim 9, wherein the separating comprises eliminating dependencies between at least one input of a software layer based on other inputs of other software layers of the software component thereby uncorrelating test cases of the multiple software layers.

11. The method of claim 10, wherein the data structures for at least some software layers mimic information passed between layers during normal operation of the software component.

12. The method of claim 11, wherein the software component comprises a cluster operating system component.

13. A computer implemented system stored on at least one computer readable medium and executing on at least one computer for evaluating a software component, comprising:

means for deriving a conceptual layering scheme of the software component, the conceptual layering scheme dividing the software component into multiple software layers, the multiple software layers taking into account relationships that exist between software layers;

an abstraction matrix in mathematical abstract form that describes the multiple software layers of the software component and accounts for the relationships that exist between software layers, the abstraction matrix further comprising at least one test case scenario for each software layer and mapped expected results therefor;

means for testing the multiple software layers of the software component using the test case scenarios to generate test results;

means for evaluating the test results using the abstraction matrix, the means for evaluating including for each software layer means for automatically comparing a test case employed in testing of the software layer to the at least one test case scenario of the abstraction matrix therefor, and if a match is found, for automatically comparing the test result for that test case with the mapped expected result for that test case scenario in the abstraction matrix; and

wherein evaluation of the software component is accomplished upon completion of the evaluation of the test results.

14. The system of claim 13, further comprising means for deriving at least some test cases for each software layer from the at least one test case scenario of the abstraction matrix for use by the means for testing.

15. The system of claim 13, wherein the software component comprises multiple states, and wherein the abstraction matrix comprises multiple test case scenarios, each test case scenario having associated with a different state of the software component.

16. The system of claim 13, further comprising means for generating an error log, said error log containing a list of test cases each having a test result which failed to match the mapped expected result therefor in a matching test case scenario of the abstraction matrix.

17. The system of claim 16, wherein said error log further includes test results of failing test cases, as well as mapped expected results from the abstraction matrix for the failing test cases.

18. The system of claim 13, further comprising means for creating at least one test results file and at least one abstraction matrix file, and wherein said means for evaluating comprises means for automatically reviewing each test result in the at least one test results file by comparing its test case to the at least one test case scenario of the abstraction matrix contained in the at least one abstraction matrix file.

19. The system of claim 13, further comprising means for creating the abstraction matrix from a functional specification of the software component.

20. The system of claim 13, wherein the means for evaluating comprises means for automatically extracting test statistics, said test statistics including a total number of test cases executed, and at least one of a total number of test cases successfully executed or a total number of test cases unsuccessfully executed.

21. A computer implemented system stored on at least one computer readable medium and executing on at least one computer for evaluating a software component, comprising:

means for deriving a conceptual layering scheme of the software component, the conceptual layering scheme dividing the software component into multiple software layers, the multiple software layers taking into account relationships that exist between software layers;

an abstraction matrix in mathematical abstract form that describes the multiple software layers of the software component and accounts for the relationships that exist between the software layers, the abstraction matrix further comprising at least one test case scenario for each software layer and mapped expected results therefor;

a first computing unit adapted to test the multiple software layers of the software component using the test case scenarios to generate test results;

a second computing unit adapted to evaluate the test results using the abstraction matrix, the evaluating including for each software layer automatically comparing a test case employed in testing of the software layer to the at least one test case scenario of the abstraction matrix therefor, and if a match is found, automatically comparing the test result for that test case with the mapped expected result for that test case scenario in the abstraction matrix; and

wherein evaluation of the software component is accomplished upon completion of the evaluation of the test results.

22. The system of claim 21, wherein the first computing unit and the second computing unit comprise a single computing unit.

23. At least one program storage device readable by a machine, tangibly embodying at least one program of instructions executable by the machine to perform a method of evaluating a software component, comprising:

deriving a conceptual layering scheme of the software component, the conceptual layering scheme dividing the software component into multiple software layers, the multiple software layers taking into account relationships that exist between software layers;

providing an abstraction matrix in mathematical abstract form that describes the multiple software layers of the software component and accounts for the relationships that exist between the software layers, the abstraction matrix further comprising at least test case scenario for each software layer and mapped expected results therefor;

testing the multiple software layers of the software component using the test case scenarios to generate test results;

evaluating the test results using the abstraction matrix, the evaluating including for each software layer automatically comparing a test case employed in the testing of the software layer to the at least one test case scenario of the abstraction matrix therefor, and if a match is found, automatically comparing the test result for that test case with the mapped expected result for that test case scenario in the abstraction matrix; and

wherein evaluation of the software component is accomplished upon completion of the evaluation of the test results.

24. The at least one program storage device of claim 23, further comprising, prior to said testing, deriving at least some test cases for each layer from the at least one test case scenario of the abstraction matrix.

25. The at least one program storage device of claim 23, wherein the software component comprises multiple states, and wherein the abstraction matrix comprises multiple test case scenarios, each test case scenario being associated with a different state of the software component.

26. The at least one program storage device of claim 23, further comprising generating an error log, said error log containing a list of test cases each having a test result which failed to match the mapped expected result therefor in a matching test case scenario of the abstraction matrix.

27. The at least one program storage device of claim 26, wherein said error log further includes test results of failing test cases, as well as mapped expected results from the abstraction matrix for the failing test cases.

28. The at least one program storage device of claim 23, further comprising creating at least one test results file and at least one abstraction matrix file, and wherein said evaluating comprises automatically reviewing each test result in the at least one test results file by comparing its test case in the at least one test case scenario of the abstraction matrix contained in the at least one abstraction matrix file.

29. The at least one program storage device of claim 23, wherein the providing comprises creating the abstraction matrix from a functional specification of the software component.

30. The at least one program storage device of claim 23, wherein the evaluating comprises automatically extracting test statistics, said test statistics including a total number of test cases executed, and at least one of a total number of test cases successfully executed or a total number of test cases unsuccessfully executed.