Computer-Implemented Systems And Methods For Software Application Testing

• US 20090007078A1
• Filed: 06/29/2007
• Published: 01/01/2009
• Est. Priority Date: 06/29/2007
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: McCain, Christina Charisse, Hoyek, Nabil Mounir, Coleman, Oita Kayonna
• Granted Patent: US 8,087,001 B2
• Time in Patent Office: Days
• US Class Current: 717/131
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

Computer-implemented systems and methods for testing one or more software applications. For example, a system and method can be configured to perform a comparison between test data results and benchmark test data in order to determine test result differences between the received test data results and the retrieved benchmark test data. Testing criteria is applied to the determined test result differences in order to identify test deviations from the benchmark test data.

Claims

1. A computer-implemented method for testing software applications, comprising:

receiving test data results that were generated during testing of a software application;

retrieving from a historical test data store benchmark test data;

after retrieving the benchmark test data, automatically performing a comparison between the received test data results and the retrieved benchmark test data in order to determine test result differences between the received test data results and the retrieved benchmark test data;

automatically applying testing criteria to the determined test result differences in order to identify test deviations from the benchmark test data; and

storing the identified test deviations.

2. The method of claim 1 further comprising:

receiving an evaluation method; and

using the received evaluation method to determine type of comparison that is to be performed between the received test data results and the retrieved benchmark test data.

3. The method of claim 2 further comprising:

providing to a user through a graphical user interface a set of candidate evaluation methods;

receiving a selection from the user of one of the candidate evaluation methods;

wherein said using step uses the selected one of the candidate evaluation methods to determine the type of comparison that is to be performed between the received test data results and the retrieved benchmark test data.

4. The method of claim 3, wherein the candidate evaluation methods include an evaluation method selected from the group consisting of an absolute difference evaluation method, a percentage difference evaluation method, and a relative difference evaluation method.

5. The method of claim 1, wherein the testing criteria includes pre-specified acceptable deviations and unacceptable deviations between test results from a software test and the benchmark test data.

6. The method of claim 1, wherein the testing criteria includes pre-specified acceptable deviations or unacceptable deviations between test results from a software test and the benchmark test data.

7. The method of claim 1 further comprising:

(a) automatically storing the received test data results in a historical data store;

(b) receiving new test data results that were generated during a new testing of the software application;

(c) retrieving from the historical test data store the automatically stored test data results for use as the benchmark test data;

(d) after retrieving the stored test data results as the benchmark test data in step (c), automatically performing a comparison between the received new test data results and the retrieved benchmark test data in order to determine test result differences between the received new test data results and the retrieved benchmark test data;

(e) automatically applying testing criteria to the determined test result differences in order to identify test deviations from the benchmark test data; and

(f) providing to a software tester the identified test deviations with respect to the new test data results.

8. The method of claim 7 further comprising:

retrieving, from the historical test data store, multiple test data results that were generated over multiple tests;

performing historical trend analysis upon the retrieved multiple test data results;

providing to the software tester the historical trend analysis.

9. The method of claim 8, wherein the historical trend analysis includes plotting a statistical measure derived from the multiple test data results with respect to the times associated with the multiple test data results.

10. The method of claim 8, wherein the historical trend analysis includes plotting a plurality of statistical measures derived from the multiple test data results with respect to the times associated with the multiple test data results.

11. The method of claim 7 further comprising:

retrieving, from the historical test data store, multiple test data results that were generated over multiple tests;

performing testing forecasting analysis upon the retrieved multiple test data results;

providing to the software tester the testing forecasting analysis.

12. The method of claim 1, wherein the testing forecasting analysis provides an indication as to a future time when a testing metric is to satisfy one or more of the testing criteria.

13. The method of claim 1, wherein the received test data results are filtered before being said automatically performing the comparison between the received test data results and the retrieved benchmark test data.

14. The method of claim 1, wherein the software tester is a human software tester, a computer software tester program, or combinations thereof.

15. The method of claim 1, wherein the testing criteria is performance testing criteria;

wherein said automatically applying the testing criteria step comprises automatically applying testing criteria to the determined test result differences in order to identify performance test deviations from the benchmark test data.

16. The method of claim 15 further comprising:

performing functional testing upon the software application to determine whether said software application performed pre-specified software application functions.

17. The method of claim 16, wherein statistical analysis is performed upon the received testing data for use in said identifying of the performance deviations.

18. The method of claim 17, wherein the statistical analysis includes determining statistical values with respect to with the received test data results;

wherein the statistical values include mean, standard deviation, minimum and maximum values;

wherein the functional testing includes determining whether or not functions of the software application performed their respective intended function.

19. The method of claim 17, wherein the statistical analysis includes determining statistical values with respect to the received test data results and with respect to previously determined test data results;

wherein forecasting is performed using the statistical values determined for the received test data results and previously determined test data results;

wherein the forecasting determines when the software application is predicted to satisfy a pre-specified acceptable performance criteria;

wherein the functional testing includes determining whether or not functions of the software application performed their respective intended function.

20. A computer-implemented system for testing software applications, comprising:

software instructions to receive test data results that were generated during testing of a software application;

a historical data store to store benchmark test data;

software instructions to retrieve benchmark test data from the historical test data store;

comparison software instructions to perform, after the retrieving of the benchmark test data, a comparison between the received test data results and the retrieved benchmark test data in order to determine test result differences between the received test data results and the retrieved benchmark test data;

software instructions to apply testing criteria to the determined test result differences in order to identify test deviations from the benchmark test data; and

wherein the identified test deviations are provided to a software tester.

21. A computer-readable storage medium encoded with instructions that cause a computer to perform a method for testing software applications, said method comprising the steps of;

receiving test data results that were generated during testing of a software application;

retrieving from a historical test data store benchmark test data;

after retrieving the benchmark test data, automatically performing a comparison between the received test data results and the retrieved benchmark test data in order to determine test result differences between the received test data results and the retrieved benchmark test data;

automatically applying testing criteria to the determined test result differences in order to identify test deviations from the benchmark test data; and

providing to a software tester the identified test deviations.