System and method for flexible performance testing

• US 20070180093A1
• Filed: 01/19/2007
• Published: 08/02/2007
• Est. Priority Date: 09/27/2005
• Status: Abandoned Application

• Current Assignee: BEA Systems Incorporated (Oracle Corporation)
• Original Assignee: BEA Systems Incorporated (Oracle Corporation)
• Inventors: Roth, Steven, Maccaux, Matthew
• Application Number:
  US11/625,246
• Publication Number:
  US 20070180093A1
• Time in Patent Office: Days
• US Class Current: 709/223
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 43/50 Testing arrangements
• CPC Class Codes: H04L 43/55 Testing of service level quality, e.g. simulating service usage

Abstract

The present invention enables a performance testing framework that enables multiple components working together to test a deployed application automatically in an unattended manner and to analyze the test results easily. At very high level, the performance testing framework can run performance tests on a tested system with one or more variations without user intervention and save the test results and configuration metadata to a database for later analysis. The test run for each variation can be run flexibly in a plurality of phases, each of which can be added, configured or skipped for the variation. This description is not intended to be a complete description of, or limit the scope of, the invention. Other features, aspects, and objects of the invention can be obtained from a review of the specification, the figures, and the claims.

Claims

1. A system to support performance testing, comprising:

a tested system, comprising one or more of;

one or more application servers operable to deploy and provide an application to a user; and

a load generating tool operable to execute a plurality of test runs to measure performance of the application;

an infrastructure, comprising one or more of;

an automation controller operable to analyze performance data of the plurality of test runs within and/or between a plurality of variations;

a variation controller operable to;

interact with the tested system and the infrastructure;

specify the plurality of variations of configuration properties of the tested system;

generate the plurality of test runs for the plurality of variations;

invoking a dispatcher to process a single run of each of the plurality of test runs; and

said dispatcher operable to manage executing each of the plurality of test runs in a plurality of phases for each of the plurality of variations.

2. The system according to claim 1, wherein:

the application is a Web-based application.

3. The system according to claim 1, wherein:

each of the plurality of variations specifies one or more of the following per-phase component configuration settings;

installer, application, portal, network configuration, domain, startup script, database, proxy, one or more configured portal entities, and one or more performance test properties.

4. The system according to claim 1, wherein:

the performance data of the application is measured by one or more of;

response time, throughput, capacity, and scalability.

5. The system according to claim 1, wherein:

the dispatcher is operable to compute per-phase configuration settings of the plurality of phases and execute the plurality of phases with those settings when executing one of the plurality of test runs automatically.

6. The system according to claim 5, wherein:

the dispatcher is operable to skip one or more of the plurality of phases during execution of each of the plurality of test runs based on the per-phase configuration settings and history.

7. The system according to claim 1, wherein:

the dispatcher is operable to monitor, detect, and report the performance data and/or issues during execution of the plurality of phases of one of the plurality of test runs.

8. The system according to claim 7, wherein:

the dispatcher is operable to skip one or more of the plurality of phases based on the performance data and/or issues of one or more of its proceeding phases.

9. The system according to claim 7, wherein:

the dispatcher is operable to organize and store the performance result and/or issues into a plurality of results directories.

10. The system according to claim 7, wherein:

the dispatcher is operable to end the current test run based on performance data and/or issues of the current test run, variation, or the plurality of phases.

11. A method to support performance testing, comprising:

specifying a plurality of variations of configuration properties for performance test of a tested system;

generating a plurality of test runs for the plurality of variations;

managing to execute each of the plurality of test runs in a plurality of phases for each of the plurality of variations; and

analyzing performance data of the plurality of test runs within and/or between the plurality of variations.

12. The method according to claim 11, further comprising:

measuring performance of the tested system by one or more of;

response time, throughput, capacity, and scalability.

13. The method according to claim 11, further comprising:

computing per-phase configuration settings of the plurality of phases and executing the plurality of phases with those settings when executing one of the plurality of test runs automatically.

14. The method according to claim 13, further comprising:

skipping one or more of the plurality of phases during execution of each of the plurality of test runs based on the per-phase configuration settings and history.

15. The method according to claim 11, further comprising:

monitoring, detecting, and reporting the performance data and/or issues during execution of the plurality of phases each of the plurality of test runs.

16. The method according to claim 15, further comprising one or more of:

skipping one or more of the plurality of phases based on the performance data and/or issues of one or more of its proceeding phases;

organizing and storing the performance result and/or issues into a plurality of results directories; and

ending the current test run based on performance data and/or issues of the current test run, variation, or the plurality of phases.

17. The method according to claim 11, further comprising one or more of:

storing the performance data and/or configuration of the plurality of test runs to a performance test database;

summarizing and reporting analysis result of the performance data to a user;

presenting the analysis result to the user on a Web browser; and

exporting the analysis result to a spreadsheet.

18. A machine readable medium having instructions stored thereon that when executed cause a system to:

specify a plurality of variations of configuration properties for performance test of a tested system;

generate a plurality of test runs for the plurality of variations;

manage to execute each of the plurality of test runs in a plurality of phases for each of the plurality of variations; and

analyze performance data of the plurality of test runs within and/or between the plurality of variations.

19. A system to support performance testing, comprising:

means for specifying a plurality of variations of configuration properties for performance test of a tested system;

means for generating a plurality of test runs for the plurality of variations;

means for managing to execute each of the plurality of test runs in a plurality of phases for each of the plurality of variations; and

means for analyzing performance data of the plurality of test runs within and/or between the plurality of variations.