Steady state computer testing

• US 8,024,615 B2
• Filed: 04/28/2008
• Issued: 09/20/2011
• Est. Priority Date: 04/28/2008
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tarta, Mihail G.
• Primary Examiner(s): GUYTON, PHILIP A
• Application Number:
  US12/110,350
• Publication Number:
  US 20090271662A1
• Time in Patent Office: 1,240 Days
• Field of Search: 714/37, 702/182, 702/183, 702/186
• US Class Current: 714/37
• CPC Class Codes: G06F 11/3672 Test management

Abstract

A steady state value of each parameter in a set of one or more parameters of a computer system under test is defined, and acts related to the parameter are performed while the system under test is processing one or more test workloads. The acts can include comparing a working value of the parameter with the steady state value of the parameter, as well as operating on one or more workloads to bring the working value closer to a steady state value. This can include increasing or decreasing the working value of the parameter, such as by admitting one or more workloads to the system under test or cancelling one or more workloads from the system under test.

Claims

1. A computer-implemented computer system testing method, comprising:

defining a target steady state value of each parameter of a set of one or more parameters of a computer system under test; and

while the system under test is processing one or more of a plurality of test workloads, for each parameter in the set of parameters;

determining whether a working value of the parameter is below the steady state value of the parameter, and if the working value of the parameter is below the steady state value, then operating on one or more of the workloads to increase the working value of the parameter; and

determining whether the working value of the parameter is above the steady state value of the parameter, and if the working value of the parameter is above the steady state value, then operating on one or more of the workloads to decrease the working value of the parameter, operating on one or more of the workloads to decrease the working value of the parameter comprising choosing one or more chosen workloads in the system under test to cancel, and instructing the system under test to cancel the one or more chosen workloads, choosing the one or more chosen workloads in the system under test comprising dynamically sorting workloads in the system under test based on runtime feedback from the system under test, and choosing the one or more chosen workloads based on the sorting.

2. The method of claim 1, wherein operating on one or more of the workloads to increase the working value of the parameter comprises admitting one or more workloads to the system under test.

3. The method of claim 1, further comprising modifying the steady state value of at least one parameter of the set of parameters during a continuous test of the system under test, the continuous test comprising a first period of time and a second period of time, such that during the first period of time the steady state value is at a low steady state value, and during the second period of time the steady state value is at a high steady state value that is higher than the low steady state value.

4. The method of claim 1, wherein the testing method comprises a long-haul steady state test of the system under test.

5. The method of claim 1, wherein the steady state value is a range.

6. The method of claim 1, wherein the set of parameters comprises at least two parameters.

7. The method of claim 6, wherein operating on one or more of the workloads to increase the working value of the parameter comprises:

choosing a workload for admission to the system under test based on an expected effect that the workload will have on the working values of at least two parameters in the set of parameters when the workload is processed by the system under test; and

admitting the chosen workload to the system under test.

8. The method of claim 6, wherein operating on one or more of the workloads to decrease the working value of the parameter comprises:

choosing a workload for cancellation from the system under test based on an expected effect that cancellation of the chosen workload will have on the working values of at least two parameters in the set of parameters; and

cancelling the chosen workload.

9. A computer testing system comprising:

at least one processor;

at least one memory storing computer-executable instructions for interfaces comprising;

an execution context interface comprising information about a system under test, wherein the computer testing system is configured for the at least one processor to use the information about the system under test to communicate with the system under test; and

a policy interface;

wherein the at least one memory comprises instructions stored thereon that when executed by the at least one processor cause the at least one processor to perform acts while the system under test is processing one or more workloads admitted to the system under test, the acts comprising;

using the policy interface to determine, based at least in part on feedback received from the system under test, whether the system under test is in an underflow state with respect to a parameter of the system under test;

operating on one or more workloads to increase a working load on the system under test if the system under test is in the underflow state with respect to the parameter of the system under test;

using the policy interface to determine, based at least in part on feedback received from the system under test, whether the system under test is in an overflow state with respect to the parameter of the system under test; and

operating on one or more workloads to decrease the working load on the system under test if the system under test is in an overflow state with respect to the parameter of the system under test, operating on one or more workloads to decrease the working load on the system under test comprising choosing one or more chosen workloads in the system under test to cancel, and instructing the system under test to cancel the one or more chosen workloads, choosing the one or more chosen workloads in the system under test comprising dynamically sorting workloads in the system under test based on runtime feedback from the system under test, and choosing the one or more chosen workloads based on the sorting.

10. The system of claim 9, wherein:

using the policy interface to determine whether the system under test is in an underflow state comprises reviewing feedback from the system under test multiple times while the system under test is processing the one or more workloads; and

operating on one or more workloads to increase the working load comprises admitting no more than a predetermined maximum number of admission workloads after each time that the testing system reviews feedback from the system under test.

11. The system of claim 9, wherein:

using the policy interface to determine whether the system under test is in an overflow state comprises reviewing feedback from the system under test multiple times while the system under test is processing the one or more workloads; and

operating on one or more workloads to decrease the working load comprises instructing the system under test to cancel no more than a predetermined maximum number of cancellation workloads after each time that the testing system reviews feedback from the system under test.

12. One or more computer-readable storage media containing one or more programs, the one or more programs comprising instructions that are executable by a computer system to perform acts comprising:

receiving feedback from a system under test while the system under test is being tested, the feedback including information about a working value of each parameter in a set of one or more parameters of the system under test; and

for each parameter in the set of parameters;

while the system under test is being tested, using the feedback to compare the working value of the parameter with a target steady state value of the parameter; and

while the system under test is being tested, operating on one or more workloads to bring the working value of the parameter closer to the steady state value of the parameter, wherein operating on one or more workloads to bring the working value of the parameter closer to the steady state value comprises choosing one or more chosen workloads in the system under test to cancel, and instructing the system under test to cancel the one or more chosen workloads, choosing the one or more chosen workloads in the system under test comprising dynamically sorting workloads in the system under test based on runtime feedback from the system under test, and choosing the one or more chosen workloads based on the sorting.

13. One or more computer readable media as recited in claim 12, wherein the set of parameters comprises at least two parameters.

14. One or more computer readable media as recited in claim 13, wherein using the feedback to compare the working value of the parameter with the steady state value comprises repeatedly using the feedback to compare the working value of the parameter with the steady state value.

15. One or more computer readable media as recited in claim 13, wherein the acts further comprise:

modifying a first steady state value during a continuous test of the system under test, the continuous test comprising a first period of time and a second period of time, such that during the first period of time the first steady state value is at a low first steady state value, and during the second period of time the first steady state value is at a high first steady state value that is higher than the low first steady state value; and

modifying a second steady state value during the continuous test, such that during the first period of time the second steady state value is at a high second steady state value, and during the second period of time the second steady state is at a low second steady state value that is lower than the high second steady state value.

16. One or more computer readable media as recited in claim 12, wherein operating on one or more of the workloads to bring the working value of the parameter closer to the steady state value comprises admitting one or more workloads to the system under test.

17. One or more computer readable media as recited in claim 12, wherein operating on one or more of the workloads to bring the working value of the parameter closer to the steady state value comprises adjusting the system under test'"'"'s usage of a workload already residing in the system under test.