Software performance testing with minimum iterations

• US 7,457,723 B2
• Filed: 03/01/2006
• Issued: 11/25/2008
• Est. Priority Date: 03/01/2006
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhu, Hui, Piira, Jeffrey S., Barber, Chris L., Beegle, Ryan B.
• Primary Examiner(s): Raymond; Edward
• Application Number:
  US11/365,163
• Publication Number:
  US 20070220349A1
• Time in Patent Office: 1,000 Days
• Field of Search: 702182-185, 702/188, 709/201, 709/212
• US Class Current: 702/182
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis

Abstract

Systems and methods are disclosed for providing software performance testing. The results of a test may be determined valid if the test reached a defined test criteria after performing N number of iterations. If the test has not reached the defined test criteria and if N is less than a maximum value, one may be added to N and additional iteration of the test may be performed until the test reaches the defined test criteria or until N reaches the maximum value. If the test reaches the defined test criteria after any additional iteration of the test before N reaches the maximum value, the results of the test may be deemed valid. Moreover, if the test reached the defined test criteria with an outlier removed from the results, the results of the test may be deemed valid. Otherwise, the results of the test may be deemed invalid.

Claims

1. A method for providing performance testing, the method comprising:

performing N number of iterations of a test;

determining that results of the test are valid if the test reached a defined test criteria after performing the N number of iterations;

adding, if the test has not reached the defined test criteria and if N is less than a maximum value, one to N and performing an additional iteration of the test until one of the following occurs;

the test reaches the defined test criteria and N reaches the maximum value, wherein if the test reaches the defined test criteria after any additional iteration of the test before N reaches the maximum value, the results of the test are deemed valid;

determining that the results of the test are valid if N reached the maximum value and if the test reached the defined test criteria with an outlier removed from the results;

determining that the results of the test are invalid if N reached the maximum value and if the test did not reach the defined test criteria with an outlier removed from results;

deeming the test as passing if the test results compare favorably to a goal value; and

deeming the test as failing if the test results compare unfavorably to the goal value.

2. The method of claim 1, wherein the test is configured to measure the amount of time it takes for a computer program to perform a predetermined operation.

3. The method of claim 1, wherein performing N number of iterations of the test comprises performing N number of iterations of the test wherein N is initially set to a minimum value determined from historic data.

4. The method of claim 1, wherein performing N number of iterations of the test comprises performing N number of iterations of the test wherein N is initially set to a minimum value comprising the number of iterations it took 50% of historic benchmark tests to reach their test criteria.

5. The method of claim 1, wherein the defined test criteria comprises:

$\frac{\mathrm{Err}}{\stackrel{\_}{X}}\le <br><br>Z;<br><br>$

where X is an average value of the results, and Err is the error margin associated with the results.

6. The method of claim 1, wherein the maximum value for N is set to a value determined from historic data.

7. The method of claim 1, wherein the maximum value for N is set to a value comprising the number of iterations it took 75% of historic benchmark tests to reach their test criteria.

8. The method of claim 1, wherein the goal value comprises a time period a computer program is tested for by the test to determine if the computer program can perform a predetermined operation within.

9. A system for providing software performance testing, the system comprising:

a memory storage; and

a processing unit coupled to the memory storage, wherein the processing unit is operative to;

perform N number of iterations of a test;

determine that results of the test are valid if the test reached a defined test criteria after performing the N number of iterations wherein the defined test criteria comprises

$\frac{\mathrm{Err}}{\stackrel{\_}{X}}\le <br><br>Z$

where X is an average value of the results and Err is the error margin associated with the results;

add, if the test has not reached the defined test criteria and if N is less than a maximum value, one to N and perform an additional iteration of the test until one of the following occurs;

the test reaches the defined test criteria and N reaches the maximum value, wherein if the test reaches the defined test criteria after any additional iteration of the test before N reaches the maximum value, the results of the test are deemed valid;

determine that the results of the test are valid if N reached the maximum value and if the test reached the defined test criteria with an outlier removed from the results;

determine that the results of the test are invalid if N reached the maximum value and if the test did not reach the defined test criteria with an outlier removed from results;

deem the test as passing if the test results compare favorably to a goal value; and

deem the test as failing if the test results compare unfavorably to the goal value.

10. The system of claim 9, wherein N is initially set to a minimum value comprising the number of iterations it took 50% of historic benchmark tests to reach their test criteria.

11. The system of claim 9, wherein Z is less than or equal to 5%.

12. The system of claim 9, wherein the maximum value for N is set to a value determined from historic data.

13. The system of claim 9, wherein the maximum value for N is set to a value comprising the number of iterations it took 75% of historic benchmark tests to reach their test criteria.

14. The system of claim 9, wherein the outlier comprises a data point within the results that is not in the range of [ X−

Err, X+Err] and the one most far away from a mean of the results where X is the average value of the results, and Err is the error margin associated with the results.

15. A computer-readable medium which stores a set of instructions which when executed performs a method for providing software performance testing, the method executed by the set of instructions comprising:

performing N number of iterations of a test wherein the test is configured to measure the amount of time it takes for a computer program to perform a predetermined operation;

determining that results of the test are valid if the test reached a defined test criteria after performing the N number of iterations;

adding, if the test has not reached the defined test criteria and if N is less than a maximum value, one to N and performing an additional iteration of the test until one of the following occurs;

the test reaches the defined test criteria and N reaches the maximum value, wherein if the test reaches the defined test criteria after any additional iteration of the test before N reaches the maximum value, the results of the test are deemed valid;

determining that the results of the test are valid if N reached the maximum value and if the test reached the defined test criteria with an outlier removed from the results wherein the outlier comprises a data point within the results that is not in the range of [ X−

Err, X+Err] and the one most far away from a mean of the results where X is the average value of the results, and Err is the error margin associated with the results;

determining that the results of the test are invalid if N reached the maximum value and if the test did not reach the defined test criteria with an outlier removed from results;

deeming the test as passing if the test results compare favorably to a goal value; and

deeming the test as failing if the test results compare unfavorably to the goal value.

16. The computer-readable medium of claim 15, wherein performing N number of iterations of the test comprises performing N number of iterations of the test wherein N is initially set to a minimum value determined from historic data.

17. The computer-readable medium of claim 15, wherein performing N number of iterations of the test comprises performing N number of iterations of the test wherein N is initially set to a minimum value comprising the number of iterations it took 50% of historic benchmark tests to reach their test criteria.

18. The computer-readable medium of claim 15, wherein the defined test criteria comprises:

$\frac{\mathrm{Err}}{\stackrel{\_}{X}}\le <br><br>5<br><br>\%;<br><br>$

where X is an average value of the results, and Err is the error margin associated with the results.

19. The computer-readable medium of claim 15, wherein the maximum value for N is set to a value determined from historic data.

20. The computer-readable medium of claim 15, wherein the maximum value for N is set to a value comprising the number of iterations it took 75% of historic benchmark tests to reach their test criteria.