Statistical analysis of sampled profile data in the identification of significant software test performance regressions

• US 20070061626A1
• Filed: 09/14/2005
• Published: 03/15/2007
• Est. Priority Date: 09/14/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Nelson, Bruce, Klamik, Brian
• Granted Patent: US 7,577,875 B2
• Time in Patent Office: Days
• US Class Current: 714/38
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

Sampled profile data provides information about processor activity during a test. Processor activity can be analyzed to determine an amount of processor resources used to execute the various functions, modules, and processes associated with a tested software activity. Statistical methods can be applied to the resource data from multiple test runs to determine whether a significant regression has occurred between a baseline test pass and a daily test pass. By collecting data at the function, module and process levels, significant regressions may be uncovered at any of the levels. Regressions may also be ranked according to their importance, which allows for identification and notification of development teams responsible for significant regressions.

Claims

1. In a system for analyzing computer test results, said test results comprising a plurality of sampled processor events, a method comprising:

computing at least one amount of processor resources that was used to execute a function;

computing at least one error amount associated with said amount of processor resources;

comparing said amount of processor resources and said error amount with amounts from a previous test;

finding a significant regression if said amount of processor resources differs from an amount of processor resources from the previous test, after accounting for said error amount and an error amount from the previous test.

2. The method of claim 1, further comprising ranking the regression among a plurality of regressions according to an amount of difference between said amount of processor resources and the amount of processor resources from the previous test.

3. The method of claim 1, wherein computing at least one amount of processor resources comprises computing a mean amount of processor resources used to execute the function, and where the mean is computed using results from multiple runs of a test.

4. The method of claim 3, wherein said computing at least one error amount comprises computing a statistical error amount based at least in part on a number of runs of the test.

5. The method of claim 1, further comprising computing at least one amount of processor resources that was used to execute a module by combining one or more amounts of processor resources that were used to execute one or more functions associated with the module.

6. The method of claim 5, further comprising computing at least one module error amount.

7. The method of claim 6, further comprising finding a module regression if said amount of processor resources that was used to execute a module differs from an amount of processor resources that was used to execute the module from the previous test, after accounting for said error amount and an error amount from the previous test.

8. A system for testing computer software, comprising:

a test harness for causing a plurality of test runs, wherein tested software executes on a computer processor in each test run;

a profiler for sampling processor activity during execution of said tested software, thereby generating sample data;

software for calculating from said sample data an amount of processor resources in each test run that was dedicated to executing at least one function associated with the tested software;

software for calculating from said amount of processor resources in each test run a mean amount of processor resources dedicated to executing the at least one function and an error amount associated with said mean amount;

software for comparing said mean amount and said error amount to a previous mean amount and a previous error amount from a previous test to determine if there is a significant regression in the amount of processor resources dedicated to executing the at least one function.

9. The system of claim 8, wherein said software for comparing comprises instructions for ranking regressions according to an amount of difference between said mean amount and said previous mean amount, while accounting for said error amount and said and previous error amount.

10. The system of claim 8, wherein said software for calculating from said sample data an amount of processor resources also calculates an amount of processor resources in each test run that was dedicated to executing at least one module associated with the tested software.

11. The system of claim 8, wherein said error amount represents a 75%-99% confidence in said mean amount of processor resources dedicated to executing the at least one function.

12. The system of claim 8, further comprising software for discovering from said sample data at least on function, module, and process associated with the tested software.

13. The system of claim 8, wherein the profiler is an Event Tracing for WINDOWS®

(ETW) profiler.

14. A method to be carried out by a computer for ranking software test regressions, comprising:

calculating a mean amount of processor resources that are dedicated to executing a function;

calculating an error associated with said mean amount;

calculating a difference between said mean amount and a previous mean amount, wherein said error amount to and a previous error amount are accounted for;

ranking a regression based on said difference.

15. The method of claim 14 wherein said calculating a mean amount, calculating an error, calculating a difference, and ranking are also carried out for a module.

16. The method of claim 14 wherein said calculating a mean amount, calculating an error, calculating a difference, and ranking are also carried out for a process.

17. The method of claim 14 wherein said processor resources are measured in Central Processing Unit (CPU) clock cycles.

18. The method of claim 14 wherein said difference is calculated using the equation |D|*(1−

E/|D|).

19. The method of claim 14, further comprising filtering the regression if the error amount and the previous error amount are greater than a simple difference between said mean amount and a previous mean amount.

20. The method of claim 14 wherein there is statistically a 75%-99% confidence that a subsequent test would dedicate to executing said function an amount of processor resources determinable from said mean amount and said error amount.