Statistical analysis of sampled profile data in the identification of significant software test performance regressions
First Claim
1. In a system for analyzing computer test results, said test results comprising a plurality of sampled processor events, a method comprising:
- computing at least one amount of processor resources that was used to execute a function;
computing at least one error amount associated with said amount of processor resources;
comparing said amount of processor resources and said error amount with amounts from a previous test;
finding a significant regression if said amount of processor resources differs from an amount of processor resources from the previous test, after accounting for said error amount and an error amount from the previous test.
2 Assignments
0 Petitions
Accused Products
Abstract
Sampled profile data provides information about processor activity during a test. Processor activity can be analyzed to determine an amount of processor resources used to execute the various functions, modules, and processes associated with a tested software activity. Statistical methods can be applied to the resource data from multiple test runs to determine whether a significant regression has occurred between a baseline test pass and a daily test pass. By collecting data at the function, module and process levels, significant regressions may be uncovered at any of the levels. Regressions may also be ranked according to their importance, which allows for identification and notification of development teams responsible for significant regressions.
-
Citations
20 Claims
-
1. In a system for analyzing computer test results, said test results comprising a plurality of sampled processor events, a method comprising:
-
computing at least one amount of processor resources that was used to execute a function;
computing at least one error amount associated with said amount of processor resources;
comparing said amount of processor resources and said error amount with amounts from a previous test;
finding a significant regression if said amount of processor resources differs from an amount of processor resources from the previous test, after accounting for said error amount and an error amount from the previous test. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A system for testing computer software, comprising:
-
a test harness for causing a plurality of test runs, wherein tested software executes on a computer processor in each test run;
a profiler for sampling processor activity during execution of said tested software, thereby generating sample data;
software for calculating from said sample data an amount of processor resources in each test run that was dedicated to executing at least one function associated with the tested software;
software for calculating from said amount of processor resources in each test run a mean amount of processor resources dedicated to executing the at least one function and an error amount associated with said mean amount;
software for comparing said mean amount and said error amount to a previous mean amount and a previous error amount from a previous test to determine if there is a significant regression in the amount of processor resources dedicated to executing the at least one function. - View Dependent Claims (9, 10, 11, 12, 13)
-
-
14. A method to be carried out by a computer for ranking software test regressions, comprising:
-
calculating a mean amount of processor resources that are dedicated to executing a function;
calculating an error associated with said mean amount;
calculating a difference between said mean amount and a previous mean amount, wherein said error amount to and a previous error amount are accounted for;
ranking a regression based on said difference. - View Dependent Claims (15, 16, 17, 18, 19, 20)
-
Specification