Statistical analysis of sampled profile data in the identification of significant software test performance regressions
First Claim
1. In a system for analyzing computer test results, said test results comprising a plurality of sampled processor events, a method comprising:
- computing an amount of processor resources used to execute each memory address in each function, a memory address comprising a subset of a function;
computing an amount of processor resources used to execute each function in each module based on the amount of processor resources used to execute each memory address of each function, a function comprising a subset of a module;
computing an amount of processor resources used to execute each module in said process based on the amount of processor resources used to execute each function of each module, a module comprising a subset of a process;
computing an amount of processor resources used to execute the process based on the amount of processor resources used to execute each module of the process;
for each amount of processor resources used to execute each process computed, each amount of processor resources used to execute each module computed, each amount of processor resources used to execute each function computed;
computing an error amount associated with said amount of processor resources;
computing said amount of processor resources and said error amount with amounts from a previous test; and
finding a significant regression if said amount of processor resources differs from an amount of processor resources from the previous test, after accounting for said error amount and an error amount from the previous test.
2 Assignments
0 Petitions
Accused Products
Abstract
Sampled profile data provides information about processor activity during a test. Processor activity can be analyzed to determine an amount of processor resources used to execute the various functions, modules, and processes associated with a tested software activity. Statistical methods can be applied to the resource data from multiple test runs to determine whether a significant regression has occurred between a baseline test pass and a daily test pass. By collecting data at the function, module and process levels, significant regressions may be uncovered at any of the levels. Regressions may also be ranked according to their importance, which allows for identification and notification of development teams responsible for significant regressions.
-
Citations
19 Claims
-
1. In a system for analyzing computer test results, said test results comprising a plurality of sampled processor events, a method comprising:
-
computing an amount of processor resources used to execute each memory address in each function, a memory address comprising a subset of a function; computing an amount of processor resources used to execute each function in each module based on the amount of processor resources used to execute each memory address of each function, a function comprising a subset of a module; computing an amount of processor resources used to execute each module in said process based on the amount of processor resources used to execute each function of each module, a module comprising a subset of a process; computing an amount of processor resources used to execute the process based on the amount of processor resources used to execute each module of the process; for each amount of processor resources used to execute each process computed, each amount of processor resources used to execute each module computed, each amount of processor resources used to execute each function computed; computing an error amount associated with said amount of processor resources; computing said amount of processor resources and said error amount with amounts from a previous test; and finding a significant regression if said amount of processor resources differs from an amount of processor resources from the previous test, after accounting for said error amount and an error amount from the previous test. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A system for testing computer software, comprising:
-
a processor; a test harness for causing a plurality of test runs, wherein a test run comprises executing tested software on the processor; a profiler for sampling processor activity during executing of said tested software, thereby generating sample data; software for calculating from said sample data an amount of processor resources in each test run used to execute each memory address in each function, a memory address comprising a subset of a function; software for calculating from said sample data an amount of processor resources in each test run used to execute each function in each module based on the amount of processor resources used to execute each memory address of each function, a function comprising a subset of a module; software for calculating from said sample data an amount of processor resources in each test run used to execute each module in said process based on the amount of processor resources used to execute each function of each module, a module comprising a subset of a process; software for calculating from said sample data an amount of processor resources in each test run used to execute the process based on the amount of processor resources used to execute each module of the process; software for calculating from said each amount of processor resources in each test run a mean amount of processor resources dedicated to executing the corresponding function, module or process and an error amount associated with said mean amount; software for comparing said mean amount and said error amount to a previous mean amount and a previous error amount from a previous test to determine if there is a significant regression in the amount of processor resources dedicated to executing the function module or process. - View Dependent Claims (9, 10, 11, 12, 13)
-
-
14. A method to be carried out by a computer for ranking software test regressions, comprising:
-
calculating a mean amount of processor resources that are dedicated to executing a function, wherein said processor resources are measured in Central Processing Unit (CPU) clock cycles; calculating an error associated with said mean amount; calculating a difference between said mean amount and a previous mean amount, wherein said error amount to and a previous error amount are accounted for; ranking a regression based on said difference; and filtering the regression if the error amount and the previous error amount are greater than a simple difference between said mean amount and a previous mean amount. - View Dependent Claims (15, 16, 17, 18, 19)
-
Specification