Automated regression testing for software applications

• US 9,355,016 B2
• Filed: 03/05/2014
• Issued: 05/31/2016
• Est. Priority Date: 03/05/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Baril, Bryce B., Gounares, Alexander G., Krajec, Russell S.
• Primary Examiner(s): LE, DIEU MINH T
• Application Number:
  US14/198,271
• Publication Number:
  US 20150254165A1
• Time in Patent Office: 818 Days
• Field of Search: 714/38.1, 714/37, 714/4.1, 717/124, 717/125, 717/126
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3676 for coverage analysis
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

Regression testing of an application may gather performance tests for multiple functions within an application and determine when performance changes from one version of the application to another. The analysis may be further broken down by input sequences that may be processed by various functions. A detailed regression analysis may be presented as a heat map or other visualizations. A regression testing system may be launched during a build process by automatically launching a set of performance tests against an application. In many cases, the application may be executed in a system with a known or consistent performance capabilities. The application may be executed and tested in a new version and at least one prior version on the same hardware and software execution environment, so that results may be normalized from one execution run to another. A regression testing system may be deployed as a paid-for service that may integrate into a source code repository.

Claims

1. A method performed on at least one computer processor on a first hardware device, said method comprising:

receiving a first version of a software module, said software module comprising a plurality of functions;

configuring a first execution environment;

causing said first version of said software module to be executed on said first execution environment with a tracer;

receiving first tracer data from said tracer, said first tracer data being collected while said first version of said software module is executed on said first execution environment, said first tracer data comprising performance data for at least some of said plurality of functions;

receiving second tracer data from said tracer, said second tracer data being collected while a second version of said software module is executed on a second execution environment, said second tracer data comprising performance data for at least some of said plurality of functions;

comparing said first tracer data to said second tracer data to populate a regression database comprising performance differences for said at least some of said plurality of functions.

2. The method of claim 1, said first execution environment being a virtual machine.

3. The method of claim 2, said virtual machine operating on said first hardware device.

4. The method of claim 1, said first execution environment being on a second device.

5. The method of claim 1:

said first tracer data further comprising a first performance dataset for a first input stream for a first function and a second input stream for said first function;

said second tracer data further comprising a second performance dataset for said first input stream for said first function and said second input stream for said first function;

said regression database further comprising said performance differences for said first input stream for said first function and said second input stream for said first function.

6. The method of claim 5 further comprising:

displaying a visualization of at least a portion of said regression database.

7. The method of claim 6, said visualization showing a first regression statistic for said first input stream of said first function and a second regression statistic for said second input stream of said first function.

8. The method of claim 6, said visualization being a heat map.

9. The method of claim 1 further comprising causing said second version of said software module to be executed on said first execution environment with said tracer and generating said second tracer data.

10. The method of claim 9 further comprising:

causing a third version of said software module to be executed on said first execution environment with said tracer and generating third tracer data; and

comparing said second tracer data to said third tracer data to further populate a regression database comprising performance differences for said at least some of said plurality of functions.

11. The method of claim 1, said performance data comprising usage data of at least one of a group composed of:

processor resources;

memory resources;

storage resources;

network resources;

peripheral resources;

input/output resources;

database resources;

local service resources; and

remote service resources.

12. A system comprising:

a testing component that;

receives a first version of a software module, said software module comprising a plurality of functions;

configures a first execution environment;

causes said first version of said software module to be executed on said first execution environment with a tracer;

an analysis component executing on a computer processor, said analysis component that;

receives first tracer data from said tracer, said first tracer data being collected while said first version of said software module is executed on said first execution environment, said first tracer data comprising performance data for at least some of said plurality of functions;

receives second tracer data from said tracer, said second tracer data being collected while a second version of said software module is executed on a second execution environment, said second tracer data comprising performance data for at least some of said plurality of functions;

compares said first tracer data to said second tracer data to populate a regression database comprising performance differences for said at least some of said plurality of functions.

13. The system of claim 12, said first execution environment being a virtual machine.

14. The system of claim 13, said virtual machine operating on said first hardware device.

15. The system of claim 12, said first execution environment being on a second device.

16. The system of claim 12:

said first tracer data further comprising a first performance dataset for a first input stream for a first function and a second input stream for said first function;

said second tracer data further comprising a second performance dataset for said first input stream for said first function and said second input stream for said first function;

said regression database further comprising said performance differences for said first input stream for said first function and said second input stream for said first function.

17. The system of claim 16 said analysis component that further:

displays a visualization of at least a portion of said regression database.

18. The system of claim 17, said visualization showing a first regression statistic for said first input stream of said first function and a second regression statistic for said second input stream of said first function.

19. The system of claim 17, said visualization being a heat map.

20. The system of claim 12 said testing component that further:

causes said second version of said software module to be executed on said first execution environment with said tracer and generates said second tracer data.

21. The system of claim 20 said testing component that further:

causing a third version of said software module to be executed on said first execution environment with said tracer and generating third tracer data; and

said analysis component that further;

compares said second tracer data to said third tracer data to further populate a regression database comprising performance differences for said at least some of said plurality of functions.

22. The system of claim 12, said performance data comprising usage data of at least one of a group composed of:

processor resources;

memory resources;

storage resources;

network resources;

peripheral resources;

input/output resources;

database resources;

local service resources; and

remote service resources.