Regression Evaluation Using Behavior Models of Software Applications

• US 20150254161A1
• Filed: 03/05/2014
• Published: 09/10/2015
• Est. Priority Date: 03/05/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Concurix Corporation
• Inventors: BARIL, Bryce B., GOUNARES, Alexander G., KRAJEC, Russell S.
• Granted Patent: US 9,594,665 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3616 using software metrics
• CPC Class Codes: G06F 2201/865 Monitoring of software

Abstract

Comparisons of different versions of an application may be compared using a behavior model of the application. A behavior model may be derived from n-gram analysis of observations of the application in production. The behavior model may include sequences of inputs received by the application or functions performed by the application, where each sequence is an n-gram observed in tracer data. Each n-gram may be coupled with a resource consumption to give a behavior model with performance data. A regression analysis may apply a behavior model derived from a first version of an application to the performance observations of a new version to create an expected performance metric for the new version. A similarly calculated metric from a previous version may be compared to the metric from a new version to determine an improvement or degradation of performance.

Claims

1. A method performed by at least one hardware processor, said method comprising:

receiving a behavior model for an application, said behavior model comprising n-gram frequencies observed from executing a first version of said application, said n-grams representing sequences of operations of said application;

determining a representative sample of said n-grams;

determining a first performance metric for said application by;

for each of said n-grams in said representative sample, determining an n-gram frequency and a first performance measurement, and multiplying said n-gram frequency by said first performance measurement to produce a first n-gram performance metric;

summing said first n-gram performance metrics for all of said n-grams in said representative sample to generate said first performance metric.

2. The method of claim 1 further comprising:

determining a second performance metric for said application by;

for each of said n-grams in said representative sample, determining an n-gram frequency and a second performance measurement, and multiplying said n-gram frequency by said second performance measurement to produce a second n-gram performance metric;

summing said second n-gram performance metrics for all of said n-grams in said representative sample to generate said second performance metric.

3. The method of claim 2, said first performance metric being observations from executing said first version of said application.

4. The method of claim 3, said second performance metric being observations from executing a second version of said application.

5. The method of claim 4, said first version of said application being executed in a production environment.

6. The method of claim 5, said second version of said application being executed in a test environment.

7. The method of claim 1, said n-grams representing input sequences to said application.

8. The method of claim 1, said n-grams representing function sequences to said application.

9. The method of claim 1, said first performance measurement being a resource consumption measurement.

10. The method of claim 9, said performance data comprising a resource consumption measurement of at least one of a group composed of:

processor resources;

memory resources;

storage resources;

network resources;

peripheral resources;

input/output resources;

database resources;

local service resources; and

remote service resources.

11. The method of claim 1, said representative sample comprising at least 50% of the total number of n-gram observations.

12. The method of claim 1, said representative sample comprising at least 75% of the total number of n-gram observations.

13. A system comprising:

a hardware processor;

a version analyzer that;

receives a behavior model for an application, said behavior model comprising n-gram frequencies observed from executing a first version of said application, said n-grams representing sequences of operations of said application;

determines a representative sample of said n-grams;

determines a first performance metric for said application by;

for each of said n-grams in said representative sample, determining an n-gram frequency and a first performance measurement, and multiplying said n-gram frequency by said first performance measurement to produce a first n-gram performance metric;

summing said first n-gram performance metrics for all of said n-grams in said representative sample to generate said first performance metric.

14. The system of claim 13, said version analyzer that further:

determines a second performance metric for said application by;

for each of said n-grams in said representative sample, determining an n-gram frequency and a second performance measurement, and multiplying said n-gram frequency by said second performance measurement to produce a second n-gram performance metric;

summing said second n-gram performance metrics for all of said n-grams in said representative sample to generate said second performance metric.

15. The system of claim 14, said first performance metric being observations from executing said first version of said application.

16. The system of claim 15, said second performance metric being observations from executing a second version of said application.

17. The system of claim 16, said first version of said application being executed in a production environment.

18. The system of claim 17, said second version of said application being executed in a test environment.

19. The system of claim 13, said n-grams representing input sequences to said application.

20. The system of claim 13, said n-grams representing function sequences to said application.