Regression evaluation using behavior models of software applications

• US 9,594,665 B2
• Filed: 03/05/2014
• Issued: 03/14/2017
• Est. Priority Date: 03/05/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Baril, Bryce B., Gounares, Alexander G., Krajec, Russell S.
• Primary Examiner(s): MILLER, VIVA L
• Application Number:
  US14/198,284
• Publication Number:
  US 20150254161A1
• Time in Patent Office: 1,105 Days
• Field of Search: 717/124, 717/127
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3616 using software metrics
• CPC Class Codes: G06F 2201/865 Monitoring of software

Abstract

Comparisons of different versions of an application may be compared using a behavior model of the application. A behavior model may be derived from n-gram analysis of observations of the application in production. The behavior model may include sequences of inputs received by the application or functions performed by the application, where each sequence is an n-gram observed in tracer data. Each n-gram may be coupled with a resource consumption to give a behavior model with performance data. A regression analysis may apply a behavior model derived from a first version of an application to the performance observations of a new version to create an expected performance metric for the new version. A similarly calculated metric from a previous version may be compared to the metric from a new version to determine an improvement or degradation of performance.

Claims

1. A method performed by at least one hardware processor, said method comprising:

receiving a first behavior model representing a frequency of observation of each of a first plurality of n-grams during execution of a first version of an application in a production environment, each n-gram in the first plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the first version of the application in the production environment, or (ii) two or more input sequences input to the first version of the application during execution of the first version of the application in the production environment;

generating a first performance metric for the first version of the application based on the first behavior model, including (i) determining a first representative sample of n-grams from the first plurality of n-grams, (ii) determining a first corresponding performance metric for each n-gram in the first representative sample of n-grams, and (iii) summing each first corresponding n-gram performance metric for all of the n-grams in the first representative sample of n-grams to obtain the first performance metric;

receiving a second behavior model representing a frequency of observation of each of a second plurality of n-grams during execution of a second version of the application in a testing environment, each n-gram in the second plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the second version of the application in the testing environment, or (ii) two or more input sequences input to the second version of the application during execution of the second version of the application in the testing environment;

generating a second performance metric for the second version of the application based on the second behavior model, including (i) determining a second representative sample of n-grams from the second plurality of n-grams, (ii) determining a second corresponding performance metric for each n-gram in the second representative sample of n-grams, and (iii) summing each second corresponding n-gram performance metric for all of the n-grams in the second representative sample of n-grams to obtain the second performance metric; and

generating a visualization comparing performance of the first version of the application in the production environment with the second version of the application in the testing environment, the visualization including;

one or more histograms visualizing the first plurality of n-grams and the second plurality of n-grams; and

one or more charts or graphs that present performance metrics for successive versions of the application, the one or more charts or graphs presenting at least the first performance metric and the second performance metric.

2. The method of claim 1, each representative sample comprising at least 50% of a total number of n-gram observations.

3. The method of claim 1, each representative sample comprising at least 75% of a total number of n-gram observations.

9. The method of claim 1, wherein each behavioral model applies different weights to performance observations based on frequency of observation of one or more operations.

10. The method of claim 1, wherein the computer system comprises the at least one hardware processor.

13. The method of claim 1, wherein determining a performance metric for an n-gram comprises:

determining an n-gram frequency for the n-gram;

determining an n-gram performance measurement associated with observation of the n-gram during execution of the application; and

multiplying the n-gram frequency by the n-gram performance measurement, to produce the performance metric for the n-gram.

14. The method of claim 13, wherein determining the n-gram performance measurement is based on at least one of processor utilization or memory utilization in connection with observation of the n-gram.

4. A system comprising:

one or more hardware processors; and

one or more computer-readable media having stored thereon computer-executable instructions that are executable by the one or more hardware processors to implement a version analyzer that is configured to perform at least the following;

receive a first behavior model representing a frequency of observation of each of a first plurality of n-grams during execution of a first version of an application in a production environment, each n-gram in the first plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the first version of the application in the production environment, or (ii) two or more input sequences input to the first version of the application during execution of the first version of the application in the production environment;

generate a first performance metric for the first version of the application based on the first behavior model, including (i) determining a first representative sample of n-grams from the first plurality of n-grams, determining a first corresponding performance metric for each n-gram in the first representative sample of n-grams, and (ii) summing each first corresponding n-gram performance metric for all of the n-grams in the first representative sample of n-grams to obtain the first performance metric;

receive a second behavior model representing a frequency of observation of each of a second plurality of n-grams during execution of a second version of the application in a testing environment, each n-gram in the second plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the second version of the application in the testing environment, or (ii) two or more input sequences input to the second version of the application during execution of the second version of the application in the testing environment;

generate a second performance metric for the second version of the application based on the second behavior model, including (i) determining a second representative sample of n-grams from the second plurality of n-grams, (ii) determining a second corresponding performance metric for each n-gram in the second representative sample of n-grams, and (iii) summing each second corresponding n-gram performance metric for all of the n-grams in the second representative sample of n-grams to obtain the second performance metric; and

generate a visualization comparing performance of the first version of the application in the production environment with the second version of the application in the testing environment, the visualization including;

one or more histograms visualizing the first plurality of n-grams and the second plurality of n-grams; and

one or more charts or graphs that present performance metrics for successive versions of the application, the one or more charts or graphs presenting at least the first performance metric and the second performance metric.

5. The system of claim 4, each representative sample comprising at least 50% of a total number of n-gram observations.

11. The system of claim 4, wherein the system comprises the computing system.

12. The system of claim 4, wherein each behavioral model applies different weights to performance observations based on frequency of observation of one or more operations.

15. The system of claim 4, wherein determining a performance metric for an n-gram comprises:

determining an n-gram frequency for the n-gram;

determining an n-gram performance measurement associated with observation of the n-gram during execution of the application; and

multiplying the n-gram frequency by the n-gram performance measurement, to produce the performance metric for the n-gram.

16. The system of claim 15, wherein determining the n-gram performance measurement is based on at least one of processor utilization or memory utilization in connection with observation of the n-gram.

6. A computer program product comprising one or more memories having stored thereon computer-executable instructions that are executable one or more processors to perform at least the following:

receive a first behavior model representing a frequency of observation of each of a first plurality of n-grams during execution of a first version of an application in a production environment, each n-gram in the first plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the first version of the application in the production environment, or (ii) two or more input sequences input to the first version of the application during execution of the first version of the application in the production environment;

generate a first performance metric for the first version of the application based on the first behavior model, including (i) determining a first representative sample of n-grams from the first plurality of n-grams, determining a first corresponding performance metric for each n-gram in the first representative sample of n-grams, and (ii) summing each first corresponding n-gram performance metric for all of the n-grams in the first representative sample of n-grams to obtain the first performance metric;

receive a second behavior model representing a frequency of observation of each of a second plurality of n-grams during execution of a second version of the application in a testing environment, each n-gram in the second plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the second version of the application in the testing environment, or (ii) two or more input sequences input to the second version of the application during execution of the second version of the application in the testing environment;

generate a second performance metric for the second version of the application based on the second behavior model, including (i) determining a second representative sample of n-grams from the second plurality of n-grams, (ii) determining a second corresponding performance metric for each n-gram in the second representative sample of n-grams, and (iii) summing each second corresponding n-gram performance metric for all of the n-grams in the second representative sample of n-grams to obtain the second performance metric; and

generate a visualization comparing performance of the first version of the application in the production environment with the second version of the application in the testing environment, the visualization including;

one or more histograms visualizing the first plurality of n-grams and the second plurality of n-grams; and

one or more charts or graphs that present performance metrics for successive versions of the application, the one or more charts or graphs presenting at least the first performance metric and the second performance metric.

7. The computer program product of claim 6, wherein each behavioral model applies different weights to performance observations based on frequency of observation of one or more operations.

8. The computer program product of claim 6, wherein the computer system comprises the one or more processors.

17. The computer program product of claim 6, wherein determining a performance metric for an n-gram comprises:

determining an n-gram frequency for the n-gram;

determining an n-gram performance measurement associated with observation of the n-gram during execution of the application; and

multiplying the n-gram frequency by the n-gram performance measurement, to produce the performance metric for the n-gram.

18. The computer program product of claim 17, wherein determining the n-gram performance measurement is based on at least one of processor utilization or memory utilization in connection with observation of the n-gram.