Machine learning comparison tools

• US 10,579,509 B2
• Filed: 05/25/2018
• Issued: 03/03/2020
• Est. Priority Date: 01/21/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Banuelos, Jorge, Cheng, Xi, Sundaresan, Neelakantan, Yao, Cenzhuo, Xu, Kexin
• Primary Examiner(s): Chowdhury, Ziaul A
• Application Number:
  US15/990,032
• Publication Number:
  US 20190227912A1
• Time in Patent Office: 648 Days
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3668 Software testing software testing in telephone exchanges H04M3/242, testing of hardware G06F11/22
• CPC Class Codes: G06F 11/368 for test version control, e.g. updating test cases to a new software version
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Machine learning techniques are used to determine the viability of user data measuring the behavior of a new version of the program when compared with user data that measured the behavior of a previous version of the program. The machine learning techniques utilize statistical techniques in a non-conventional manner to train a system to learn from data obtained from the usage of both a new version of the program and a previous version that accounts for the variability in the user population, time variability of the results of the previous version, and feature coverage between the two test results in order to ensure the suitability of the user data in making estimations or predictions about the performance and reliability of the new version.

Claims

1. A method comprising:

receiving, on a computing device, beta test data including usage features of each user associated with testing a beta version of a program and previous test data including usage features of each user associated with testing a previous version of the program;

receiving metric data including metrics associated with the beta version of the program and metrics associated with the previous version of the program; and

using at least one machine learning technique to generate an estimation of the performance and/or reliability of the beta version of the program, wherein the one or more machine learning techniques is selected from among;

a first machine learning technique that compares samples of a first metric from a first group of users having tested the previous version of the program with samples of the first metric from a second group of users having tested the beta version of the program, wherein each user in the first group is matched with a user in the second group having a closest similarity of the usage features;

a second machine learning technique that determines if there is sufficient feature coverage of one or more features tested in the beta version compared with the one or more features tested in the previous version; and

a third machine learning technique that performs a cohort analysis to compare samples from metrics of the previous version from at least two time periods to formulate a distribution that is used to identify performance of the previous version for a user that tested the beta version.

2. The method of claim 1, wherein the first machine learning technique performs actions comprising:

formulating matched pairs of users, a matched pair including a user having tested a previous version of a program and a user having tested a beta version of the program, the matched pair having similar usage features;

averaging values of a first metric from the matched pairs of users having tested the previous version and averaging values of the first metric from the matched pairs of users having tested the beta version;

comparing the average value of the first metric from the previous version with the average value of the first metric from the beta version to determine a significance of the difference between both average values; and

generating a reliability of the first metric based on a significance of the difference between both average values.

3. The method of claim 2, wherein formulating the matched pairs of users, further comprises:

computing similarity between each usage feature associated with a user in a matched pair having tested the beta version of the program with each usage feature associated with a user in the matched pair having tested a previous version of the program; and

forming the matched pair with a user in the matched pair having tested the beta version with a user in the matched pair having tested the previous version having a smallest distance between each value associated with each usage feature of each user of the matched pair.

4. The method of claim 2, wherein comparing the average value of the first metric from the previous version with the average value of the first metric from the beta version is computed using a student t test.

5. The method of claim 4, further comprising:

testing a test statistic from the student t test for the first metric to determine that the beta version is less reliable or more reliable than the release version.

6. The method of claim 5, further comprising:

forming a confidence interval for each metric from the average values and variances for each metric.

7. The method of claim 6, further comprising:

using the confidence interval for each metric to output a confidence representing a quality of the matched pair.

8. A system comprising:

one or more processors; and

a memory;

one or more modules, wherein the one or more modules are configured to be executed by the one or more processors to perform actions comprising;

receiving beta test data including usage features of each user associated with testing a beta version of a program and previous test data including usage features of each user associated with testing a previous version of the program;

receiving metric data including metrics associated with the beta version of the program and metrics associated with the previous version of the program; and

using at least one machine learning technique to generate an estimation of the performance and/or reliability of the beta version of the program, wherein the one or more machine learning techniques is selected from among;

a first machine learning technique that compares samples of a first metric from a first group of users having tested the previous version of the program with samples of the first metric from a second group of users having tested the beta version of the program, wherein each user in the first group is matched with a user in the second group having a closest similarity of the usage features;

a second machine learning technique that determines if there is sufficient feature coverage of one or more features tested in the beta version compared with the one or more features tested in the previous version; and

a third machine learning technique that performs a cohort analysis to compare samples from metrics of the previous version from at least two time periods to formulate a distribution that is used to identify performance of the previous version for a user that tested the beta version.

9. The system of claim 8, wherein the usage features include a plurality of features associated with a user'"'"'s testing environment and/or usage of the program.

10. The system of claim 8, wherein the second machine learning technique compares a percentage of users testing one or more features of the previous version and a percentage of users testing the one or more features of the beta version to determine whether the beta test had insufficient coverage of the one or more features.

11. The system of claim 10, wherein a proportion of a percentage test is used to perform the comparison.

12. The system of claim 8, wherein the second machine learning technique determines whether frequency of usage of a feature by the users in the beta version is sufficient.

13. The system of claim 8, wherein the second machine learning technique compares an average frequency of usage of a first feature by the users of the beta version with an average frequency of usage of the first feature by the users of the previous version to determine sufficiency of data for the first feature.

14. The system of claim 13, wherein the second machine learning technique performs a sample mean test to perform the comparison of the average frequency of usage of a feature by the users of the beta version with an average frequency of usage of the feature by the users of the previous version.

15. A device, comprising:

a memory coupled to at least one processor;

wherein the at least one processor is configured to;

receive beta test data including usage features of each user associated with testing a beta version of a program and previous test data including usage features of each user associated with testing a previous version of the program;

receive metric data including metrics associated with the beta version of the program and metrics associated with the previous version of the program; and

utilize at least one machine learning technique to generate an estimation of the performance and/or reliability of the beta version of the program, wherein the one or more machine learning techniques is selected from among;

a first machine learning technique that compares samples of a first metric from a first group of users having tested the previous version of the program with samples of the first metric from a second group of users having tested the beta version of the program, wherein each user in the first group is matched with a user in the second group having a closest similarity of the usage features;

a second machine learning technique that determines if there is sufficient feature coverage of one or more features tested in the beta version compared with the one or more features tested in the previous version; and

a third machine learning technique that performs a cohort analysis to compare samples from metrics of the previous version from at least two time periods to formulate a distribution that is used to identify performance of the previous version for a user that tested the beta version.

16. The device of claim 15, wherein the third machine learning technique performs actions that:

compare differences between metrics from a first time period with metrics from a second time period, wherein the metrics are generated by users who tested the previous version, the first time period and the second time period differ;

formulate a test distribution from the compared differences, the test distribution including a first critical value and a second critical value;

compare, for a single user, metrics from testing the beta version with metrics from testing the previous version, the comparison resulting in a first test statistic; and

estimate the performance of the beta version for the single user based on where the first test statistic fits within the first critical value and the second critical value.

17. The device of claim 16, wherein the single user'"'"'s performance of the beta version exceeds the single user'"'"'s performance of the previous version when the first test statistic exceeds a 95% confidence level.

18. The device of claim 17, wherein the single user'"'"'s performance of the beta version is slower than the single use'"'"'s performance of the previous version when the first test statistic is less than a 95% confidence level.

19. The device of claim 16, wherein the comparison of differences between metrics from a first time period with metrics from a second time period is performed utilizing a Mann Whitney U test.

20. The device of claim 16, wherein the comparison, for a single user, of the metrics from testing the beta version with metrics from testing the previous version is performed utilizing a Mann Whitney U test.