Machine learning comparison tools
First Claim
1. A method comprising:
- receiving, on a computing device, beta test data including usage features of each user associated with testing a beta version of a program and previous test data including usage features of each user associated with testing a previous version of the program;
receiving metric data including metrics associated with the beta version of the program and metrics associated with the previous version of the program; and
using at least one machine learning technique to generate an estimation of the performance and/or reliability of the beta version of the program, wherein the one or more machine learning techniques is selected from among;
a first machine learning technique that compares samples of a first metric from a first group of users having tested the previous version of the program with samples of the first metric from a second group of users having tested the beta version of the program, wherein each user in the first group is matched with a user in the second group having a closest similarity of the usage features;
a second machine learning technique that determines if there is sufficient feature coverage of one or more features tested in the beta version compared with the one or more features tested in the previous version; and
a third machine learning technique that performs a cohort analysis to compare samples from metrics of the previous version from at least two time periods to formulate a distribution that is used to identify performance of the previous version for a user that tested the beta version.
2 Assignments
0 Petitions
Accused Products
Abstract
Machine learning techniques are used to determine the viability of user data measuring the behavior of a new version of the program when compared with user data that measured the behavior of a previous version of the program. The machine learning techniques utilize statistical techniques in a non-conventional manner to train a system to learn from data obtained from the usage of both a new version of the program and a previous version that accounts for the variability in the user population, time variability of the results of the previous version, and feature coverage between the two test results in order to ensure the suitability of the user data in making estimations or predictions about the performance and reliability of the new version.
27 Citations
20 Claims
-
1. A method comprising:
-
receiving, on a computing device, beta test data including usage features of each user associated with testing a beta version of a program and previous test data including usage features of each user associated with testing a previous version of the program; receiving metric data including metrics associated with the beta version of the program and metrics associated with the previous version of the program; and using at least one machine learning technique to generate an estimation of the performance and/or reliability of the beta version of the program, wherein the one or more machine learning techniques is selected from among; a first machine learning technique that compares samples of a first metric from a first group of users having tested the previous version of the program with samples of the first metric from a second group of users having tested the beta version of the program, wherein each user in the first group is matched with a user in the second group having a closest similarity of the usage features; a second machine learning technique that determines if there is sufficient feature coverage of one or more features tested in the beta version compared with the one or more features tested in the previous version; and a third machine learning technique that performs a cohort analysis to compare samples from metrics of the previous version from at least two time periods to formulate a distribution that is used to identify performance of the previous version for a user that tested the beta version. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A system comprising:
-
one or more processors; and
a memory;one or more modules, wherein the one or more modules are configured to be executed by the one or more processors to perform actions comprising; receiving beta test data including usage features of each user associated with testing a beta version of a program and previous test data including usage features of each user associated with testing a previous version of the program; receiving metric data including metrics associated with the beta version of the program and metrics associated with the previous version of the program; and using at least one machine learning technique to generate an estimation of the performance and/or reliability of the beta version of the program, wherein the one or more machine learning techniques is selected from among; a first machine learning technique that compares samples of a first metric from a first group of users having tested the previous version of the program with samples of the first metric from a second group of users having tested the beta version of the program, wherein each user in the first group is matched with a user in the second group having a closest similarity of the usage features; a second machine learning technique that determines if there is sufficient feature coverage of one or more features tested in the beta version compared with the one or more features tested in the previous version; and a third machine learning technique that performs a cohort analysis to compare samples from metrics of the previous version from at least two time periods to formulate a distribution that is used to identify performance of the previous version for a user that tested the beta version. - View Dependent Claims (9, 10, 11, 12, 13, 14)
-
-
15. A device, comprising:
-
a memory coupled to at least one processor; wherein the at least one processor is configured to; receive beta test data including usage features of each user associated with testing a beta version of a program and previous test data including usage features of each user associated with testing a previous version of the program; receive metric data including metrics associated with the beta version of the program and metrics associated with the previous version of the program; and utilize at least one machine learning technique to generate an estimation of the performance and/or reliability of the beta version of the program, wherein the one or more machine learning techniques is selected from among; a first machine learning technique that compares samples of a first metric from a first group of users having tested the previous version of the program with samples of the first metric from a second group of users having tested the beta version of the program, wherein each user in the first group is matched with a user in the second group having a closest similarity of the usage features; a second machine learning technique that determines if there is sufficient feature coverage of one or more features tested in the beta version compared with the one or more features tested in the previous version; and a third machine learning technique that performs a cohort analysis to compare samples from metrics of the previous version from at least two time periods to formulate a distribution that is used to identify performance of the previous version for a user that tested the beta version. - View Dependent Claims (16, 17, 18, 19, 20)
-
Specification