Method for determining a trend of a user engagement metric
First Claim
1. A computer-implemented method for determining a trend of a user engagement metric with respect to a web service, the method being executable by a server, the method comprising:
- receiving, at the server, a plurality of user device requests relating to a web service during an experimental period, the plurality of user device requests originating from a plurality of user devices;
providing, by the server, a test version of the web service to a test group selected from the plurality of user devices, the test version of the web service being the web service having an experimental treatment applied thereto;
providing, by the server, a control version of the web service to a control group selected from the plurality of user devices, the control version of the web service being a version of the web service without the experimental treatment applied thereto;
acquiring an amplitude metric and a phase metric for each one of the user devices of the control and test groups, the acquiring including, for a given one of the user devices of the control and test groups;
acquiring, by the server, a plurality of indications for the given one, the plurality of indications being based on interactions of the given one with its corresponding web service,accessing the plurality of indications for the given one,calculating a periodicity metric based at least in part on a discretization transform performed on the plurality of indications,calculating the amplitude metric based at least in part on a magnitude of the periodicity metric, the amplitude metric representing a magnitude of change of the user engagement metric with respect to the corresponding web service, andcalculating the phase metric based at least in part on an imaginary part of the periodicity metric, the phase metric representing a direction of change of the user engagement metric with respect to the corresponding web service;
determining, by the server, average group metrics including;
calculating a control average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the control group,calculating a test average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the test group, andcalculating a test average phase metric by averaging phase metrics calculated for each of the user devices of the test group;
determining the trend of the user engagement metric with respect to the web service, the determining the trend being based on analyzing of the control average amplitude metric and the test average amplitude and phase metrics; and
when a difference between the test average amplitude metric and the control average amplitude metric and the test average phase metric have oppositely signed values, determining that the experimental treatment applied to the web service has caused a decreasing trend in the user engagement metric over the web service without the experimental treatment,wherein;
the difference between the test average amplitude metric and the control average amplitude metric is negative and the test average phase metric is positive, andthe decreasing trend in the user engagement metric indicates an increase in user engagement.
4 Assignments
0 Petitions
Accused Products
Abstract
A computer-implemented method and a server with a processor are presented for determining a trend of a user engagement metric with respect to a web service. The method comprises receiving a plurality of user device requests, providing a test version of the web service to a test group and a control version of the web service to a control group, acquiring an amplitude metric and a phase metric for each one of the user devices of at least the test group, determining average group metrics, and determining the trend of the user engagement metric with respect to the web service, the determining the trend being based on analyzing of the control average amplitude metric and the test average amplitude and phase metrics.
-
Citations
18 Claims
-
1. A computer-implemented method for determining a trend of a user engagement metric with respect to a web service, the method being executable by a server, the method comprising:
-
receiving, at the server, a plurality of user device requests relating to a web service during an experimental period, the plurality of user device requests originating from a plurality of user devices; providing, by the server, a test version of the web service to a test group selected from the plurality of user devices, the test version of the web service being the web service having an experimental treatment applied thereto; providing, by the server, a control version of the web service to a control group selected from the plurality of user devices, the control version of the web service being a version of the web service without the experimental treatment applied thereto; acquiring an amplitude metric and a phase metric for each one of the user devices of the control and test groups, the acquiring including, for a given one of the user devices of the control and test groups; acquiring, by the server, a plurality of indications for the given one, the plurality of indications being based on interactions of the given one with its corresponding web service, accessing the plurality of indications for the given one, calculating a periodicity metric based at least in part on a discretization transform performed on the plurality of indications, calculating the amplitude metric based at least in part on a magnitude of the periodicity metric, the amplitude metric representing a magnitude of change of the user engagement metric with respect to the corresponding web service, and calculating the phase metric based at least in part on an imaginary part of the periodicity metric, the phase metric representing a direction of change of the user engagement metric with respect to the corresponding web service; determining, by the server, average group metrics including; calculating a control average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the control group, calculating a test average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the test group, and calculating a test average phase metric by averaging phase metrics calculated for each of the user devices of the test group; determining the trend of the user engagement metric with respect to the web service, the determining the trend being based on analyzing of the control average amplitude metric and the test average amplitude and phase metrics; and when a difference between the test average amplitude metric and the control average amplitude metric and the test average phase metric have oppositely signed values, determining that the experimental treatment applied to the web service has caused a decreasing trend in the user engagement metric over the web service without the experimental treatment, wherein; the difference between the test average amplitude metric and the control average amplitude metric is negative and the test average phase metric is positive, and the decreasing trend in the user engagement metric indicates an increase in user engagement. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
-
-
18. A server comprising a processor, the processor being configured to determine a trend of a user engagement metric with respect to an experimental web service, the processor being configured to render the server to execute:
-
receiving, at the server, a plurality of user device requests relating to a web service during an experimental period, the plurality of user device requests originating from a plurality of user devices; providing, by the server, a test version of the web service to a test group selected from the plurality of user devices, the test version of the web service being the web service having an experimental treatment applied thereto; providing, by the server, a control version of the web service to a control group selected from the plurality of user devices, the control version of the web service being a version of the web service without the experimental treatment applied thereto; acquiring an amplitude metric and a phase metric for each one of the user devices of the control and test groups, the acquiring including, for a given one of the user devices of the control and test groups; acquiring, by the server, a plurality of indications for the given one, the plurality of indications being based on interactions of the given one with its corresponding web service, accessing the plurality of indications for the given one, calculating a periodicity metric based at least in part on a discretization transform performed on the plurality of indications, calculating the amplitude metric based at least in part on a magnitude of the periodicity metric, the amplitude metric representing a magnitude of change of the user engagement metric with respect to the corresponding web service, and calculating the phase metric based at least in part on an imaginary part of the periodicity metric, the phase metric representing a direction of change of the user engagement metric with respect to the corresponding web service; determining, by the server, average group metrics including; calculating a control average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the control group, calculating a test average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the test group, and calculating a test average phase metric by averaging phase metrics calculated for each of the user devices of the test group; and determining the trend of the user engagement metric with respect to the web service, the determining the trend being based on analyzing of the control average amplitude metric and the test average amplitude and phase metrics; and when a difference between the test average amplitude metric and the control average amplitude metric and the test average phase metric have oppositely signed values, determining that the experimental treatment applied to the web service has caused a decreasing trend in the user engagement metric over the web service without the experimental treatment, wherein; the difference between the test average amplitude metric and the control average amplitude metric is negative and the test average phase metric is positive, and the decreasing trend in the user engagement metric indicates an increase in user engagement.
-
Specification