Method for determining a trend of a user engagement metric

• US 10,277,694 B2
• Filed: 08/01/2016
• Issued: 04/30/2019
• Est. Priority Date: 04/04/2016
• Status: Active Grant

• Current Assignee: YE Hub Armenia LLC
• Original Assignee: Yandex Europe AG (Yandex N.V.)
• Inventors: Drutsa, Aleksey Valerevich
• Primary Examiner(s): Vostal, Ondrej C
• Application Number:
  US15/224,833
• Publication Number:
  US 20170289284A1
• Time in Patent Office: 1,002 Days
• Field of Search: 709224
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A computer-implemented method and a server with a processor are presented for determining a trend of a user engagement metric with respect to a web service. The method comprises receiving a plurality of user device requests, providing a test version of the web service to a test group and a control version of the web service to a control group, acquiring an amplitude metric and a phase metric for each one of the user devices of at least the test group, determining average group metrics, and determining the trend of the user engagement metric with respect to the web service, the determining the trend being based on analyzing of the control average amplitude metric and the test average amplitude and phase metrics.

Claims

1. A computer-implemented method for determining a trend of a user engagement metric with respect to a web service, the method being executable by a server, the method comprising:

receiving, at the server, a plurality of user device requests relating to a web service during an experimental period, the plurality of user device requests originating from a plurality of user devices;

providing, by the server, a test version of the web service to a test group selected from the plurality of user devices, the test version of the web service being the web service having an experimental treatment applied thereto;

providing, by the server, a control version of the web service to a control group selected from the plurality of user devices, the control version of the web service being a version of the web service without the experimental treatment applied thereto;

acquiring an amplitude metric and a phase metric for each one of the user devices of the control and test groups, the acquiring including, for a given one of the user devices of the control and test groups;

acquiring, by the server, a plurality of indications for the given one, the plurality of indications being based on interactions of the given one with its corresponding web service,

accessing the plurality of indications for the given one,

calculating a periodicity metric based at least in part on a discretization transform performed on the plurality of indications,

calculating the amplitude metric based at least in part on a magnitude of the periodicity metric, the amplitude metric representing a magnitude of change of the user engagement metric with respect to the corresponding web service, and

calculating the phase metric based at least in part on an imaginary part of the periodicity metric, the phase metric representing a direction of change of the user engagement metric with respect to the corresponding web service;

determining, by the server, average group metrics including;

calculating a control average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the control group,

calculating a test average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the test group, and

calculating a test average phase metric by averaging phase metrics calculated for each of the user devices of the test group;

determining the trend of the user engagement metric with respect to the web service, the determining the trend being based on analyzing of the control average amplitude metric and the test average amplitude and phase metrics; and

when a difference between the test average amplitude metric and the control average amplitude metric and the test average phase metric have oppositely signed values, determining that the experimental treatment applied to the web service has caused a decreasing trend in the user engagement metric over the web service without the experimental treatment,

wherein;

the difference between the test average amplitude metric and the control average amplitude metric is negative and the test average phase metric is positive, and

the decreasing trend in the user engagement metric indicates an increase in user engagement.

2. The method of claim 1, wherein the analyzing of the control average amplitude metric and the test average amplitude and phase metrics includes determining a difference between the test average amplitude metric and the control average amplitude metric.

3. The method of claim 1, further comprising determining that the experimental treatment applied to the web service has caused an increasing trend in the user engagement metric over the web service without the experimental treatment when:

a difference between the test average amplitude metric and the control average amplitude metric is positive; and

the test average phase metric is positive.

4. The method of claim 3, wherein:

an increasing value of the user engagement metric indicates a positive effect of the experimental treatment on user engagement; and

the increasing trend in the user engagement metric indicates an increase in user engagement over the experimental period.

5. The method of claim 1, further comprising determining that the experimental treatment applied to the web service has caused an increasing trend in the user engagement metric over the web service without the experimental treatment when:

a difference between test average amplitude metric and the control average amplitude metric is negative; and

the test average phase metric is negative.

6. The method of claim 5, wherein:

a decreasing value of the user engagement metric indicates a positive effect of the experimental treatment on user engagement; and

the increasing trend in the user engagement metric indicates a decrease in user engagement.

7. The method of claim 1, further comprising determining that the experimental treatment applied to the web service has caused a decreasing trend in the user engagement metric over the web service without the experimental treatment when:

a difference between the test average amplitude metric and the control average amplitude metric is positive; and

the test average phase metric is negative.

8. The method of claim 7, wherein:

an increasing value of the user engagement metric indicates a positive effect of the experimental treatment on user engagement; and

the decreasing trend in the user engagement metric indicates a decrease in user engagement.

9. The method of claim 1, wherein:

a difference between the test average amplitude metric and the control average amplitude metric is zero; and

further comprising determining that the experimental treatment applied to the web service has caused;

an increasing trend in the user engagement metric over the unmodified version of the web service when the test average phase metric is positive; and

a decreasing trend in the user engagement metric over the unmodified version of the web service when the test average phase metric is negative.

10. The method of claim 1, wherein the interactions of the given user device include at least a number of sessions.

11. The method of claim 1, wherein the interactions of the given user device include at least one of a dwell time and a time per session.

12. The method of claim 1, wherein the interactions of the given user device include at least a number of clicks.

13. The method of claim 1, wherein the discretization transform is performed using a discrete Fourier transform.

14. The method of claim 1, wherein the discretization transform is performed using at least one of a wavelet transform and a Laplace transform.

15. The method of claim 1, wherein the interactions of the given user device include at least a number of queries.

16. The method of claim 1, wherein the web service is a search engine.

17. The method of claim 1, wherein the web service is a search engine results page.

18. A server comprising a processor, the processor being configured to determine a trend of a user engagement metric with respect to an experimental web service, the processor being configured to render the server to execute:

receiving, at the server, a plurality of user device requests relating to a web service during an experimental period, the plurality of user device requests originating from a plurality of user devices;

providing, by the server, a test version of the web service to a test group selected from the plurality of user devices, the test version of the web service being the web service having an experimental treatment applied thereto;

providing, by the server, a control version of the web service to a control group selected from the plurality of user devices, the control version of the web service being a version of the web service without the experimental treatment applied thereto;

acquiring an amplitude metric and a phase metric for each one of the user devices of the control and test groups, the acquiring including, for a given one of the user devices of the control and test groups;

acquiring, by the server, a plurality of indications for the given one, the plurality of indications being based on interactions of the given one with its corresponding web service,

accessing the plurality of indications for the given one,

calculating a periodicity metric based at least in part on a discretization transform performed on the plurality of indications,

calculating the amplitude metric based at least in part on a magnitude of the periodicity metric, the amplitude metric representing a magnitude of change of the user engagement metric with respect to the corresponding web service, and

calculating the phase metric based at least in part on an imaginary part of the periodicity metric, the phase metric representing a direction of change of the user engagement metric with respect to the corresponding web service;

determining, by the server, average group metrics including;

calculating a control average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the control group,

calculating a test average amplitude metric by averaging amplitude metrics calculated for each of the user devices of the test group, and

calculating a test average phase metric by averaging phase metrics calculated for each of the user devices of the test group; and

determining the trend of the user engagement metric with respect to the web service, the determining the trend being based on analyzing of the control average amplitude metric and the test average amplitude and phase metrics; and

when a difference between the test average amplitude metric and the control average amplitude metric and the test average phase metric have oppositely signed values, determining that the experimental treatment applied to the web service has caused a decreasing trend in the user engagement metric over the web service without the experimental treatment,

wherein;

the difference between the test average amplitude metric and the control average amplitude metric is negative and the test average phase metric is positive, and

the decreasing trend in the user engagement metric indicates an increase in user engagement.