Flexible units for experimentation

• US 10,462,239 B1
• Filed: 07/29/2016
• Issued: 10/29/2019
• Est. Priority Date: 07/29/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Xu, Ya, Nieh, Kylan Matthew, Duan, Weitao, Liu, Bo
• Primary Examiner(s): Hoang, Hieu T
• Application Number:
  US15/223,865
• Time in Patent Office: 1,187 Days
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06Q 30/0243 Comparative campaigns
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: H04L 51/42 Mailbox-related aspects, e.g. synchronisation of mailboxes
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A machine may be configured to perform A/B testing on customized experiment units. For example, the machine receives a specification of an experiment unit that identifies a type of subject of an experiment for execution on a social networking service (SNS), and a value of the experiment unit. The machine generates, for the value of the experiment unit, tracking data that tracks user interactions by one or more users of the SNS, via one or more browsers, with content provided during an execution of the experiment. The machine generates, for the value of the experiment unit, metric data that measures an attribute associated with the experimental unit. The machine generates an experiment report based on the tracking data and the metric data. The machine causes a presentation of the experiment report in a user interface of a client device.

Claims

1. A method comprising:

receiving, from a client device, a selection of an identifier of an experiment unit from a plurality of identifiers of a plurality of experiment units displayed in a user interface for specification of an experiment for execution on a social networking service (SNS), and a value of the experiment unit, the plurality of experiment units including an online company page, an email message, and a job page, and the value of the experiment unit including an identifier of a particular online company page hosted by the SNS;

generating, for the value of the experiment unit, logging data that logs user interactions by one or more users of the SNS, via one or more browsers, with content provided during an execution of the experiment, the generating of the logging data being performed by one or more hardware processors;

generating, for the value of the experiment unit, metric data that measures an attribute associated with the experimental unit, the attribute associated with the experimental unit and measured by the metric data, including a number of unique visitors to the particular online company page;

generating an experiment report based on the logging data and the metric data; and

causing a presentation of the experiment report in a user interface of the client device.

2. The method of claim 1, wherein the receiving of the specification of the experiment unit includes:

receiving a reference to an identifier of the experiment unit and the value of the experiment unit from the client device via a command-line user interface.

3. The method of claim 1, wherein the generating, for the value of the experiment unit, of the logging data includes:

generating the logging data for a plurality of values of the experiment unit.

4. The method of claim 1, wherein the generating of the metric data includes:

parsing the logging data based on at least one of a type of variant of the experiment, the experiment unit, the value of the experiment unit, or a timestamp, the parsing resulting in parsed logging data.

5. The method of claim 1, wherein the generating of the experiment report based on the logging data and the metric data includes:

determining, based on the metric data pertaining to a period of time, a value of the attribute that is associated with the experimental unit and that is measured by the metric data; and

performing a join operation on the logging data pertaining to the period of time and the metric data pertaining to the period of time based on the value of the experiment unit, the performing of the join operation resulting in joint logging and metric data,

wherein the experiment report includes the joint logging and metric data and the value of the attribute that is associated with the experimental unit and that is measured by the metric data.

6. A system comprising:

one or more hardware processors; and

a machine-readable medium for storing instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising;

receiving, from a client device, a selection of an identifier of an experiment unit from a plurality of identifiers of a plurality of experiment units displayed in a user interface for specification of an experiment for execution on a social networking service (SNS), and a value of the experiment unit, the plurality of experiment units including an online company page, an email message, and a job page, and the value of the experiment unit including an identifier of a particular online company page hosted by the SNS;

generating, for the value of the experiment unit, logging data that logs user interactions by one or more users of the SNS, via one or more browsers, with content provided during an execution of the experiment;

generating, for the value of the experiment unit, metric data that measures an attribute associated with the experimental unit, the attribute associated with the experimental unit and measured by the metric data, including a number of unique visitors to the particular online, company page;

generating an experiment report based on the logging data and the metric data; and

causing a presentation of the experiment report in a user interface of the client device.

7. The system of claim 6, wherein the receiving of the specification of the experiment unit includes:

receiving a reference to an identifier of the experiment unit from the client device.

8. The system of claim 6, wherein the generating, for the value of the experiment unit, of the logging data includes:

generating the logging data for a plurality of values of the experiment unit.

9. The system of claim 6, wherein the generating of the metric data includes:

parsing the logging data based on at least one of a type of variant of the experiment, the experiment unit, the value of the experiment unit, or a timestamp, the parsing resulting in parsed logging data.

10. The system of claim 6, wherein the generating of the experiment report based on the logging data and the metric data includes:

determining, based on the metric data pertaining to a period of time, a value of the attribute that is associated with the experimental unit and that is measured by the metric data; and

performing a join operation on the logging data pertaining to the period of time and the metric data pertaining to the period of time based on the value of the experiment unit, the performing of the join operation resulting in joint logging and metric data,

wherein the experiment report includes the joint logging and metric data and the value of the attribute that is associated with the experimental unit and that is measured by the metric data.

11. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more hardware processors of a machine, cause the one or more hardware processors to perform operations comprising:

receiving, from a client device, a selection of an identifier of an experiment unit from a plurality of identifiers of a plurality of experiment units displayed in a user interface for specification of an experiment for execution on a social networking service (SNS), and a value of the experiment unit, the plurality of experiment units including an online company page, an email message, and a job page, and the value of the experiment unit including an identifier of a particular online company page hosted by the SNS;

generating, for the value of the experiment unit, logging data that logs user interactions by one or more users of the SNS, via one or more browsers, with content provided during an execution of the experiment;

generating, for the value of the experiment unit, metric data that measures an attribute associated with the experimental unit, the attribute associated with the experimental unit and measured by the metric data, including a number of unique visitors to the particular online company page;

generating an experiment report based on the logging data and the metric data; and

causing a presentation of the experiment report in a user interface of the client device.

12. The non-transitory machine-readable storage medium of claim 11, wherein the generating of the experiment report based on the logging data and the metric data includes:

determining, based on the metric data pertaining to a period of time, a value of the attribute that is associated with the experimental unit and that is measured by the metric data; and

performing a join operation on the logging data pertaining to the period of time and the metric data pertaining to the period of time based on the value of the experiment unit, the performing of the join operation resulting in joint logging and metric data,

wherein the experiment report includes the joint logging and metric data and the value of the attribute that is associated with the experimental unit and that is measured by the metric data.