Dynamic playbook: experimentation platform for social networks

• US 8,706,673 B2
• Filed: 12/07/2012
• Issued: 04/22/2014
• Est. Priority Date: 11/11/2011
• Status: Active Grant

• Current Assignee: Wetpaint.com Incorporated (Function(x), Inc.)
• Original Assignee: Wetpaint.com Incorporated (Function(x), Inc.)
• Inventors: Weinstein, Alex David, Finneman, Bo Steven, Newell, Christopher Edgar, Koepf, Werner, Frenkel, Dmitry
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Smith, Paulinho E
• Application Number:
  US13/708,566
• Publication Number:
  US 20130124634A1
• Time in Patent Office: 501 Days
• Field of Search: None
• US Class Current: 706/47
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: H04L 12/1859 adapted to provide push services, e.g. data channels
• CPC Class Codes: H04L 51/06 Message adaptation to terminal or network requirements

Abstract

Embodiments are directed towards dynamically managing contextual recommendations that can be employed with content when provided to a channel. A test for a variable having at least two outcomes may be determined to be employed with selected content. Each outcome may correspond to a different contextual recommendation. An outcome to employ with the content may be randomly determined/selected such that each of the outcomes of the test is selected a substantially similar number of times. The content may be modified and/or provided to the channel based on contextual recommendations that correspond to the determined outcome. Actions associated with a plurality of content may be monitored and/or analyzed to determine metrics for each outcome of a test. A comparison of the metrics for each outcome may be utilized to determine a preferred outcome for the test.

Claims

1. A method for providing content to a plurality of users over a network, wherein at least one network device performs actions, comprising:

determining a test for a variable having at least two outcomes, wherein each outcome corresponds to a different contextual recommendation to employ with a piece of a plurality of content;

determining one of the at least two outcomes to employ with each piece of the plurality of content, wherein each of the at least two outcomes is determined with a different subset of the plurality of content and each subset includes at least two pieces of the plurality of content; and

providing each piece of the plurality of content to a distribution channel based on at least the contextual recommendation corresponding to the determined outcome; and

removing the test from a plurality of tests and re-employing the removed test by automatically adding the removed test to the plurality of tests.

2. The method of claim 1, further comprising:

determining a first metric for a first outcome of the at least two outcomes based on actions performed on a first subset of the plurality of content that employed the first outcome;

determining a second metric for a second outcome of the at least two outcomes based on actions performed on a second subset of the plurality of content that employed the second outcome; and

if a result of a comparison of the first metric and the second metric is above a threshold, determining one of the first outcome and the second outcome as a preferred outcome.

3. The method of claim 1, further comprising:

analyzing actions associated with each of the plurality of content to determine a metric for each of the at least two outcomes; and

determining a preferred outcome from the at least two outcomes based on a comparison of the metric for each of the at least two outcomes.

4. The method of claim 1, further comprising:

determining a preferred outcome for the test from the at least two outcomes based on monitored actions associated with each of the plurality of content, wherein the test is determined from the plurality of tests;

removing the test from the plurality of tests; and

re-employing the removed test by automatically adding the removed test to the plurality of tests based on other monitored actions associated with the plurality of content.

5. The method of claim 1, further comprising:

removing a completed test from the plurality of tests; and

automatically adding another test for another variable having at least another two outcomes to the plurality of tests.

6. The method of claim 1, further comprising:

selecting the test from a first subset of the plurality of tests, wherein the first subset of the plurality of tests includes tests available for selection and a second subset of the plurality of tests includes tests unavailable for selection;

determining if the selected test is complete based at least on a number of pieces of the plurality of content provided to the distribution channel; and

if the selected test is complete, enabling actions, including;

removing the test from the first subset of the plurality of tests; and

automatically adding another test from the second subset of the plurality of tests to the first subset of the plurality of tests.

7. The method of claim 1, wherein determining the one of the at least two outcomes, further comprises:

randomly selecting the one of the at least two outcomes such that a number of times that each of the least two outcomes is selected is substantially similar.

8. The method of claim 1, wherein the test includes a test size that corresponds to a minimum threshold number of pieces of content and the test size is based on a predetermined statistical confidence level for determining a preferred outcome from the at least two outcomes.

9. The method of claim 1, wherein providing each piece of the plurality of content to the distribution channel, further comprises:

modifying at least one piece of the plurality of content based on the contextual recommendation corresponding to the outcome determined for the at least one piece of the plurality of content; and

providing the modified at least one piece of the plurality of content to the distribution channel.

10. A system for providing content to a plurality of users over a network, comprising:

at least one network device, comprising;

a memory or storing data and instructions;

a processor that executes the instructions to enable actions, comprising;

determining a test for a variable having at least two outcomes, wherein each outcome corresponds to a different contextual recommendation to employ with a piece of a plurality of content;

determining one of the at least two outcomes to employ with each piece of the plurality of content, wherein each of the at least two outcomes is determined with a different subset of the plurality of content and each subset includes at least two pieces of the plurality of content;

providing each piece of the plurality of content to a distribution channel based on at least the contextual recommendation corresponding to the determined outcome; and

removing the test from a plurality of tests and re-employing the removed test by automatically adding the removed test to the plurality of tests.

11. The system of claim 10, further comprising:

determining a first metric for a first outcome of the at least two outcomes based on actions performed on a first subset of the plurality of content that employed the first outcome;

determining a second metric for a second outcome of the at least two outcomes based on actions performed on a second subset of the plurality of content that employed the second outcome; and

if a result of a comparison of the first metric and the second metric is above a threshold, determining one of the first outcome and the second outcome as a preferred outcome.

12. The system of claim 10, further comprising:

analyzing actions associated with each of the plurality of content to determine a metric for each of the at least two outcomes; and

determining a preferred outcome from the at least two outcomes based on a comparison of the metric for each of the at least two outcomes.

13. The system of claim 10, further comprising:

determining a preferred outcome for the test from the at least two outcomes based on monitored actions associated with each of the plurality of content, wherein the test is determined from the plurality of tests;

removing the test from the plurality of tests; and

re-employing the removed test by automatically adding the removed test to the plurality of tests based on other monitored actions associated with the plurality of content.

14. The system of claim 10, further comprising:

removing a completed test from the plurality of tests; and

automatically adding another test for another variable having at least another two outcomes to the plurality of tests.

15. The system of claim 10, further comprising:

selecting the test from a first subset of the plurality of tests, wherein the first subset of the plurality of tests includes tests available for selection and a second subset of the plurality of tests includes tests unavailable for selection;

determining if the selected test is complete based at least on a number of pieces of the plurality of content provided to the distribution channel; and

if the selected test is complete, enabling actions, including;

removing the test from the first subset of the plurality of tests; and

automatically adding another test from the second subset of the plurality of tests to the first subset of the plurality of tests.

16. The system of claim 10, wherein determining the one of the at least two outcomes, further comprises:

randomly selecting the one of the at least two outcomes such that a number of times that each of the least two outcomes is selected is substantially similar.

17. The system of claim 10, wherein the test includes a test size that corresponds to a minimum threshold number of pieces of content and the test size is based on a predetermined statistical confidence level for determining a preferred outcome from the at least two outcomes.

18. The system of claim 10, wherein providing each piece of the plurality of content to the distribution channel, further comprises:

modifying at least one piece of the plurality of content based on the contextual recommendation corresponding to the outcome determined for the at least one piece of the plurality of content; and

providing the modified at least one piece of the plurality of content to the distribution channel.

19. A processor readable non-transitory storage media that includes instructions for providing content to a plurality of users over a network, wherein the execution of the instructions by a processor enables actions, comprising:

determining a test for a variable having at least two outcomes, wherein each outcome corresponds to a different contextual recommendation to employ with a piece of content;

determining one of the at least two outcomes to employ with each piece of the plurality of content, wherein each of the at least two outcomes is determined with a different subset of the plurality of content and each subset includes at least two pieces of the plurality of content;

providing each piece of the plurality of content to the distribution channel based on at least the contextual recommendation corresponding to the determined outcome; and

removing the test from a plurality of tests and re-employing the removed test by automatically adding the removed test to the plurality of tests.

20. The media of claim 19, further comprising:

determining a first metric for a first outcome of the at least two outcomes based on actions performed on a first subset of the plurality of content that employed the first outcome;

determining a second metric for a second outcome of the at least two outcomes based on actions performed on a second subset of the plurality of content that employed the second outcome; and

if a result of a comparison of the first metric and the second metric is above a threshold, determining one of the first outcome and the second outcome as a preferred outcome.

21. The media of claim 19, further comprising:

analyzing actions associated with each of the plurality of content to determine a metric for each of the at least two outcomes; and

determining a preferred outcome from the at least two outcomes based on a comparison of the metric for each of the at least two outcomes.

22. The media of claim 19, further comprising:

determining a preferred outcome for the test from the at least two outcomes based on monitored actions associated with each of the plurality of content, wherein the test is determined from the plurality of tests;

removing the test from the plurality of tests; and

re-employing the removed test by automatically adding the removed test to the plurality of tests based on other monitored actions associated with the plurality of content.

23. The media of claim 19, further comprising:

removing a completed test from the plurality of tests; and

automatically adding another test for another variable having at least another two outcomes to the plurality of tests.

24. The media of claim 19, further comprising:

selecting the test from a first subset of the plurality of tests, wherein the first subset of the plurality of tests includes tests available for selection and a second subset of the plurality of tests includes tests unavailable for selection;

determining if the selected test is complete based at least on a number of pieces of the plurality of content provided to the distribution channel; and

if the selected test is complete, enabling actions, including;

removing the test from the first subset of the plurality of tests; and

automatically adding another test from the second subset of the plurality of tests to the first subset of the plurality of tests.

25. The media of claim 19, wherein determining the one of the at least two outcomes, further comprises:

randomly selecting the one of the at least two outcomes such that a number of times that each of the least two outcomes is selected is substantially similar.

26. The media of claim 19, wherein the test includes a test size that corresponds to a minimum threshold number of pieces of content and the test size is based on a predetermined statistical confidence level for determining a preferred outcome from the at least two outcomes.

27. The media of claim 19, wherein providing each piece of the plurality of content to the distribution channel, further comprises:

modifying at least one piece of the plurality of content based on the contextual recommendation corresponding to the outcome determined for the at least one piece of the plurality of content; and

providing the modified at least one piece of the plurality of content to the distribution channel.