DIGITAL MARKETING OPTIMIZATION

• US 20100100419A1
• Filed: 10/21/2008
• Published: 04/22/2010
• Est. Priority Date: 10/21/2008
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: Murray, Shane, Choi, Jong-won, Natoli, Claudio
• Granted Patent: US 9,002,729 B2
• Time in Patent Office: Days
• US Class Current: 705/10
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0202 Market predictions or forecasting for commercial activities

Abstract

A system for experimentation includes an experiment engine which can define an experiment relating to various treatments for a set of content elements. The experiment engine conducts the experiment by delivering treatments to control groups over the network. The selection of the treatments to deliver is optimized to provide treatments having a greater likelihood of satisfying an objective of the experiment.

Claims

1. A method for conducting an experiment over a data network, the method comprising:

determining a first set of treatments to be delivered to at least one control group over the network for the experiment, wherein each treatment includes a combination of content elements;

delivering the first set of treatments to the at least one control group over the network;

tracking behavior of each control group as each control group is presented with at least one of the treatments in the first set;

determining a confidence for each treatment based on the tracked behavior, wherein each confidence is an estimation of an ability of the corresponding treatment or the combination of content elements in the corresponding treatment to be used to satisfy one or more objectives of the experiment; and

determining a new set of treatments to deliver to the at least one control group based at least on the confidences.

2. The method of claim 1, wherein determining a new set of treatments to deliver to the at least one control group based at least on the determined confidences further comprises:

identifying candidate treatments for the new set;

determining a confidence for each candidate treatment;

weighting each candidate treatment based on the corresponding confidence; and

selecting the candidate treatments for the new set based on the weightings.

3. The method of claim 2, further comprising:

determining a proportion of the candidate treatments for the new set to deliver to each control group based on the weightings.

4. The method of claim 3, wherein candidate treatments determined to have higher weightings as a result of having higher confidences are delivered in higher proportion to each control group than candidate treatments determined to have lower weightings.

5. The method of claim 3, further comprising:

using a confidence threshold to determine treatments not to include in the new set.

6. The method of claim 2, further comprising:

storing confidences for the first set of treatments and any other treatments delivered to the at least one control group.

7. The method of claim 6, wherein determining a confidence for each candidate treatment further comprises:

retrieving a stored confidence for each candidate treatment.

8. The method of claim 1, wherein each treatment comprises a combination of elements, and each element includes a variation of a plurality of variations for the element.

9. The method of claim 1, wherein determining a new set of treatments to deliver to the at least one control group based at least on the confidences further comprises:

determining the new set of treatments based on a prediction of which treatments are estimated to achieve the one or more objectives.

10. The method of claim 9, further comprising:

determining a predictive model from observed behavior of control groups including the at least one control group; and

determining the prediction from the predictive model.

11. The method of claim 1, wherein the new set of treatments include one or more treatments having a same combination of elements of at least one treatment in the first set determined to have a high confidence relative to other treatments previously delivered to the at least one control group.

12. The method of claim 1, further comprising:

determining whether the one or more objectives are satisfied; and

repeating determining a new set of treatments based on confidences for previously delivered treatments if the one or more objectives are not satisfied.

13. A system configured to provide an experiment for evaluating content, the system comprising:

a content system operable to store content, the content comprising treatments, each treatment having a combination of content elements, and

the content system is operable to store a confidence for each treatment, wherein each confidence is an estimation of an ability of the corresponding treatment to be used to satisfy one or more objectives; and

a communication management system in communication with the content system and operable to define an experiment, conduct the experiment by delivering treatments to at least one control group over a data network, and collect over the data network observation data from the at least one control group, the observation data relating to user behavior for each treatment,

wherein the communication management system is operable to determine the stored confidences and the content system is operable to select new treatments to deliver to the at least one control group for the experiment based on the confidences of previously delivered treatments to the at least one control group.

14. The system of claim 13, wherein the content system selects candidate treatments based on the confidences of the previously delivered treatments, and determines a proportion of the candidate treatments to deliver to the control groups based on weightings of the candidate treatments determined from the confidences of the candidate treatments.

15. The system of claim 14, wherein the proportion identifies a ratio of higher to lower confidences for treatments to be delivered.

16. The system of claim 13, wherein the communications management system comprises a prediction engine operable to determine a prediction describing content to be delivered to the at least one control group to achieve the one or more objectives, and the prediction is used to select the new treatments.

17. One or more computer readable mediums storing computer code that when executed by a computer system performs a method comprising:

determining a first set of treatments to be delivered to at least one control group over the network for the experiment, wherein each treatment includes a combination of content elements;

delivering the first set of treatments to the at least one control group over the network;

tracking behavior of each control group as each control group is presented with at least one of the treatments in the first set;

determining a confidence for each treatment based on the tracked behavior, wherein each confidence is an estimation of an ability of the corresponding treatment to be used to satisfy one or more objectives; and

determining a new set of treatments to deliver to the at least one control group based at least on the confidences.

18. The one or more computer readable mediums of claim 17, wherein determining a new set of treatments to deliver to the at least one control group based at least on the determined confidences further comprises:

identifying candidate treatments for the new set;

determining a confidence for each candidate treatment;

weighting each candidate treatment based the confidence score; and

selecting the candidate treatments for the new set based on the weightings.

19. The one or more computer readable mediums of claim 18, wherein the method further comprises:

determining a proportion of the candidate treatments for the new set to deliver to each control group based on the weightings.

20. The one or more computer readable mediums of claim 19, wherein treatments determined to have higher weightings as a result of having higher confidences are delivered in higher proportion to each control group than treatments determined to have lower weightings.