Automated on-line experimentation to measure users behavior to treatment for a set of content elements

• US 6,934,748 B1
• Filed: 08/25/2000
• Issued: 08/23/2005
• Est. Priority Date: 08/26/1999
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Memetrics Holdings Pty Limited (Accenture PLC)
• Inventors: Phillips, Hikaru, Louviere, Jordan J., Bennett, Jason P.
• Primary Examiner(s): Prieto, B.
• Application Number:
  US09/648,429
• Time in Patent Office: 1,824 Days
• Field of Search: 709/201, 709/203, 709217-219, 709223-224, 345/962, 705/10, 705/52
• US Class Current: 709/224
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

An automated system for experimentation includes an experiment engine which can define an experiment relating to various treatments for a set of content elements. The experiment engine conducts the experiment over a data network. An observation module collects observation data relating to user behavior for each treatment. A scripting/scheduling engine coordinates the operation of the experiment engine and the observation module.

Claims

1. An automated computer-implemented method comprising:

defining an experiment to gauge user reaction to various treatments for a set of content elements,

conducting the experiment over a data network; and

collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises;

statistically sampling to specify a plurality of control groups, each control group comprising at least one user;

specifying a particular treatment to be delivered to the at least one user in each control group;

receiving identical requests for content from the respective at least one user in each of the control groups; and

in response to the identical requests, delivering to the at least one user in each control group a different treatment.

2. The method of claim 1 wherein defining an experiment comprises:

identifying desired objectives for user behavior; and

identifying which treatments may influence user behavior related to the desired objectives.

3. The method of claim 2 further comprising:

defining at least one control variable for the various treatments; and

assigning a respective level for the control variable for each treatment.

4. The method of claim 1 wherein conducting the experiment comprises:

selecting at least one treatment for delivery to users;

grouping users into a plurality of segments, each segment comprising users with similar behavioral characteristics; and

specifying a particular segment of users to receive the selected treatment.

5. The method of claim 1 further comprising observing site-related behavior of users receiving the various treatments.

6. The method of claim 5 wherein collecting observation data comprises recording observed behavior in each control group during the experiment.

7. The method of claim 1 wherein the data network comprises the internet.

8. The method of claim 1 further comprising identifying elements of content which potentially influence behavior of users.

9. The method of claim 1 wherein the experiment is delivered over the data network to respective control groups of users according to the experiment and wherein the method further comprises generating a model for predicting anticipated behavior of users based upon the observed behavior;

and generating a prediction for delivering one of the treatments to users according to the model in order to optimize a desired outcome.

10. The method of claim 9 wherein defining an experiment comprises identifying elements of content which potentially influence behavior of users.

11. An automated system comprising:

a content system operable to store content, the content comprising a set of content elements; and

a communication management system in communication with the content system and operable to define an experiment relating to various treatments for the set of content elements, conduct the experiment over a data network, and collect over the data network observation data relating to user behavior for each treatment, wherein the system is configured for statistically sampling to specify a plurality of control groups, each control group comprising at least one user;

specifying a particular treatment to be delivered to the at least one user in each control group;

receiving identical requests for content from the respective at least one user in each of the control groups; and

in response to the identical requests, delivering to the at least one user in each control group a different treatment.

12. The system of claim 11 wherein the communication management system is operable to identify content elements which may influence user behavior.

13. The system of claim 11 wherein the communication management system is operable to generate a set of experiment rules for allocating treatments during the experiment.

14. The system of claim 11 wherein the content system is operable to allocate treatments to users according to the experiment rules.

15. The system of claim 11 wherein the communication management system is operable to group users into a plurality of segments, each segment comprising users with similar behavioral characteristics.

16. The system of claim 11 wherein said data network comprises the internet.

17. The system of claim 11 wherein the communication management system is operable to define at least one control variable for the various treatments and to assign a respective level for the control variables for each treatment.

18. The system of claim 11 wherein the content system and the communication management system are connected to the data network.

19. The system of claim 11 wherein the content system comprises an allocator module operable to support an interface with the communication management system.

20. The system of claim 11 wherein the communication management system comprises an allocator interface object operable to support an interface with the content system.

21. The system of claim 11 further comprising a content provider interface operable to support an interface between the communication management system and a manager user.

22. The system of claim 11 wherein the communication management system is operable to generate a set of prediction rules for allocating treatments to users with specific behavioral characteristics.

23. The system of claim 22 wherein the content system is operable to allocate treatments to users according to the prediction rules.

24. The system of claim 11 wherein the communication management system implements a personalization process.

25. The system of claim 11 wherein the designed experiment reduces the number of combinations included in the experiment by measuring and estimating only those effects that are of a predetermined interest.

26. A computer-implemented method for on-line experimentation comprising:

defining a plurality of treatments for a set of content elements;

receiving requests for content from users over a data network;

statistically sampling to create at least one control group of users;

allocating over the data network a first treatment to each user in the control group;

allocating over the data network a second treatment to each user not in the control group; and

collecting observation data for observed behavior of users in the control group and users not in the control group.

27. The method of claim 26 further comprising identifying elements of content which may influence user behavior.

28. The method of claim 26 wherein providing a website comprises providing a web page at which the set of content elements is available in the form of any of the treatments.

29. The method of claim 26 wherein the data network comprises the internet.

30. An online experimentation system configured to automatically create various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set, and to allocate the treatments to users to test user behavior in response to the various treatments, the system being further configured to capture data relating to the observed behavior of users relative to the various treatments and to perform one or more methods comprising:

(1) analyze the data to derive a model in form of a contingency table;

(2) process the data for pathologies, including at least one of missing data, and structural dependencies, and processing the data to be model ready;

(3) make inferences about some variables that influence user behavior based on random utility theory; and

(4) modeling user behavior based on Bayesian Markov Chain Monte Carlo estimation procedure.

31. A machine readable medium including a set of instructions for:

defining an experiment to gauge user reaction to various treatments for a set of content elements;

conducting the experiment over a data network; and

collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises;

statistically sampling to specify a plurality of control groups, each control group comprising at least one user;

specifying a particular treatment to be delivered to the at least one user in each control group;

receiving identical requests for content from the respective at least one user in each of the control groups; and

in response to the identical requests, delivering to the at least one user in each control group a different treatment.

32. A content page comprising a set of content elements generated by a computer implemented system for:

defining an experiment to gauge user reaction to various treatments for a set of content elements;

conducting the experiment over a data network; and

collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises;

statistically sampling to specify a plurality of control groups, each control group comprising at least one user;

specifying a particular treatment to be delivered to the at least one user in each control group;

receiving identical requests for content from the respective at least one user in each of the control groups; and

in response to the identical requests, delivering to the at least one user in each control group a different treatment.

33. An automated system for experimentation, the system being configured to:

generate and allocate a set of treatments from a predefined set of control elements, according to experiment designs, the system capable of determining a design by selecting from design criteria comprising;

the specific effects allowed;

the number of factors and factor levels included; and

the amount of information produced in relation to the objective outcome; and

collect observation data of user behavior to the set of treatments and determine the effect of each control element on user behavior according to choice models, behavioral models, or contingency tables.

34. The system of claim 33 wherein the experiment engine is operable to implement statistical sampling procedures to deliver over the data network the various treatments to respective control groups of users.

35. The system of claim 33 wherein the data network comprises the internet.

36. The system of claim 33 wherein the system is further configured to determine and allocate a treatment from the set of content elements to increase the probability of achieving a desired objective based on the determination of the effects of the content elements.

37. A computer implemented method comprising:

automatically creating various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set;

allocating the treatments to users to test user behavior in response to the various treatments;

capturing data relating to the observed behavior of users relative to the various treatments;

performing one or more of;

(1) analyzing the data to derive a model in form of a contingency table;

(2) processing the data for pathologies, including at least one of missing data, and structural dependencies, and processing the data to be model ready;

(3) making inferences about some variables that influence user behavior based on random utility theory; and

(4) modeling user behavior based on Bayesian Markov Chain Monte Carlo estimation procedure.

38. A computer implemented method, comprising:

generating and allocating a set of treatments from a predefined set of content elements, according to experiment designs, the system capable of determining a design by selecting from design criteria comprising;

the specific effects allowed;

the number of factors and factor levels included; and

the amount of information produced in relation to the objective outcome; and

collecting observation data of user behavior for the set of treatments and determining the effect of each content element on user behavior according to choice models, behavioral models, or contingency tables.

39. The method of claim 38 further comprising determining and allocating a treatment from the set of content elements to increase the probability of achieving a desired objective based on the determination of the effects of the content elements.

40. A machine readable medium storing a set of instructions for performing the steps comprising:

automatically creating various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set;

allocating the treatments to users to test user behavior in response to the various treatments

capturing data relating to the observed behavior of users relative to the various treatments;

performing one or more of;

(1) analyzing the data to derive a model in form of a contingency table;

(2) processing the data for pathologies, including at least one of missing data, and structural dependencies, and processing the data to be model ready;

(3) making inferences about some variables that influence user behavior based on random utility theory; and

(4) modeling user behavior based on Bayesian Markov Chain Monte Carlo estimation procedure.

41. A machine readable medium storing a set of instructions for performing the steps comprising:

generating and allocating a set of treatments from a predefined set of content elements, according to experiment designs, the system capable of determining a design by selecting from design criteria comprising;

the specific effects allowed;

the number of factors and factor levels included; and

the amount of information produced in relation to the objective outcome; and

collecting observation data of user behavior for the set of treatments and determining the effect of each content element on user behavior according to choice models, behavioral models, or contingency tables.

42. The machine readable medium of claim 41 further comprising instructions for determining and allocating a treatment from the set of content elements to increase the probability of achieving a desired objective based on the determination of the effects of the content elements.

43. A computer implemented method, comprising:

defining an experiment to gauge user reaction to various treatments for a set of content elements;

conducting the experiment over a data network; and

collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises;

statistically sampling to specify a plurality of control groups, each control group comprising at least one user;

specifying a particular treatment to be delivered to the at least one user in each control group;

receiving identical requests for content from the respective at least one user in each of the control groups; and

in response to the identical requests, delivering to the at least one user in each control group a different treatment.