On-line experimentation

• US 20050159921A1
• Filed: 03/14/2005
• Published: 07/21/2005
• Est. Priority Date: 08/26/1999
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services GmbH (Accenture PLC)
• Inventors: Phillips, Hikaru, Louviere, Jordan J., Bennett, Jason P.
• Granted Patent: US 7,308,497 B2
• Time in Patent Office: Days
• US Class Current: 702/181
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

An automated system for experimentation includes an experiment engine which can define an experiment relating to various treatments for a set of content elements. The experiment engine conducts the experiment over a data network. An observation module collects observation data relating to user behavior for each treatment. A scripting/scheduling engine coordinates the operation of the experiment engine and the observation module.

Claims

1-46. -46. (canceled)

47. A computer system for automatically managing delivery of content to users during interactive sessions between the system and users conducted via a user interface, the computer system comprising:

a computer memory content store to hold content items each of which is made up of a set of elemental components;

a programmed experiment engine to hold rules governing the systematic variation of the elemental components, and rules governing the delivery of the variations to users;

a programmed allocator module to retrieve content items from the computer memory content store, and interact with the programmed experiment engine to deliver variations to the users during the interactive sessions according to the rules, in order to conduct an experiment to test the influence of the variations on the likelihood of achieving a predetermined outcome; and

,

a programmed observation module to record information about the actions taken by the users in response to the variations delivered during the experiment, and to relate the variations to the likelihood of the predetermined outcome.

48. A computer system according to claim 47, wherein the rules which govern the delivery of the variations to users are situation specific rules, including experiment rules, prediction rules or user-defined rules.

49. A computer system according to claim 47, wherein the allocator module samples users at the user interface during an experiment and determines which users will be assigned to receive experimental content and which will be used as experimental controls.

50. A computer system according to claim 49, wherein the system implements statistical sampling procedures to deliver the different variations to respective users.

51. A computer system according to claim 47, wherein the system conducts different experiments at the same time to increase the likelihood of achieving the predetermined outcome.

52. A computer system according to claim 47, wherein the system conducts different experiments over different user interfaces to increase the likelihood of achieving the predetermined outcome.

53. A computer system according to claim 47, wherein the model engine identifies the degree to which the variations influence the behavior of users.

54. A computer system according to claim 53, wherein the model engine creates one or more behavioral models which provide abstract descriptions of observed user behavior based on recorded actions in response to variations.

55. A computer system according to claim 54, wherein a prediction engine automatically converts the behavioral models into predictive models that predict the likelihood of future actions by users in response to variations.

56. A computer system according to claim 55, wherein the prediction engine uses the predictive models to determine a variation for achieving the predetermined outcome, and recommends that variation for delivery to users.

57. A computer system according to claim 56, wherein the allocation module delivers the recommended variation.

58. A computer system according to claim 55, wherein the prediction engine automatically modifies a content item using the predictive models, to influence the likelihood of achieving the predetermined outcome when a future user is presented with that modified content item by the allocator module.

59. A computer system according to claim 55, wherein the system dynamically segments users into a plurality of segments each of which comprises users with similar behavioral characteristics.

60. A computer system according to claim 59, wherein the content store collects and holds data about users from extrinsic databases.

61. A computer system according to claim 59, wherein the prediction engine also generates prediction rules for targeting specific content to particular segments.

62. A computer system according to claim 47, wherein it further comprises a scripting/scheduling engine to automate the operation of the entire system by using the recorded information to generate new content variations, new rules governing the systematic variation of the elemental components, and new rules governing the delivery of the variations to users.

63. A computer system according to claim 62, wherein the experiment engine generates new experiments by determining the number of content items, the number of elemental components in each content item, the specific content variations that will be delivered during the experiments, and the number of deliveries of the variations that will be made to users.

64. A computer system according to claim 47, wherein all the steps are repeated to incrementally improve the information to relate the variations to the likelihood of the predetermined outcome more accurately.

65. A method for automatically managing delivery of content to users during interactive sessions between a computer system and users conducted via a programmed user interface, the method comprising causing the computer system to perform the following steps:

storing content items each made up of a set of elemental components in a computer memory;

storing rules governing the systematic variation of the elemental components, and rules governing the delivery of the variations to users in the computer memory;

retrieving content items from the computer memory, varying the components, and delivering variations to the users at the user interface according to the rules to conduct an experiment to test the influence of the variations on the likelihood of achieving a predetermined outcome;

recording the actions taken by the users at the user interface in response to the variations delivered during the experiment, and

processing the recorded information to relate the variations to the likelihood of the predetermined outcome.

66. A method according to claim 65, wherein the rules which govern the delivery of the variations to users are situation specific rules, including experiment rules, prediction rules or user-defined rules.

67. A method according to claim 65, wherein users are sampled at the user interface during an experiment to determine which users will be assigned to receive experimental content and which will be used as experimental controls.

68. A method according to claim 67, wherein the system implements statistical sampling procedures to deliver the different variations to respective users.

69. A method according to claim 65, wherein the system conducts different experiments at the same time to increase the likelihood of achieving the predetermined outcome.

70. A method according to claim 65, wherein the system conducts different experiments over different user interfaces to increase the likelihood of achieving the predetermined outcome.

71. A method according to claim 65, wherein the degree to which the variations influence the behavior of users is identified.

72. A method according to claim 71, wherein one or more behavioral models are created to provide abstract descriptions of observed user behavior based on recorded actions in response to variations.

73. A method according to claim 72, wherein the behavioral models are automatically converted into predictive models that predict the likelihood of future actions by users in response to variations.

74. A method according to claim 73, wherein the predictive models are used to determine a variation for achieving the predetermined outcome, and recommends that variation for delivery to users.

75. A method according to claim 74, wherein the recommended variation is delivered.

76. A method according to claim 73, wherein a content item is automatically modified using the predictive models, to influence the likelihood of achieving the predetermined outcome when a future user is presented with that modified content item.

77. A method according to claim 73, wherein the system dynamically segments users into a plurality of segments each of which comprises users with similar behavioral characteristics.

78. A method according to claim 77, wherein data about users is collected from extrinsic databases and stored.

79. A method according to claim 77, wherein prediction rules are also generated for targeting specific content to particular segments.

80. A method according to claim 65, wherein the entire system is automatically operated by using the recorded information to generate content variations, rules governing the systematic variation of the elemental components, and rules governing the delivery of the variations to users, and thereby generate new experiments.

81. A method according to claim 80, wherein new experiments are generated by determining the number of content items, the number of elemental components in each content item, the specific content variations that will be delivered during the experiments, and the number of deliveries of the variations that will be made to users.

82. A method according to claim 65, wherein all the steps are repeated to incrementally improve the information to relate the variations to the likelihood of the predetermined outcome more accurately.

83. A machine readable medium including a set of instructions for:

managing delivery of content to users during interactive sessions between a computer system and users conducted via a programmed user interface;

storing content items each made up of a set of elemental components in a computer memory;

storing rules governing the systematic variation of the elemental components, and rules governing the delivery of the variations to users in the computer memory;

retrieving content items from the computer memory, varying the components, and delivering variations to the users at the user interface according to the rules to conduct an experiment to test the influence of the variations on the likelihood of achieving a predetermined outcome;

recording the actions taken by the users at the user interface in response to the variations delivered during the experiment, and

processing the recorded information to relate the variations to the likelihood of the predetermined outcome.

84. The machine readable medium of claim 83, wherein the rules which govern the delivery of the variations to users are situation specific rules, including experiment rules, prediction rules or user-defined rules.

87. The machine readable medium of claim 83, wherein the system conducts different experiments at the same time to increase the likelihood of achieving the predetermined outcome.

88. The machine readable medium of claim 83, wherein the system conducts different experiments over different user interfaces to increase the likelihood of achieving the predetermined outcome.

98. The machine readable medium of claim 83, wherein it further comprises a scripting/scheduling engine to automate the operation of the entire system by using the recorded information to generate new content variations, new rules governing the systematic variation of the elemental components, and new rules governing the delivery of the variations to users.

99. The machine readable medium of claim 98, wherein the experiment engine generates new experiments by determining the number of content items, the number of elemental components in each content item, the specific content variations that will be delivered during the experiments, and the number of deliveries of the variations that will be made to users.

100. The machine readable medium of claim 83, wherein all the steps are repeated to incrementally improve the information to relate the variations to the likelihood of the predetermined outcome more accurately.

85. The machine readable medium of 83, wherein the allocator module samples users at the user interface during an experiment and determines which users will be assigned to receive experimental content and which will be used as experimental controls.

86. The machine readable medium of 85, wherein the system implements statistical sampling procedures to deliver the different variations to respective users.

89. The machine readable medium of 83, wherein the model engine identifies the degree to which the variations influence the behavior of users.

90. The machine readable medium of 89, wherein the model engine creates one or more behavioral models which provide abstract descriptions of observed user behavior based on recorded actions in response to variations.

91. The machine readable medium of claim 90, wherein a prediction engine automatically converts the behavioral models into predictive models that predict the likelihood of future actions by users in response to variations.

92. The machine readable medium of claim 91, wherein the prediction engine uses the predictive models to determine a variation for achieving the predetermined outcome, and recommends that variation for delivery to users.

94. The machine readable medium of claim 91, wherein the prediction engine automatically modifies a content item using the predictive models, to influence the likelihood of achieving the predetermined outcome when a future user is presented with that modified content item by the allocator module.

95. The machine readable medium of claim 91, wherein the system dynamically segments users into a plurality of segments each of which comprises users with similar behavioral characteristics.

96. The machine readable medium of claim 95, wherein the content store collects and holds data about users from extrinsic databases.

97. The machine readable medium of claim 95, wherein the prediction engine also generates prediction rules for targeting specific content to particular segments.

93. The machine readable medium of 92, wherein the allocation module delivers the recommended variation.

101. An online experimentation system configured to automatically create various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set, and to allocate the treatments to users to test user behavior in response to the various treatments, the system being further configured to capture data relating to the observed behavior of users relative to the various treatments.

102. An automated system for delivery of content, comprising:

a communication management system including a prediction engine configured to generate predictions on how users who are to be presented a treatment comprising a set of content elements will react to the treatment, the prediction engine generating predictions based on data collected from one or more experiments that tested user behavior to content elements in the set, the communication management system delivering treatments to users in accordance with the predictions to increase the probability of a desired user behavior in response to the presented treatment.

103. An automated method, comprising:

generating predictions on how users who are to be presented a treatment comprising a set of content elements will react to the treatment, the predictions being generated from data collected from one or more experiments that tested user behavior to content elements in the set; and

delivering treatments to users in accordance with the predictions to increase the probability of a desired user behavior in response to the presented conduct.

104. A machine readable medium including a set of instructions for:

generating predictions on how users who are to be presented a treatment comprising a set of content elements will react to the treatment, the predictions being generated from data collected from one or more experiments that tested user behavior to content elements in the set; and

delivering treatments to users in accordance with the predictions to increase the probability of a desired user behavior in response to the presented conduct.