SYSTEM AND METHOD FOR CONCURRENTLY CONDUCTING CAUSE-AND-EFFECT EXPERIMENTS ON CONTENT EFFECTIVENESS AND ADJUSTING CONTENT DISTRIBUTION TO OPTIMIZE BUSINESS OBJECTIVES

• US 20100174671A1
• Filed: 01/04/2010
• Published: 07/08/2010
• Est. Priority Date: 01/07/2009
• Status: Active Grant

• Current Assignee: 3M Innovative Properties Company (3M Company)
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Arthur, Jonathan B., Markell, Craig G., Linzie, Brian L., Brooks, Brian E., Stankiewicz, Brian J.
• Granted Patent: US 8,458,103 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0244 Optimization

Abstract

The present invention is directed to systems, articles, and computer-implemented methods for assessing effectiveness of communication content and optimizing content distribution to enhance business objectives. Embodiments of the present invention are directed to computer-implemented methods for a computer-implemented method, comprising conducting an experiment using experimental content to determine effectiveness of communication content and executing, while conducting the experiment, a machine learning routine (MLR) using MLR content to enhance an effectiveness metric.

Claims

1. A computer-implemented method, comprising:

conducting an experiment using experimental content to determine effectiveness of communication content; and

executing, while conducting the experiment, a machine learning routine (MLR) using MLR content to enhance an effectiveness metric.

2. The method of claim 1 wherein the experiment is a cause and effect experiment.

3. The method of claim 1 wherein the experiment is a quasi-experiment.

4. The method of claim 1 wherein the experiment is a correlational design.

5. The method of claim 1 wherein the experiment ensures that the experimental content of the communication content are not confounded.

6. The method of claim 1, wherein the experiment is conducted in accordance with a schedule, and the machine learning routine executes in coordination with the schedule.

7. The method of claim 1, wherein the experiment is conducted in accordance with a schedule comprising a plurality of time periods, and at least some of the time periods unused for conducting the experiment are used for the machine learning routine.

8. The method of claim 1, wherein the experiment and the machine learning routine are implemented in accordance with a schedule comprising a plurality of time periods, the experiment implemented using a first plurality of the time periods and the machine learning routine implemented using a second plurality of the time periods that are interspersed with the first plurality of time periods.

9. The method of claim 1, wherein the experiment and the machine learning routine are implemented in accordance with a schedule comprising a plurality of time periods, at least a portion of each of the experiment and the machine learning routine implemented using the same time period of the plurality of portions, and experimental content and MLR content are unrelated.

10. The method of claim 1, comprising displaying experimental content and MLR content concurrently on a display, wherein the experimental content and MLR content are unrelated.

11. The method of claim 1, wherein the experiment is conducted in accordance with a schedule, and conducting the experiment comprises assigning communication content to time-slot samples of the schedule in a manner that ensures experimental content of the communication content are not confounded.

12. The method of claim 1, wherein the machine learning routine comprises a reinforcement learning routine, a logistic regression routine, an unsupervised learning routine, a semi-supervised learning routine, or use of one or more neural networks.

13. The method of claim 1, wherein:

the machine learning routine comprises a reinforcement learning routine, the reinforcement learning routine comprising an explore routine associated with explore content and an exploit routine associated with exploit content; and

the experiment is conducted in accordance with a schedule comprising a plurality of time periods, a first plurality of the time periods unused for conducting the experiment are used for the explore routine and a second plurality of the time periods unused for conducting the experiment are used for the exploit routine.

14. The method of claim 13, comprising identifying a frequency rate for presenting the explore content and the exploit content that results in enhancement of the predetermined effectiveness metric.

15. The method of claim 1, comprising performing an experiment to differentiate whether a correlation discovered using the MLR is a spurious correlation or a causal-effect relationship.

16. The method of claim 1, wherein:

the machine learning routine comprises a reinforcement learning routine, the reinforcement learning routine comprising one or both of an explore routine associated with explore content and an exploit routine associated with exploit content;

the experiment is conducted in accordance with a schedule comprising a plurality of time-slot samples;

for experimental content that are related to either of explore or exploit content, using a first plurality of the time-slot samples unused for conducting the experiment for the explore or exploit routine; and

for experimental content that are unrelated to either of explore or exploit content, using at least some of the experimental content and either of explore and exploit content in the same time-slot sample of the plurality of time-slot samples.

17. The method of claim 16, comprising generating the time-slot samples comprising:

defining a viewer visit duration (VVD) that target viewers spend at a location where content is to be presented;

defining time intervals (TI) for data collection for data streams of interest that target viewers can affect during their visit to the location;

determining a time-slot sample duration (TSSD) using VVD and TI; and

generating the time-slot samples each having a duration defined by the TSSD and a data collection period defined for at least a portion of the TSSD.

18. The method of claim 17, wherein a first plurality of the time-slot samples are generated using a first VVD of a first duration and a second plurality of the time-slot samples are generated using a second VVD of a second duration differing from the first duration, and TI, TSSD, and data collection periods are determined based on each of the first and second VVDs.

19. The method of claim 1, comprising performing an evaluation to determine, for any given time period, if using experimental content has more value than using MLR content for the time period.

20. The method of claim 1, comprising:

receiving one or more of (a) a value of the experimental hypothesis or hypotheses, (b) a value of each category of effectiveness metric, (c) VVD for each business objective, (d) content restrictions, and (e) urgency information; and

generating a schedule for implementing the experiment and machine learning routine in accordance with one or more of (a) through (e).

21. The method of claim 20, comprising continuously adjusting the schedule in terms of times and display locations to maximize at least one effectiveness metric.

22. The method of claim 1, comprising analyzing measurement data indicative of MLR content effectiveness for the machine learning routine, and storing the measurement data in a historical database.

23. The method of claim 1, comprising producing output data indicative of return on investment for each piece of MLR content based on one or both of daypart and display location.

24. A computer-implemented method, comprising:

generating a plurality of schedules each unrelated to one another and each comprising a plurality of time periods for presenting content and collecting data indicative of content effectiveness; and

using a digital signage network comprising a plurality of geographically disparate displays and the plurality of schedules for;

concurrently conducting at least two cause-and-effect experiments on effectiveness of communication content that ensures that experimental content of the communication content are not confounded using at least two of the plurality of schedules;

concurrently executing at least two machine learning routines (MLR) using MLR content to enhance a predetermined business goal using at least two of the plurality of schedules;

or

conducting at least one of the cause-and-effect experiments while executing at least one of the machine learning routines using at least two of the plurality of schedules.

25. The method of claim 24, wherein each of the plurality of schedules is defined by at least:

a viewer visit duration (VVD) that target viewers spend at a location where content is to be presented;

time intervals (TI) for data collection for data streams of interest that target viewers can affect during their visit to the location;

a time-slot sample duration (TSSD) using VVD and TI; and

a data collection period defined for at least a portion of the TSSD.

26. The method of claim 24, wherein at least one or VVD, TI, TSSD, and data collection period differs as among the plurality of scheduled.

27. The method of claim 24, wherein each of VVD, TI, TSSD, and data collection period differs as among the plurality of scheduled.

28. A computer-implemented method, comprising:

receiving a viewer visit duration (VVD) for viewers at a location where content is to be presented;

generating a schedule comprising a plurality of time periods for implementing a machine learning routine (MLR) based, in part on the VVD and an effectiveness metric; and

executing, using a digital signage network comprising a plurality of geographically disparate displays, the machine learning routine associated with MLR content in accordance with the schedule to determine effectiveness of the MLR content.

29. The method of claim 28, comprising continuously adjusting the schedule in terms of times and display locations at which each piece of MLR content is shown for each display to maximize at least one effectiveness metric.

30. The method of claim 28, wherein the machine learning routine comprises a reinforcement learning routine, the reinforcement learning routine comprising one or both of an explore routine associated with explore content and an exploit routine associated with exploit content.

31. The method of claim 30, comprising randomly selecting between execution of the explore routine and the exploit routine with a constraint that the total time allocated for running each of the explore and exploit routines is governed in a predetermined manner.

32. The method of claim 28, comprising collecting data about MLR content effectiveness, and storing the collected data in a historical database.

33. The method of claim 32, comprising:

assigning MLR content to time periods of the schedule to generate an adjusted schedule based, in part on the collected data stored in the historical database; and

distributing the assigned MLR content to the displays in accordance with the adjusted schedule.

34. A computer-implemented method, comprising:

performing an evaluation to determine, for any given time period, if using experimental content has more value than using MLR content for the time period; and

assigning experimental content or MLR content to the time period based on the result of the evaluation.

38. The method of claim 34 comprising, after determining an experiment should run, conducting an experiment using experimental content to determine effectiveness of communication content;

and

executing, while conducting the experiment, a machine learning routine (MLR) using MLR content to enhance an effectiveness metric

35. A computer-implemented method, comprising:

receiving data gathered in accordance with a schedule comprising a plurality of time-slot samples; and

executing a machine learning routine (MLR) using content collected from within time-slot samples to enhance an effectiveness metric.

36. The method of claim 35 wherein the data is collected using an experiment.

37. The method of claim 35 wherein the MLR is executed using content identified in the experiment.