Methods, systems, and articles of manufacture for determining optimal parameter settings for business initiative testing models

• US 8,571,916 B1
• Filed: 03/01/2006
• Issued: 10/29/2013
• Est. Priority Date: 01/30/2004
• Status: Expired

• Current Assignee: Applied Predictive Technologies (MasterCard Incorporated)
• Original Assignee: Applied Predictive Technologies (MasterCard Incorporated)
• Inventors: Bruce, Anthony Dean, D'Agostino, Mark James, Vaudo, Corey Christian, Zhao, Dacheng
• Primary Examiner(s): Dunham, Jason
• Assistant Examiner(s): Mehta, Nancy
• Application Number:
  US11/364,197
• Time in Patent Office: 2,799 Days
• Field of Search: 705/7.11, 705 728- 738, 717/124, 717/126, 717/127, 717/131
• US Class Current: 705/7.28
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements

Abstract

A system, method, and article of manufacture is disclosed for determining optimal parameter settings for a business initiative testing model used for testing initiatives for business locations included in a business network. In one aspect, a method is disclosed that includes defining a first test type of a business initiative testing model having a set of parameter settings. Each parameter setting may include a set of one or more parameter setting options. The method may also include performing virtual tests on a set of virtual test sites based on the defined test type. Each virtual test site may reflect a selected business location in the business network. Also, the method may include determining a set of optimal parameter settings for the first test type of the business initiative testing model based on results from the virtual test. Moreover, the method may include configuring the business initiative testing model using the optimal parameter settings from the set for the first test type to test a business initiative to apply to the business network.

Claims

1. A method for determining optimal parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network, comprising:

identifying, by a computer, a business initiative testing model having a set of parameter settings;

selecting a first parameter setting set for performing the virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the business initiative testing model;

performing, by a computer, a virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test performed on test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is performed on the virtual test sites using a variation of each parameter setting;

determining, by a computer, actual performance data associated with the set of virtual test sites;

determining, by a computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining a noise value for the first parameter setting set, the noise value reflecting an inconsistency between performance data associated, with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining, by a computer, a set of optimal parameter settings for the business initiative testing model based on results from the virtual test whereby the optimal parameter settings best minimize noise from the results; and

configuring, by a computer, the business initiative testing model using the optimal parameter settings to test a business initiative for application in the business network.

2. The method of claim 1, wherein performing the virtual test includes:

performing a series of virtual tests for each possible combination of the parameter settings for the business initiative testing model.

3. The method of claim 2, wherein iteratively performing the series of virtual tests includes:

determining a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.

4. The method of claim 3, further including:

for each iteration of the series of virtual tests for a given parameter settings combination,

storing the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.

5. The method of claim 3, further including:

for each given parameter settings combination,

determining an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.

6. The method of claim 2, wherein performing the series of virtual tests for each possible combination of the parameter settings includes:

for each iteration of the series of virtual tests,

randomly selecting the set of virtual test sites, and

randomly selecting a test start date for collecting historical data associated with each of the randomly selected virtual test sites.

7. The method of claim 1, further including:

iteratively performing the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.

8. The method of claim 1, wherein iteratively performing the virtual test includes:

storing a noise value for each iteration and first parameter setting in a data structure.

9. The method of claim 1, further including:

selecting a second parameter setting set when the iteration threshold is met; and

iteratively performing the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.

10. The method of claim 1, wherein iteratively performing the virtual test includes:

storing a noise value for each iteration and second parameter setting in the data structure.

11. The method of claim 1, wherein determining optimal parameter settings for the first test type includes:

analyzing the noise values associated with the first and second parameter settings sets; and

selecting either the first or second parameter settings set as the optimal parameter settings based on whether the first or second parameter settings sets have a lower average noise value over the iterations of the virtual test.

12. The method according to claim 1, further comprising defining a test type associated with the business initiative testing model, wherein the test type is defined based on an attribute of the business initiative test.

13. The method of claim 12, wherein determining the set of optimal parameter settings includes:

identifying the parameter settings combination having the lowest average noise value as the optimal parameter settings for the first test type of the business initiative testing model.

14. The method of claim 12, wherein configuring the business initiative testing model further includes:

receiving a request identifying the first test type to test the business initiative to apply to the business network.

15. The method of claim 12, wherein defining the first test type includes:

defining a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.

16. The method of claim 12, further including:

defining a second test type of the business initiative model;

performing the virtual test based on the defined second test type to determine second optimal parameter settings for the second test type; and

performing a business initiative test using the second test type of the business initiative testing model that is automatically configured using the second optimal parameter settings.

17. The method according to claim 1, wherein each parameter setting comprises a set of one or more parameter setting options, wherein a parameter setting option further defines a parameter setting.

18. A system for determining optimal parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network, the system including:

a computer configured to;

identify a business initiative testing model having a set of parameter settings;

selecting a first parameter setting set for performing the virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the business initiative testing model;

execute a virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test for-test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is performed on the virtual test sites using a variation of each parameter setting;

determine actual performance data associated with the set of virtual test sites;

determine actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining a noise value for the first parameter setting set, the noise value reflecting an inconsistency between performance data associated, with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determine a set of optimal parameter settings for the business initiative testing model based on results from the virtual test whereby the optimal parameter settings best minimize noise from the results; and

configure the business initiative testing model using the optimal parameter settings to a test on a business initiative to apply to the business network.

19. The system of claim 18, wherein the computer is further configured to:

perform a series of virtual tests for each possible combination of the parameter settings for the business initiative testing model.

20. The system of claim 19, wherein the computer is further configured to perform the series of virtual tests by:

determining a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.

21. The system of claim 20, wherein the computer is further configured to:

for each iteration of the series of virtual tests for a given parameter settings combination,

store the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.

22. The system of claim 20, wherein the computer is further configured to:

for each given parameter settings combination,

determine an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.

23. The system of claim 22, wherein the computer is further configured to determine the set of optimal parameter settings by identifying the parameter settings combination having the lowest average noise value as the optimal parameter settings of the business initiative testing model.

24. The system of claim 19, wherein the computer is further configured to perform the series of virtual tests for each possible combination of the parameter settings by:

for each iteration of the series of virtual tests, randomly selecting the set of virtual test sites, and

randomly selecting a test start date for collecting historical'"'"' data associated with each of the randomly selected virtual test sites.

25. The system of claim 24, wherein the computer is further configured to:

defining a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.

26. The system of claim 18, wherein the computer is further configured to:

receive a request identifying the first test type to test the business initiative to apply to the business network, and

define the first test type of the business initiative testing model based on the request.

27. The system of claim 18, wherein the computer is further configured to:

iteratively perform the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.

28. The system of claim 18, wherein the computer is further configured to iteratively perform the virtual test by storing a noise value for each iteration and first parameter setting in a data structure.

29. The system of claim 18, wherein the computer is further configured to:

select a second parameter setting set when the iteration threshold is met; and

iteratively perform the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.

30. The system of claim 18, wherein the computer is further configured to iteratively perform the virtual test by storing a noise value for each iteration and second parameter setting in the data structure.

31. The system of claim 18, wherein the computer is further configured to determine optimal parameter settings by:

analyzing the noise values associated with the first and second parameter settings sets; and

selecting either the first or second parameter settings set as the optimal parameter settings based on whether the first or second parameter settings sets have a lower average noise value over the iterations of the virtual test.

32. The system according to claim 18, wherein a test type is associated with the business initiative testing model and is defined based on an attribute of the business initiative test.

33. The system of claim 32, wherein the computer is further configured to:

define a second test type of the business initiative model;

perform the virtual test based on the defined second test type to determine second optimal parameter settings for the second test type; and

perform a business initiative test using the second test type of the business initiative testing model that is configured using the second optimal parameter settings.

34. A non-transitory computer-readable medium including instructions for, when executed by a processor, performing a method for determining optimal parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network, the method comprising:

identifying, by a computer, a business initiative testing model having a set of parameter settings;

selecting a first parameter setting set for performing the virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the business initiative testing model;

performing, by a computer, a virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test using a variation of each parameter setting and is performed on test sites where no actual initiative test is performed on the virtual test sites;

determining, by a computer, actual performance data associated with the set of virtual test sites;

determining, by a computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining a noise value for the first parameter setting set, the noise value reflecting an inconsistency between performance data associated, with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining, by a computer, a set of optimal parameter settings for the business initiative testing model based on results from the virtual test whereby the optimal parameter settings best minimize noise from the results; and

configuring, by a computer, the business initiative testing model using the optimal parameter settings to test a business initiative to apply to the business network.

35. A method for testing business initiatives comprising:

receiving, by a computer, a request to test a business initiative to be applied to a set of business locations in a business network;

determining, by a computer, actual performance data associated with a set of virtual test sites;

determining, by a computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

configuring, by a computer, a business initiative testing model based on the request and a default set of parameter settings for the model, each default parameter setting having a set of parameter setting options and the default set of parameter settings reflecting optimal parameter settings for the model that are determined by identifying a combination of parameter setting options that produce the least amount of noise during iterations of a set of virtual business initiative tests performed for selected virtual test sites, wherein each virtual test is a simulated business initiative test performed on test sites where no actual initiative test has been implemented at those test sites and wherein noise is a measurement of inconsistency between performance data associated with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings; and

testing, by a computer, the business initiative using the configured business initiative testing model to produce results reflecting a predicted performance of the business initiative if applied to the set of business locations.