Choice modelling system and method

• US 8,799,186 B2
• Filed: 11/02/2011
• Issued: 08/05/2014
• Est. Priority Date: 11/02/2010
• Status: Active Grant

• Current Assignee: Survey Engine Pty Limited
• Original Assignee: Survey Engine Pty Limited
• Inventors: White, Ben, Cross, Clayton, Kennedy-White, Joshua
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Traktovenko, Ilya
• Application Number:
  US13/882,867
• Publication Number:
  US 20130282626A1
• Time in Patent Office: 1,007 Days
• Field of Search: 706/11, 706/62
• US Class Current: 706/11
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 40/00 Handling natural language data speech analysis or synthesis, speech recognition G10L
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data

Abstract

Systems and methods for choice experiments to model user behavior are disclosed. Choice experiment surveys require respondents to choose by performing trade-offs between combinations of features included in alternatives under consideration. Users specify constraints and an attribute space of features to investigate, and generate candidate experimental designs based on the constraints and attributes. Users select an optimum design tailored to the problem to be solved, instead of having to modify their problem to match a known experimental design. An online survey assembly module generates survey templates used by a data collection and sampling unit to display treatments to survey respondents. A model generation module analysis the collected data, and a model explorer module enables exploration of results. The system has the advantage of making choice modelling accessible to a wider range of users, and enables considerable freedom and scope to investigate problems of specific interest.

Claims

1. A computational system for performing an online choice model, the system comprising:

at least one processor;

at least one memory operatively coupled to the at least one processor; and

a plurality of modules, each of the modules comprising instructions for execution by the at least one processor, the plurality of modules comprising;

a problem definition module comprising a problem definition user interface for receiving a plurality of attributes from a user wherein each attribute has an associated plurality of attribute levels;

an online choice model survey module comprising;

an experimental design generator module for generating a survey experimental design and an associated plurality of treatments;

comprising a library of experimental designs,

wherein the experimental design generator module determines the signature of the attribute space from the received plurality of attributes and associated attribute levels; and

for one or more experimental designs in the library of experimental designs, performs one or more transformations until the signature of the transformed experimental design matches the signature of the attribute space to obtain one or more matching transformed experimental designs;

wherein each transformation preserves the information properties of the untransformed experimental design;

wherein the experimental design generator module selects the survey experimental design from the one or more matching transformed experimental designs; and

wherein the experimental design generator module obtains a set of treatments from the selected survey experimental design;

an online survey assembly module which receives the plurality of treatments and assembles an online survey from one or more survey templates pages, and a plurality of treatment representations created using the plurality of treatments received from the experimental design module;

a data collection and sampling module for conducting the assembled online survey, wherein the data collection and sampling module allocates treatments to survey respondents and collects responses using the assembled online survey;

a model generation module for receiving the data collected by the data collection and sampling module and building a model to obtain a plurality of model parameter estimates and errors from which a utility estimate can be obtained for each attribute level; and

a model explorer module comprising a model explorer user interface for allowing the user to enter one or more attribute levels and obtain a model prediction of the expected utility;

wherein the one or more transformations comprise one or more of the following group of transformations;

factorial splitting of a factor F into two sub factors A, B where A×

B=F and A or B match at least one unmatched factor in the signature;

factorial expansion of a factor F into a new factor A×

F where A×

F matches at least one unmatched factor in the signature;

factor truncation of a factor F into a new factor F−

A where F−

A matches at least one unmatched factor in the signature;

full factorization by generation of a new factor F where F matches at least one unmatched factor in the signature; and

deleting a factor F when all the other factors in the signature are matched.

2. The computational system as claimed in claim 1, wherein the problem definition user interface receives one or more survey constraints.

3. The computational system as claimed in claim 2, wherein the survey constraints comprise one or more criteria for selecting a survey experimental design.

4. The computational system as claimed in claim 1, wherein each transformation preserves the information properties of the untransformed experimental design.

5. The computational system as claimed in claim 1, wherein for each matching transformed experimental design, simulations are performed to estimate the performance characteristics of each matching design;

wherein the survey constraints comprise one or more criteria for selecting a survey experimental design; and

wherein the survey experimental design is selected based on the matching transformed design with the optimum performance characteristic according to the received criteria.

6. The computational system as claimed in claim 1, wherein the online choice model survey module further comprises an adaptive survey module for performing the survey in one or more survey phases, each phase comprising:

generating a survey experimental design using the current signature;

collecting a predefined number of samples;

generating a model using the collected responses during the current phase;

storing significant model coefficients using a predefined significance cutoff; and

modifying the plurality of attributes and associated plurality of attribute levels based upon significant model coefficients identified and updating the current signature based upon the modified plurality of attributes and associated plurality of attribute levels.

7. The computational system as claimed in claim 1, wherein model generation module uses a Generalized Extreme Value (GEV) logit Model.

8. A computational system for performing an online choice model, the system comprising:

at least one processor;

at least one memory operatively coupled to the at least one processor; and

a plurality of modules, each of the modules comprising instructions for execution by the at least one processor, the plurality of modules comprising;

a problem definition module comprising a problem definition user interface for receiving a plurality of attributes from a user wherein each attribute has an associated plurality of attribute levels;

an online choice model survey module comprising;

an experimental design generator module for generating a survey experimental design and an associated plurality of treatments;

comprising a library of experimental designs;

wherein the experimental design generator module determines the signature of the attribute space from the received plurality of attributes and associated attribute levels; and

for one or more experimental designs in the library of experimental designs, performs one or more transformations until the signature of the transformed experimental design matches the signature of the attribute space to obtain one or more matching transformed experimental designs;

wherein each transformation preserves the information properties of the untransformed experimental design;

wherein the experimental design generator module selects the survey experimental design from the one or more matching transformed experimental designs; and

wherein the experimental design generator module obtains a set of treatments from the selected survey experimental design;

an online survey assembly module which receives the plurality of treatments and assembles an online survey from one or more survey templates pages, and a plurality of treatment representations created using the plurality of treatments received from the experimental design module;

a data collection and sampling module for conducting the assembled online survey;

wherein the data collection and sampling module allocates treatments to survey respondents and collects responses using the assembled online survey;

a model generation module for receiving the data collected by the data collection and sampling module and building a model to obtain a plurality of model parameter estimates and errors from which a utility estimate can be obtained for each attribute level; and

a model explorer module comprising a model explorer user interface for allowing the user to enter one or more attribute levels and obtain a model prediction of the expected utility;

wherein the data collection and sampling module comprises a treatment allocation module, the treatment allocation module comprising;

an allocation frequency counter for counting the allocation of each treatment in the set of treatments, wherein the allocation frequency counter is initialized to be zero;

a receiver for receiving a request for a treatment to be provided to a survey respondent; and

an allocator for allocating a treatment to a survey respondent from the set of treatments;

wherein the allocated treatment is selected from the subset of treatments which have not previously been allocated to the survey respondent and whose allocation frequency differs by no more than a predetermined amount from the most allocated treatment in the set of treatments;

the computational system further comprising;

a length N global binary Deck Vector D which is initialized as a string of N ones indicating that all treatments are available where N is the number of treatments;

respondent allocation vector R of length N for each respondent for indicating whether a given treatments has been delivered to the respondent and is initialized as string of N ones;

a treatment for a respondent is selected by performing a binary AND between the Deck Vector D and respondent allocation vector R to obtain an availability vector A in which a one represents an available treatment,

the respondent is allocated a treatment by randomly selecting one of the positions in the availability vector having a value of one; and

then updating the global binary Deck Vector D and the respondent vector R by changing the state of the allocated position to a zero in both vectors.

9. The computational system as claimed in claim 8, further comprising an N length overflow vector which is initialized as a string of N ones;

and

wherein if there are no available positions in the availability vector A then;

a binary AND is performed between the respondent vector and the overflow vector to obtain a new availability vector A;

respondent is allocated a treatment by randomly selecting one of the positions in the availability vector having a value of one; and

then the overflow vector and the respondent vector are updated by changing the state of the allocated position to a zero in both vectors.

10. A method for computationally performing an online choice model, the method comprising:

receiving a plurality of attributes from a user wherein each attribute has an associated plurality of attribute levels;

generating a survey experimental design and an associated set of treatments, comprising the steps of;

determining the signature of the attribute space from the received plurality of attributes and associated attribute levels;

selecting one or more experimental designs from a library of experimental designs;

for each selected experimental design;

performing one or more transformations until the signature of the transformed experimental design matches the signature of the attribute space to obtain one or more matching transformed experimental designs;

wherein each transformation preserves the information properties of the untransformed experimental design;

selecting a survey experimental design from the one or more matching transformed experimental designs;

obtaining a set of treatments from the selected survey experimental design;

assembling an online survey, comprising the steps of;

creating a plurality of survey templates pages;

creating a plurality of treatment representations based on the set of treatments associated with the survey experimental design;

assembling the plurality of survey templates pages and plurality of treatment representations to form an online survey;

conducting an online survey, the online survey comprising

allocating each treatment to one or more respondents;

providing a plurality of combinations of treatments to the one or more respondents;

receiving the responses of the one or more respondents;

generating a model based upon the received responses to obtain a plurality of model parameter estimates and errors from which a utility estimate can be obtained for each attribute level; and

providing a model explorer user interface for allowing the user to enter one or more attribute levels and obtain a model prediction of the expected utility;

wherein the one or more transformations comprise one or more of the following group of transformations;

factorial splitting of a factor F into two sub factors A, B where A×

B=F and A or B match at least one unmatched factor in the signature;

factorial expansion of a factor F into a new factor A×

F where A×

F matches at least one unmatched factor in the signature;

factor truncation of a factor F into a new factor F−

A where F−

A matches at least one unmatched factor in the signature;

full factorization by generation of a new factor F where F matches at least one unmatched factor in the signature; and

deleting a factor F when all the other factors in the signature are matched.

11. The method as claimed in claim 10, wherein each transformation preserves the information properties of the untransformed experimental design.

12. The method as claimed in claim 10, further comprising:

receiving one or more criteria for selecting the survey experimental design;

simulating the performance characteristics of each matching transformed experimental design; and

selecting the matching transformed experimental design as the survey experimental design with the optimum performance based upon the received one or more criteria.

13. The method as claimed in claim 10, wherein the online choice model survey is an adaptive survey comprising one or more survey phases, each phase comprising:

generating a survey experimental design using the current plurality of attributes and associated attribute levels;

collecting a predefined number of samples;

generating a model using the collected responses during the current phase;

storing significant model coefficients using a predefined significance cutoff; and

modifying the plurality of attributes and associated plurality of attribute levels based upon significant model coefficients identified and updating the current signature based upon the modified plurality of attributes and associated plurality of attribute levels.

14. The method as claimed in claim 13, wherein the modification of the plurality of attributes and associated plurality of attribute levels comprises removing attribute levels with non-significant model coefficients.

15. The method as claimed in claim 13, wherein the modification of the plurality of attributes and associated plurality of attribute levels comprises removing attribute levels with significant model coefficients.

16. The method as claimed in claim 13, wherein the modification of the plurality of attributes and associated plurality of attribute levels comprises:

removing attribute levels with non-significant model coefficients; and

creating a plurality of cross effects for a plurality of the significant model coefficients.

17. The method as claimed in claim 10, wherein a Generalized Extreme Value (GEV) logit Model is generated.

18. The method as claimed in claim 10, wherein the step of receiving a plurality of attributes from a user further comprises:

providing a user interface for receiving content and layout parameters relating to the online choice model survey from a user;

generating a plurality of survey template pages based on the content and layout parameters received from the user;

wherein at least one of the plurality of survey template pages comprises at least one treatment display zone, each treatment display zone comprising one or more associated layout parameters;

sending the set of treatments and the one or more layout parameters associated with each treatment zone to a content generator for generating a visual representation of a treatment; and

receiving a plurality of treatment representations from the content generator;

wherein in use, an online choice model survey is provided to a survey respondent comprising one or more web pages based upon the one or more survey template pages in which each of the treatment display zones displays one of the received visual representations of a treatment.

19. The method as claimed in claim 10, wherein the step of receiving a plurality of attributes from a user further comprises:

providing an attribute input user interface for receiving a plurality of attributes from a user for use in performing the online choice model, comprising;

one or more attribute zones wherein each attribute zone receives an attribute and a plurality of attribute levels associated with the attribute from the user; and

providing a model explorer user interface further comprises;

an attribute level selection zone for each attribute in the choice model;

wherein each attribute level selection zone allows a user to select one of the associated levels;

a prediction output zone which displays an estimate of the predicted response and variability estimate from the online choice model for the current attribute levels selected in the attribute level selection zone.

20. The method as claimed in claim 19, wherein attribute user interface further comprises a survey constraints zone for receiving one or more survey constraints.

21. The method as claimed in claim 20, wherein the survey constraints comprise a minimum accuracy.

22. The method as claimed in claim 20, wherein the survey constraints comprise a maximum survey size.

23. The method as claimed in claim 20, wherein the survey constraints comprise one or more criteria for selecting the survey experimental design.

24. The method as claimed in claim 19, wherein the attribute user interface allows the user to choose whether to perform the type of online choice model survey from the group consisting of Best/Worst, Paired Comparison or Alternative Specific, and the one or more attribute zones to display is based upon the selected choice model type.

25. The method as claimed in claim 19, wherein the step of creating a plurality of survey template pages is performed using a survey assembly user interface comprises:

a content and layout user interface for receiving content and layout parameters from a user and generating a plurality of survey template pages;

wherein at least one of the plurality of survey template pages comprises at least one treatment display zone;

wherein each treatment display zone comprises one or more associated layout parameters; and

a content generator interface for generating a visual representation of each treatment for use in a treatment display zone.

26. The method as claimed in claim 25, wherein the content generator comprises an uploading interface for digital content representing an attribute level for one or more attribute levels.

27. A method for generating a survey experimental design and an associated set of treatments for use in an online choice model, comprising the steps of:

determining the signature of the attribute space from the received plurality of attributes and associated attribute levels;

selecting one or more experimental designs from a library of experimental designs;

for each selected experimental design;

performing one or more transformations until the signature of the transformed experimental design matches the signature of the attribute space to obtain one or more matching transformed experimental designs;

selecting a survey experimental design from the one or more matching transformed experimental designs; and

obtaining a set of treatments from the selected survey experimental design;

wherein the one or more transformations comprise one or more of the following group of transformations;

factorial splitting of a factor F into two sub factors A, B where A×

B=F and A or B match at least one unmatched factor in the signature;

factorial expansion of a factor F into a new factor A×

F where A×

F matches at least one unmatched factor in the signature;

factor truncation of a factor F into a new factor F−

A where F−

A matches at least one unmatched factor in the signature;

full factorization by generation of a new factor F where F matches at least one unmatched factor in the signature; and

deleting a factor F when all the other factors in the signature are matched.

28. The method as claimed in claim 27, wherein each transformation preserves the information properties of the untransformed experimental design.

29. The method as claimed in claim 27, further comprising:

receiving one or more criteria for selecting the survey experimental design;

simulating the performance characteristics of each matching transformed experimental design; and

selecting the matching transformed experimental design as the survey experimental design with the optimum performance based upon the received one or more criteria.

30. The method as claimed in claim 29, wherein the simulations are performed for a range of sample sizes.

31. A method for allocating a treatment to a survey respondent from a set of treatments in an online choice model survey, comprising the steps of:

receiving a set of treatments for use in an online choice model survey;

initializing an allocation frequency counter for each treatment in the set of treatments;

receiving a request for a treatment to be provided to a survey respondent;

creating a length N global binary Deck Vector D and initializing as a string of N ones indicating that all treatments are available where N is the number of treatments;

creating a respondent allocation vector R of length N for each respondent for indicating whether a given treatments has been delivered to the respondent and initializing each respondent vector as string of N ones; and

allocating a treatment to a survey respondent from the set of treatments;

wherein the allocated treatment is selected from the subset of treatments which have not previously been allocated to the survey respondent and whose allocation frequency differs by no more than a predefined maximum difference amount from the most allocated treatment in the set of treatments;

wherein allocating a treatment to a survey respondent from the set of treatments comprises;

generating an availability vector A by performing a binary AND between the global binary Deck Vector D and respondent allocation vector R to obtain an availability vector A in which a one represents an available treatment;

randomly selecting one of the available positions in the availability vector A;

allocating the selected treatment to the respondent; and

updating the global binary Deck Vector D and the respondent vector R by changing the state of the allocated position to a zero in both vectors.

32. The method as claimed in claim 31, wherein the predefined maximum difference amount is 1.

33. A method for allocating a treatment to a survey respondent from a set of treatments in an online choice model survey, comprising the steps of:

receiving a set of treatments for use in an online choice model survey;

initializing an allocation frequency counter for each treatment in the set of treatments;

receiving a request for a treatment to be provided to a survey respondent; and

allocating a treatment to a survey respondent from the set of treatments;

wherein the allocated treatment is selected from the subset of treatments which have not previously been allocated to the survey respondent and whose allocation frequency differs by no more than a predefined maximum difference amount from the most allocated treatment in the set of treatments;

wherein allocating a treatment to a survey respondent from the set of treatments comprises;

creating an N length overflow vector O which is initialized as a string of N ones;

creating a new availability vector if there are no available positions in the availability vector A by performing a binary AND between the respondent vector and the overflow vector;

randomly selecting one of the available positions in the new availability vector A;

allocating the selected treatment to the respondent; and

updating the overflow vector O and the respondent vector R by changing the state of the allocated position to a zero in both vectors.

34. A non-transitory computer readable medium, comprising computer executable instructions for causing a processor to perform a method comprising:

receiving a plurality of attributes from a user wherein each attribute has an associated plurality of attribute levels;

generating a survey experimental design and an associated set of treatments, comprising the steps of;

determining the signature of the attribute space from the received plurality of attributes and associated attribute levels;

selecting one or more experimental designs from a library of experimental designs;

for each selected experimental design;

performing one or more transformations until the signature of the transformed experimental design matches the signature of the attribute space to obtain one or more matching transformed experimental designs;

selecting a survey experimental design from the one or more matching transformed experimental designs;

obtaining a set of treatments from the selected survey experimental design;

assembling an online survey, comprising the steps of;

creating a plurality of survey templates pages;

creating a plurality of treatment representations based on the set of treatments associated with the survey experimental design;

assembling the plurality of survey templates pages and plurality of treatment representations to form an online survey;

conducting an online survey, the online survey comprising

allocating each treatment to one or more respondents;

providing a plurality of combinations of treatments to the one or more respondents;

receiving the responses of the one or more respondents;

generating a model based upon the received responses to obtain a plurality of model parameter estimates and errors from which a utility estimate can be obtained for each attribute level; and

providing a model explorer user interface for allowing the user to enter one or more attribute levels and obtain a model prediction of the expected utility,

wherein the one or more transformations comprise one or more of the following group of transformations;

factorial splitting of a factor F into two sub factors A, B where A×

B=F and A or B match at least one unmatched factor in the signature;

factorial expansion of a factor F into a new factor A×

F where A×

F matches at least one unmatched factor in the signature;

factor truncation of a factor F into a new factor F−

A where F−

A matches at least one unmatched factor in the signature;

full factorization by generation of a new factor F where F matches at least one unmatched factor in the signature; and

deleting a factor F when all the other factors in the signature are matched.