Method for estimating respondent rank order of a set of stimuli

• US 20050060222A1
• Filed: 09/15/2004
• Published: 03/17/2005
• Est. Priority Date: 09/17/2003
• Status: Active Grant

• Current Assignee: William Donald White
• Original Assignee: Mentor Marketing, LLC
• Inventors: White, W. Donald
• Granted Patent: US 7,191,144 B2
• Time in Patent Office: Days
• US Class Current: 705/10
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0203 Market surveys; Market polls

Abstract

A computer implemented method to estimate a respondent'"'"'s or a set of respondents'"'"' rank order with respect to a number of attribute options (i.e. alternative prices, features, benefits, ways of expressing benefits, brand names, slogans, logos, graphic treatments such as package design elements and any other stimuli that the researcher believes may influence the individual'"'"'s opinion such as a statement, graphic or brief audio). The method can be used when the number of stimuli to be presented exceeds the number that can be presented at one time to a respondent, most typically, exceeding the number that can be legibly displayed on one computer screen for rank ordering by a respondent. The estimated rank order is highly accurate, especially for the highest ranked stimuli, but requires only a small fraction of the respondent input that would be needed to capture a complete rank order from that respondent. This abstract is provided as an aid for finding relevant disclosures and not as a limitation on the scope of the claims.

Claims

1. A method for creating an approximation of a respondent'"'"'s rank order preferences for at least a portion of a set of N stimuli where N is a positive integer equal to R multiplied by S, the method comprising:

A) presenting to a respondent a subset of R stimuli selected from the set of N stimuli;

B) receiving from the respondent, a set of rank order preferences for the presented subset of R stimuli;

C) repeating A and B until all N stimuli have been presented in S subsets of R stimuli and S sets of rank order preferences have been received for the sets of R stimuli;

D) presenting a set H comprised of the highest ranked stimuli from each of the previously presented S subsets;

E) receiving from the respondent, a set of rank order preferences for the presented set of stimuli in set H;

F) associating each of the rank order preferences for the presented set of stimuli from set H with a value representative of an estimated overall rank order ranking for a stimuli receiving a particular ranking in set H for a set of N stimuli presented in S subsets of R stimuli;

G) presenting a set T comprised of the third highest ranked stimuli from each of the previously presented S subsets;

H) receiving from the respondent, a set of rank order preferences for the presented set of stimuli in set T;

I) associating each of the rank order preferences for the presented set of stimuli from set T with a value representative of an estimated overall rank order ranking for a stimuli receiving a particular ranking in set T for a set of N stimuli presented in S subsets of R stimuli;

J) using the values associated in steps F and I with the highest and third highest ranked values of each of the original subsets to estimate the overall rank order ranking for any stimuli originally ranked second highest in each of the subsets S for second ranked stimuli lacking an estimated overall rank order ranking; and

K) ordering the stimuli ranked first, second, or third, in the each of the original subsets S by estimated overall rank order ranking.

2. The method of claim 1 further comprising the steps of:

J2) in addition to presenting the set H of the highest ranked stimuli, and the set T of the third highest ranked stimuli, present additional sets comprised of similarly ranked values and receive from the respondent additional sets of rank order preferences for the presented set of stimuli in each set until receiving a set of sets of rank order rankings for stimuli initially ranked 1, 3, 5, up to Rth in the original subsets of R stimuli;

J3) in addition to associating each of the rank order preferences for the presented set of stimuli from sets H and T with a value representative of an estimated overall rank order ranking for a stimuli receiving a particular ranking in that set for a set of N stimuli presented in S subsets of R stimuli, associating each of the rank order preferences for the presented set of stimuli from the remaining sets with a value representative of an estimated overall rank order ranking for a stimuli receiving a particular ranking in that set for a set of N stimuli presented in S subsets of R stimuli; and

J4) in addition to using the values associated in steps F and I with the highest and third highest ranked values of each of the original subsets to estimate the overall rank order ranking for each of the stimuli originally ranked second highest in each of the subsets S;

estimate the overall rank order of other stimuli given an even number rank order preference when the respondent viewed the initial set of S subsets through use of the overall rank order values associated with the next higher and next lower ranked stimuli from the original subset;

wherein the step of ordering the stimuli ranked first, second, or third, in the each of the original subsets S by estimated overall rank order ranking further comprises ordering all stimuli by the estimated overall rank order ranking to create an estimated overall rank order ranking of N stimuli.

3. The method of claim 1 further comprising associating an average overall rank order ranking to each of the lowest ranked stimuli for use in estimating the overall rank order ranking of each of the second lowest ranked stimuli from the original S subsets of stimuli based on the estimated overall rank order ranking of the lowest and a higher ranked stimuli from the original S subsets of stimuli whereby the respondent is not asked to provide a set of rank order preferences for the set of lowest rank stimuli thus reducing requirements for respondent input to create an estimated overall rank order ranking of N stimuli.

4. The method of claim 1 wherein the step associating each of the rank order preferences for the presented set of stimuli from set H with a value representative of an estimated overall rank order ranking for a stimuli receiving a particular ranking in set H for a set of N stimuli presented in S subsets of R stimuli uses estimated overall rank order rankings derived from Monte Carlo simulations.

5. The method of claim 1 wherein the step associating each of the rank order preferences for the presented set of stimuli from set H with a value representative of an estimated overall rank order ranking for a stimuli receiving a particular ranking in set H for a set of N stimuli presented in S subsets of R stimuli uses estimated overall rank order rankings than include noninteger values.

6. The method of claim 1 wherein S is less than R such that presenting all of the stimuli given the highest rank in the S original subsets presents less than R stimuli to the respondent, the method further characterized by:

set H consists of S stimuli given the highest rank order preferences by the respondent from the original S subsets and (R minus S) other stimuli that were not given the highest rank order preference by the respondent from the original S subsets.

7. The method of claim 1 wherein the approximation of the respondent'"'"'s rank order preferences for at least a portion of N stimuli is to be combined with the approximations of other respondent'"'"'s rank order preferences for at least a portion of N stimuli, the method further characterized by altering the order of the N stimuli so that the set of respondents are not all presented with the stimuli in the same order in the same S subsets.

8. The method of claim 1 wherein the approximation of the respondent'"'"'s rank order preferences for at least a portion of N stimuli is to be combined with the approximations of other respondents'"'"' rank order preferences for at least a portion of N stimuli, the method further characterized by randomizing the order of the N stimuli for each respondent so that the first R stimuli presented in the first subset is not consistent across the set of respondents.

9. The method of claim 1 wherein the steps A and B are further characterized by initially presenting R different stimuli to the respondent and then removing each selected choice as an possible choice for subsequent selection until only two choices remain then recording the next selection as the second lowest ranked choice and recording the unselected value as the lowest ranked choice.

10. The method of claim 1 wherein the steps A and B are further characterized by allowing the respondent to indicate that the remaining unselected choices are unimportant and marking the unselected stimuli to indicate that the respondent deemed them unimportant.

11. The method of claim 1 wherein the set of stimuli ordered by estimated overall rank order ranking for that respondent is not aggregated with a set of stimuli ordered by estimated overall rank order ranking for a second respondent if the second respondent'"'"'s responses fail a screening test for order bias.

12. The method of claim 11 wherein the screening test for order bias fails the second respondent'"'"'s responses when the respondent chooses X different stimuli sharing a same relative position in different presented subsets to be the highest ranked stimulus for that subset and the number X is above an acceptance threshold.

13. The method of claim 11 wherein the screening test for order bias fails the second respondent'"'"'s responses when the second respondent'"'"'s rank order preferences for various presented sets of stimuli indicates a correlation between second respondent'"'"'s sets of rank order preference rank and a relative position of the stimuli within the presented set of stimuli, and the chance of that correlation occurring randomly is less than acceptance threshold Y.

14. The method of claim 1 wherein the stimuli are exclusively audio stimuli and the respondent provides sets of rank order preferences through use of a telephone keypad.

15. A method for creating an approximation of a respondent'"'"'s rank order preferences for at least a portion of a set of N stimuli the method comprising:

A) presenting to a respondent a subset of R stimuli selected from the set of N stimuli;

B) receiving from the respondent, a set of rank order preferences for the presented set of R stimuli;

C) repeating A and B until all N stimuli have been presented in S subsets of R stimuli and S sets of rank order preferences have been received for the sets of R stimuli;

D) presenting a set H comprised of the highest ranked stimuli from each of the previously presented S subsets and at least one lowest ranked stimulus from one of the previously presented S subsets;

E) receiving from the respondent, a set of rank order preferences for the presented set of stimuli in set H; and

F) not aggregating the approximation of the this respondent'"'"'s rank order preferences for at least a portion of a set of N stimuli with the approximations of other respondents'"'"' rank order preferences if the respondent'"'"'s rank order preferences for the set of stimuli in set H places the previously lowest ranked stimuli above a number of previously highest ranked stimuli.

16. The method of claim 15 where the approximation of a respondent'"'"'s rank order preferences for at least a portion of a set of N stimuli is not aggregated when the respondent set of rank order preferences for the presented set of stimuli in set H ranks the previously lowest ranked stimuli above two previously highest ranked stimuli.

17. The method for creating an approximation of a respondent'"'"'s rank order preferences for at least a portion of a set of N stimuli, the method comprising:

A) presenting to a respondent a subset of stimuli selected from the set of N stimuli;

B) receiving from the respondent, a set of rank order preferences for the presented subset of stimuli;

C) repeating A and B until all N stimuli have been presented in a first round of rank ordering in S subsets of stimuli and S sets of rank order preferences have been received for the sets of stimuli;

D) arranging the stimuli for a second round of rank ordering where the arrangement of stimuli into the second round subsets is done based on the sets of rank order preferences received from the respondent in the first round;

E) presenting to the respondent a first set of arranged stimuli and receiving from the respondent a set of rank order preferences for this set of second round subsets of stimuli;

F) presenting to the respondent additional sets of arranged stimuli and receiving from the respondent additional sets of rank order preferences for these additional sets of second round subsets of stimuli;

G) associating with each of at least some of the stimuli a value representing an estimated overall rank order preference rank based at least in part on the rank order preferences provided to a stimuli and its placement into a specific second round subset of stimuli; and

H) creating the approximation of a respondent'"'"'s rank order preferences for at least a portion of a set of N stimuli based on the stimuli'"'"'s estimated overall rank order ranking.

18. The method of claim 17 wherein the respondent can identify a stimulus in one of the first round subsets as unimportant and the full set of unimportant stimuli are not presented in the second round.

19. The method of claim 18 wherein the approximation of the respondent'"'"'s rank order preferences for at least a portion of a set of N stimuli includes all N stimuli and the X stimuli marked as unimportant in the first round are placed at the end of the approximation of the respondent'"'"'s rank order preferences and given a value equal to the average rank value for the lowest X ranks.

20. A method for creating an approximation of a respondent'"'"'s rank order preferences for at least a portion of a set of N stimuli where N is a positive integer equal to R multiplied by S, the method comprising:

A) presenting to a respondent a subset of R stimuli selected from the set of N stimuli;

B) receiving from the respondent, a set of rank order preferences for the presented set of R stimuli;

C) repeating A and B until all N stimuli have been presented in S subsets of R stimuli and S sets of rank order preferences have been received for the sets of R stimuli;

D) presenting a set H comprised of the highest ranked stimuli from each of the previously presented S subsets;

E) receiving from the respondent, a set of rank order preferences for the presented set of stimuli in set H;

F) associating each of the rank order preferences for the presented set of stimuli from set H with a value representative of an estimated overall rank order ranking for a stimuli receiving a particular ranking in set H for a set of N stimuli presented in S subsets of R stimuli;

G) presenting a set HH comprised of the second highest ranked stimuli from each of the previously presented S subsets;

E) receiving from the respondent, a set of rank order preferences for the presented set of stimuli in set HH;

F) associating each of the rank order preferences for the presented set of stimuli from set HH with a value representative of an estimated overall rank order ranking for a stimuli receiving a particular ranking in set HH for a set of N stimuli presented in S subsets of R stimuli;

G) presenting a set F comprised of the fourth highest ranked stimuli from each of the previously presented S subsets;

H) receiving from the respondent, a set of rank order preferences for the presented set of stimuli in set F;

I) associating each of the rank order preferences for the presented set of stimuli from set F with a value representative of an estimated overall rank order ranking for a stimuli receiving a particular ranking in set F for a set of N stimuli presented in S subsets of R stimuli;

J) using the values associated in steps F and I with the second highest and fourth highest ranked values of each of the original subsets to estimate the overall rank order ranking for any stimulus originally ranked third highest in each of the subsets S for third ranked stimuli lacking an estimated overall rank order ranking; and

K) ordering the stimuli ranked first, second, third, or fourth in the each of the original subsets S by estimated overall rank order ranking.

21. A method of estimating overall rank order ranking for a particular respondent for a stimulus presented to the respondent only once, the method comprising;

A) presenting to a respondent a first subset of R stimuli taken from a set of N stimuli;

B) receiving from the respondent a set of rank order preferences for the first subset of R stimuli including a designation that one stimulus is the Xth highest ranked stimulus in that subset and a designation that another stimulus is the Yth highest ranked stimulus in that subset where there is at least one stimulus in that subset designated by that respondent as ranked higher than the Yth highest ranked stimuli but lower than the Xth highest ranked stimuli;

C) repeating the process of presenting additional subsets of R stimuli to the respondent and receiving the set of rank order preferences including a designation that one stimulus is the Xth highest ranked stimuli in that subset and a designation that another stimulus is the Yth highest ranked stimulus in that subset where there is at least one stimulus in that subset designated by that respondent as ranked higher than the Yth highest ranked stimulus but lower than the Xth highest ranked stimulus;

D) presenting a second round set AA of stimuli comprising the Xth highest ranked stimulus from the first subset and additional stimuli identified by the respondent as the Xth highest ranked stimuli from other subsets and receiving the respondent'"'"'s rank order preferences for this second round set;

E) presenting a second round set BB of stimuli comprising the Yth highest ranked stimulus from the first subset and additional stimuli identified by the respondent as the Yth highest ranked stimulus from other subsets and receiving the respondents rank order preferences for this second round set;

F) associating values with the rank order preferences received in response to the presentations of second round sets AA and BB where the values are estimated overall rank order rankings; and

G) associating with each of the at least one stimulus in the original first set of R stimuli designated by that respondent as ranked higher than the Yth highest ranked stimulus but lower than the Xth highest ranked stimulus with a value selected based on the value associated with the Xth highest ranked stimulus in the original first set of R stimuli and the value associated with the Yth highest ranked stimulus in the original first subset of R stimuli.

22. The method of claim 21 wherein:

the received set of rank order preferences for the first set of N stimuli comprises the Xth highest ranked stimulus, the X+1highest ranked stimulus, the X+2 highest ranked stimulus and the Yth highest ranked stimulus where X is higher than X+1 which is higher than X+2 which is higher than Y; and

the value associated with the X+1 highest ranked stimulus from the first set of N stimuli is equal to the value associated with the Xth highest ranked stimulus plus ⅓

^rd of the difference between the value associated with the Xth highest ranked stimulus and the value associated with the Yth highest ranked stimulus.

23. The method of claim 21 wherein

the lowest ranked stimulus in the received rank order preferences for the first set of N stimuli is associated with a value that is the average estimated overall order ranking for a set of stimuli receiving the lowest rankings from the respondent when initially presented with a series of subsets of R stimuli taken from the set of N stimuli.

24. A method for preparing an approximation of a respondent'"'"'s rank order preference for a set of stimuli, the method comprising:

A) identifying a motivational question;

B) identifying a list of factors relevant to the motivational question and representing the factors as a set of N stimuli;

C) identifying R, which is a maximum number of stimuli that can be presented to a respondent for obtaining the respondent'"'"'s rank order preference of those R stimuli;

D) selecting a first round presentation sequence consisting of a number of subsets of not more than R stimuli to be presented to the respondent for obtaining the respondent'"'"'s rank order preference for the stimuli in each subset;

E) selecting a second round presentation sequence consisting of a number of second round subsets of not more than R stimuli to be presented to the respondent for obtaining the respondent'"'"'s rank order preference for the stimuli in that second round subset;

the step of selecting a second round presentation sequence including the establishment of a set of selection criteria for qualifying a particular stimulus to become a member of a particular second round set based on the rank given to that stimulus by the respondent in the first round;

F) establishing a set of values based on estimated overall rank order rankings for each rank position obtained from the respondent'"'"'s second round set of rank order preferences;

G) obtaining the respondents set of rank order preference responses to the set of S subsets of the N stimuli;

H) obtaining the respondent'"'"'s set of rank order preferences to the set of second round sets of stimuli; and

I) associating values for the estimated overall rank order rankings with the stimuli ranked in the second round subsets.

25. The method of claim 24 wherein:

some stimuli are not presented as members of the second round sets;

a first round only stimuli not presented as a member of any of the second round sets is given an estimated overall rank order ranking based on the values given to a high stimulus and a low stimulus where both the high and the low stimuli were presented in second round sets and both the high and low stimuli were presented in the same first round subset as this particular first round only stimuli.

26. The method of claim 24 wherein:

some stimuli are not presented as members of the second round subsets;

a low ranking first round only stimulus not presented as a member of any second round set is given an estimated overall rank order ranking based on the low ranking received from the respondent in the first round.

27. The method of claim 24 wherein the responses from the respondent must pass a consistency test in order for the approximation of the respondent'"'"'s rank order preferences to be aggregated with a set of approximations of other respondents'"'"' rank order preferences as part of an effort to produce an aggregated approximation of the rank order preferences for a set of respondents.

28. The method of claim 24 wherein the responses from the respondent must pass an order bias test in order for the approximation of the respondent'"'"'s rank order preferences to be aggregated with a set of approximations of other respondents'"'"' rank order preferences as part of an effort to produce an aggregated approximation of the rank order preferences for a set of respondents.

29. The method of claim 24 further characterized by:

identifying a factor that is best tested by monadic testing;

selecting a set of at least two different stimuli instances to be used to measure the relevance of that identified factor to the motivational question;

presenting a first stimulus instance but not a second stimulus instance as part of the N stimuli presented to a first set of respondents;

presenting the second stimulus instance but not the first stimulus instance as part of the N stimuli presented to a second set of respondents; and

comparing the aggregated estimated overall rank order ranking of the first stimulus instance by the first set of respondents to the aggregated estimated overall rank order ranking of the second stimulus instance by the second set of respondents to obtain an indication of the sensitivity of respondents to changes in that particular factor from the first stimulus instance to the second stimulus instance.

30. The method of claim 29 wherein the presentation of the first stimulus instance and the second stimulus instance in the first round of subsets is controlled so that these stimuli are always presented in the same relative position within the same first round subset to minimize any impact from order bias.

31. The method of claim 29 where the factor is sales price and the first stimulus instance is a first price and the second stimulus instance is a second price different from the first price so that testing obtains an indication of the sensitivity of the respondents to changes in sales price from the first price to the second price.

32. The method of claim 24 further comprising establishing of a threshold value for screening rank order bias;

and

aggregating the approximation of respondent'"'"'s rank order preferences with approximations for other respondents'"'"' rank order preferences if the obtained sets of respondent'"'"'s rank order preferences passes a set of at least one screening test, but not aggregating if a numeric value representing an analysis of the obtained sets of respondent'"'"'s rank order preferences is on the wrong side of the threshold value.