CONFIDENCE BASED SELECTION FOR SURVEY SAMPLING

• US 20060121434A1
• Filed: 12/03/2004
• Published: 06/08/2006
• Est. Priority Date: 12/03/2004
• Status: Active Grant

• Current Assignee: RYMA Technology Solutions Incorporated (OneDesk, Inc.)
• Original Assignee: Orasi Software, Inc.
• Inventors: Azar, James R.
• Granted Patent: US 8,121,886 B2
• Time in Patent Office: Days
• US Class Current: 434/350
• CPC Class Codes: G06Q 30/0203 Market surveys; Market polls
• CPC Class Codes: G06Q 30/0282 Rating or review of business operators or products
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A system and method for confidence-based selection of items for use in conducting a computer-implemented survey. The survey presents information about a selected plurality of items to a survey participant, to elicit survey feedback information. Information regarding the plurality of items is stored, the stored information including display information about each of the plurality of items for presentation to a survey participant. A subset of items for presentation to a survey participant is selected in accordance with a predetermined selection algorithm. Information corresponding to the selected subset of items is displayed to the survey participant via a survey user interface. Rating information is input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the presented subset of items. The rating information is utilized in various manners to affect the selection algorithm for a subsequent survey.

Claims

1. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant and presentation number information corresponding to the number of times a particular item has been previously shown to survey participants;

selecting a subset of items for presenting to a survey participant in accordance with a predetermined selection algorithm that utilizes the presentation number information to influence the selection of items for the subset;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface; and

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented.

2. The method of claim 1, further comprising the step of utilizing the rating information input by the survey participant to affect the probability of selection of the items in the selected subset in a subsequent selection of a subset of items for presentation to a subsequent survey participant using the predetermined selection algorithm.

3. The method of claim 1, wherein the predetermined selection algorithm utilizes an adjustment factor to cause items to be selected more or less often as a function of the rating information obtained by previous participants of a survey.

4. The method of claim 1, wherein the predetermined selection algorithm utilizes, at least in part, a random number selection of items in the plurality of items.

5. The method of claim 4, wherein the plurality of items includes a predetermined duplication of items in a set of the plurality of items, with the number of duplications of particular items influenced by the rating information.

6. The method of claim 1, wherein the predetermined selection algorithm is self-adjusting based on previous responses received during a previous survey of the plurality of items.

7. The method of claim 6, wherein the self-adjusting is based on modifying the probability of selecting an item from the plurality of items, the modifying in turn based collection of new items input by survey participants and rating of new items in comparison of previously included items.

8. The method of claim 1, wherein the predetermined selection algorithm is operative initially to randomly select items for the subset of items, and thereafter operative to select items based on utilization of the rating information.

9. The method of claim 8, wherein the probability of selection of a given item is continuously adjusted as items are rated by survey participants.

10. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant and frequency information corresponding to the number of times a particular item has been previously shown to survey participants;

selecting a subset of items for presenting to a survey participant in accordance with a predetermined selection algorithm that utilizes the frequency information;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented; and

utilizing the rating information input by the survey participant to affect the probability of selection of the items in the selected subset in a subsequent selection of a subset of items for presentation to a subsequent survey participant using the predetermined selection algorithm.

12. The method of claim 10, wherein the step of selecting a subset of items in accordance with the predetermined selection algorithm comprises selecting based on a ranking of items using rating information from previous participants, such that the probability of selection of particular items for presenting in a subsequent survey is influenced by the rating information, whereby certain items that have been rated lower than other items are more likely to be selected for a survey so as to increase the number of presentations of such items.

13. The method of claim 10, further comprising the step of providing the rating information for each item as an output of the method indicative of survey results.

14. The method of claim 10, wherein the memory is a random access memory array.

15. The method of claim 10, wherein information regarding the plurality of items is stored in an ordered array and selected according to a probability index.

16. The method of claim 15, wherein the information about each item in the ordered array is stored in a data field in the ordered array.

17. The method of claim 10, wherein the subset of items selected for presentation to the survey participant is an initial subset, and wherein the subsequent selection of a set of items for presentation comprises a selection from the plurality of items that may include one or more of the items from the initial subset.

18. The method of claim 10, wherein the information regarding the plurality of items includes a unique identifier for each item for use as a primary key to access the item in the memory.

19. The method of claim 10, further comprising the step of storing a users item table for storing information provided by a survey participant relating to an additional item for inclusion in the plurality of items.

20. The method of claim 10, wherein the information regarding the plurality of items includes status information about each item.

21. The method of claim 20, wherein the status information is indicative whether an item has been previously shown to a survey participant or not.

22. The method of claim 21, wherein the predetermined selection algorithm also utilizes the status information in conjunction with the frequency information.

23. The method of claim 10, wherein the frequency information is stored in a times-shown field for each item in the array.

24. The method of claim 10, wherein the predetermined selection algorithm utilizes a selection score in selecting items for presentation.

25. The method of claim 24, wherein the selection score is based upon a confidence score.

26. The method of claim 25, wherein the selection score is further based on an adjusted mean rating determined from the rating information.

27. The method of claim 25, wherein the selection score is further based on a rank influence factor.

28. The method of claim 27, wherein the rank influence factor is an arbitrary number used to adjust the probability of an item being selected based on ranking information.

29. The method of claim 28, wherein the ranking information comprises information corresponding to the actual ranking of an item in relation to other items in the plurality of items.

30. The method of claim 10, wherein the selection algorithm selects an item from the plurality of items based on a computation of a probability index.

31. The method of claim 30, wherein the probability index is determined based on a normalized selection score.

32. The method of claim 31, wherein the normalized selection score is utilized to determine a probability of selection for each item in the plurality of items, the probability of selection of each item is used to determine how many times an item is represented in the plurality of items for selection.

33. The method of claim 10, wherein the plurality of items are represented in a computer system as a pool of selectable items stored in an array of items, with each item in the pool having a high index number and a low index number, with the index numbers representing how many times an item is represented in the pool of selectable items, and wherein the step of selecting a subset of items comprises selecting from the pool of items based on a random number used to index into the array of index numbers.

34. The method of claim 33, wherein the subset of items for presenting to a survey participant is selected by repeating the step of selecting utilizing the random number, until a predetermined number of items corresponding to the size of the selected subset of items has been chosen for presentation.

35. The method of claim 10, wherein a selected subset of items comprises a unique sample of items in the plurality of items of a predetermined sample size that meets predetermined selection criteria according to status information associated with the item.

36. The method of claim 35, wherein the predetermined sample size comprises the maximum number of items presented to a survey participant in the survey.

37. The method of claim 35, wherein the status information comprises information indicative of a condition associated with an item.

38. The method of claim 10, wherein the status information is selected from the group comprising:

whether an item is scheduled, whether an item has been approved, whether an item is implemented, whether an item is active, whether an item is in or under review, whether an item has been submitted, whether an item has been declined, or other equivalent information indicative of a condition of an item.

39. The method of claim 10, wherein the selected subset of items is selected for presentation based at least in part on an indication of interest of a participant.

40. The method of claim 39, wherein the indication of interest of a participant is obtained by input of interest information by a survey participant in response to a query prior to selection of the subset.

41. The method of claim 39, wherein the indication of interest of a participant is obtained by examining items previously submitted by the survey participant, and by selecting other items from the plurality of items based on the topical similarity of other items in the plurality of items.

42. The method of claim 39, wherein the indication of interest of a participant is obtained by executing a query of keywords relating to items submitted by the survey participant.

43. The method of claim 10, wherein the selected subset of items for presentation is a first selected subset of a predetermined small number of items, where “

small”

is relative but determined based on a number that is deemed by a survey manager to be acceptable for purposes of a particular survey, and further comprising the step of selecting additional items for presentation to a survey participant.

44. The method of claim 43, wherein the step of selecting additional items for presentation to a survey participant is based on information provided by a survey participant indicating a desire to view and rate more items.

45. The method of claim 44, wherein the survey participant is provided with a display offering an opportunity to request an additional sample of items for rating, and wherein the information provided by the survey participant indicating a desire to view and rate more items is input by the survey user interface.

46. The method of claim 45, wherein the opportunity to request an additional sample of items for rating is providing during a survey session.

11. The method of claim A1, wherein the rating information is selected from the group comprising:

ranking of items relative to each other, ranking of the items on a scale, grading the items, ordering of the items, allocating points among items, scaling the items, choosing an item over other items, categorizing items, and other equivalent methods of indicating a preference of one item over another.

47. A method for conducting a computer-implemented survey of a plurality of items, comprising the steps of:

arranging the plurality of items in a memory in ordered array;

providing a unique identifier for each item in the array;

providing information about each item in the array for presentation to a survey participant;

providing a status information field for each item in the array;

providing a times-shown field for each item in the array;

selecting a subset of items for presenting to a survey participant in accordance with a predetermined selection algorithm that utilizes the information in the times-shown field of the items;

presenting information corresponding to the selected subset of items to the survey participant via a survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to the subset of items presented;

utilizing the rating information input by the survey participant to affect the probability of selection of the items in the selected subset of times for a subsequent selection of a set of items for presentation to a subsequent survey participant using the predetermined selection algorithm.

48. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

receiving rating information input by a particular subset of a plurality of survey participants via a survey user interface indicating the survey participants'"'"' preferences as to items in a presented subset of items of the plurality of items;

computing the mean of the rating information;

computing the standard error of the mean of the rating information;

determining a confidence score utilizing the standard error of the rating information; and

utilizing the confidence score to select a different subset of the plurality of items for presentation to a different subset of the plurality of survey participants,

whereby items that might benefit from additional ratings by additional participants are selected for presentation in a subsequent survey.

49. A method for dynamically selecting a subset of items from a plurality of items for presentation to a survey participant in a computer-implemented survey relating to the plurality of items, comprising the steps of:

storing rating information input by a plurality of prior survey participants indicating such prior survey participants'"'"' preferences as to items in one or more subsets of items presented to such survey participants in a prior survey;

determining the number of participants that have previously rated particular items in the plurality of items;

determining a measure of agreement by the determined number of participants on the stored rating information of the particular previously rated items in the plurality of items;

utilizing the measure of agreement on the ratings of such previously rated particular items to adjust the probability of selection of such previously rated particular items for a subsequent selection; and

selecting a subset of items for presentation to the survey participant as a function of the adjusted probability of selection.

50. The method of claim 49, wherein the measure of agreement comprises the standard deviation of the mean ratings provided by the prior survey participants.

51. The method of claim 49, wherein the measure of agreement comprises the average of the mean ratings provided by the prior survey participants.

52. The method of claim EE1, further comprising the step of continuously changing the items in the selected subset of items based on the determined measure of agreement on the stored rating information on previously rated items.

53. A method for dynamically selecting a subset of items from a plurality of items for presentation to a survey participant in a computer-implemented survey relating to the plurality of items, comprising the steps of:

storing rating information input by a plurality of prior survey participants indicating such prior survey participants'"'"' preferences as to items in one or more subsets of items presented to such survey participants in a prior survey;

determining the number of participants that have previously rated particular items in the plurality of items;

determining a measure of agreement by the determined number of participants on the stored rating information of the particular previously rated items in the plurality of items;

continuously adjusting the probability of selection of such previously rated particular items for a subsequent selection based on the determined measure of agreement on the stored rating information on previously rated items;

selecting a subset of items for presentation to the survey participant as a function of the adjusted probability of selection.

54. The method of claim 53, wherein the measure of agreement comprises the standard deviation of the mean ratings provided by the prior survey participants.

55. The method of claim 53, wherein the measure of agreement comprises the average of the mean ratings provided by the prior survey participants.

56. The method of claim 53, further comprising the step of continuously changing the items in the selected subset of items based on the determined measure of agreement on the stored rating information on previously rated items.

57. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant;

selecting a subset of items for presenting to a survey participant in accordance with a predetermined selection algorithm;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented;

receiving information input by the survey participant corresponding to an additional item for inclusion in the plurality of items; and

selecting a second subset of items from the plurality of items that now includes the additional item for presenting to a subsequent survey participant in accordance with the predetermined selection algorithm.

58. The method of claim 57, further comprising the step of utilizing the rating information input by the survey participant to affect the probability of selection of the items in a subsequent selection of a subset of items for presentation to a subsequent survey participant.

59. The method of claim 57, wherein the information input by the survey participant is provided via an additional item submission user interface.

60. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant;

conducting a selection operation involving selecting a subset of items for presenting to a survey participant in accordance with a function that utilizes a probability of selection;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented;

determining an adjustment factor for the probability of selection of items in the subset of items as a function of the number of times that the items have already been selected and presented to previous survey participants; and

applying the adjustment factor to the probability of selection for a subsequent selection operation.

61. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant;

conducting a selection operation involving selecting a subset of items for presenting to a survey participant in accordance with a function that utilizes a probability of selection;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented;

determining an adjustment factor for the probability of selection of items in the subset of items as a function of the rating information on items that have already been selected and presented to previous survey participants; and

applying the adjustment factor to the probability of selection for a subsequent selection operation.

62. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant;

conducting a selection operation involving selecting a subset of items for presenting to a survey participant in accordance with a function that utilizes a probability of selection, the selection operation initially operative to select a subset of items on random basis;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented;

determining an adjustment factor for the probability of selection of items in the subset of items as a function of the rating information on items that have already been selected and presented to previous survey participants and on the number of survey participants that have rated particular items; and

applying the adjustment factor to the probability of selection for a subsequent selection operation,

whereby the probability of each item being selected is continuously adjusted to be less random and more biased toward selection of unrated items and/or infrequently viewed items as items are rated by survey participants and rating information on particular items is collected.

63. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant and predefined attribute information relating to predetermined attributes of each of the plurality of items;

conducting a selection operation involving selecting a subset of items for presenting to a survey participant in accordance with a function that utilizes the predefined attribute information;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface; and

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented.

64. The method of claim 63, wherein the selection operation is further a function of probability of selection of the plurality of items as well as the attribute information, and further comprising the steps of:

determining an adjustment factor for the probability of selection of items in the subset of items as a function of the rating information on items that have already been selected and presented to previous survey participants and on the number of survey participants that have rated particular items; and

applying the adjustment factor to the probability of selection for a subsequent selection operation,

whereby the probability of each item being selected is continuously adjusted to be less random and more biased toward selection of unrated items and/or infrequently viewed items as items are rated by survey participants and rating information on particular items is collected.

65. The method of claim 63, wherein the attribute information comprises one or more of the following:

a category, originator, priority, purpose.

66. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

(a) storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant;

(b) conducting a selection operation involving selecting a subset of items for presenting to a survey participant in accordance with a predetermined function;

(c) presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface; and

(d) receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented;

(e) receiving an indication input by a survey participant of willingness to view and rate additional items; and

(f) in response to receipt of the indication input by a survey participant of willingness to view and rate additional items, conducting a subsequent selection operation (b) and repeating the steps (c) through (f).

67. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant;

conducting a selection operation comprising selecting a subset of items for presenting to a survey participant in accordance with a predetermined selection algorithm;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented;

determining that particular item in the plurality of items should be presented more frequently so as to obtain additional rating data; and

adjusting a parameter of the predetermined selection algorithm so as to increase the likelihood that the particular item will be selected during a subsequent selection operation for a subsequent survey.

68. The method of claim 67, wherein the step of adjusting a parameter of the selection algorithm comprise computing a confidence score among two or more items in the plurality of items, comparing the confidence scores, and using the results of the comparison to adjust the probability of selection of a particular item for which additional data is needed.

69. The method of claim 67, wherein the parameter of the selection algorithm is adjusted as a function of the number of survey participants that have previously rated the particular item.

70. The method of claim 67, wherein the parameter of the selection algorithm is adjusted as a function of the ratings of the particular item by survey participants that have previously rated the particular item.

71. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant;

conducting a selection operation comprising selecting a subset of items for presenting to a survey participant in accordance with a predetermined selection algorithm;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented;

determining that a particular item in the plurality of items is relatively newer item than other items in the plurality of items;

adjusting the predetermined selection algorithm so as to increase the likelihood that the relatively newer item will be selected during a subsequent selection operation for a subsequent survey,

whereby the relatively newer item will be presented more frequently so as to obtain additional survey data for such newer item.

72. The method of claim 71, further comprising the step of receiving new item information input by a survey participant corresponding to the submission of a new item for inclusion in the plurality of items for the survey, such that the new item is the relatively newer item.

73. The method of claim 71, wherein the relatively newer item is determined according to the time of inclusion of the item in the plurality of items, compared with other items.

74. The method of claim 71, wherein the relatively newer item is determined according to the number of times that the item has been presented in prior surveys.

75. The method of claim 71, wherein the relatively newer item is determined according to the frequency that the item has been presented in prior surveys.

76. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants, comprising the steps of:

storing information regarding the plurality of items in a memory, the stored information including display information about each of the plurality of items for presentation to a survey participant and frequency information indicating a number of times that items in the plurality of items have been presented in a survey;

biasing the plurality of items as a function of the frequency information in anticipation of a selection operation;

conducting a probabilistic selection operation comprising a random selection within the plurality of items to select a subset of items for presenting to a survey participant;

presenting the display information corresponding to the selected subset of items to the survey participant via a survey user interface; and

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented,

whereby a biased random selection of items to be presented is conducted so as to avoid presenting only newer, less frequently presented items to later survey participants and to ensure presentation of some early ideas to such later survey participants.

77. A method for conducting a computer-implemented survey relating to a plurality of items from a plurality of survey participants so as to favor selection of items with a higher mean rating, comprising the steps of:

storing information regarding the plurality of items, the stored information including display information about each of the plurality of items for presentation to a survey participant;

determining a mean rating of items previously presented to survey participants;

biasing the plurality of items as a function of the mean rating of items from the mean rating determining step in anticipation of a selection operation;

conducting a selection operation involving selecting a subset of items from the plurality of items for presenting to a survey participant in accordance with a predetermined function that utilizes a probability of selection;

presenting the display information corresponding to the selected subset of items to a survey participant via a survey user interface; and

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented,

whereby a biased random selection of items to be presented is conducted so as to favor selection of items with a higher mean rating.

78. The method of claim 77, wherein the biasing step involves application of a rank influence factor (rif) that may be adjusted to increase or decrease the probability of an item being selected based on the ranking of one item in relation to another item.