Confidence based selection for survey sampling

• US 8,121,886 B2
• Filed: 12/03/2004
• Issued: 02/21/2012
• Est. Priority Date: 12/03/2004
• Status: Expired due to Fees

• Current Assignee: RYMA Technology Solutions Incorporated (OneDesk, Inc.)
• Original Assignee: RYMA Technology Solutions Incorporated (OneDesk, Inc.)
• Inventors: Azar, James R.
• Primary Examiner(s): Thai, Xuan
• Assistant Examiner(s): Musselman, Timothy A
• Application Number:
  US10/904,913
• Publication Number:
  US 20060121434A1
• Time in Patent Office: 2,636 Days
• Field of Search: 705/10, 705/12, 705/26, 705/7.11, 705/7.12, 705/7.29, 705/7.32, 705/26.7, 705/347, 434/323
• US Class Current: 705/7.32
• CPC Class Codes: G06Q 30/0203 Market surveys; Market polls
• CPC Class Codes: G06Q 30/0282 Rating or review of business operators or products
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A system and method for confidence-based selection of items for use in conducting a computer-implemented survey. The survey presents information about a selected plurality of items to a survey participant, to elicit survey feedback information. Information regarding the plurality of items is stored, the stored information including display information about each of the plurality of items for presentation to a survey participant. A subset of items for presentation to a survey participant is selected in accordance with a predetermined selection algorithm. Information corresponding to the selected subset of items is displayed to the survey participant via a survey user interface. Rating information is input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the presented subset of items. The rating information is utilized in various manners to affect the selection algorithm for a subsequent survey.

Claims

1. A method for conducting a computer-implemented survey to collect survey information relating to a plurality of items from a survey participant by the display of information relating to a selected subset of a set of items in the survey to a survey participant via a survey user interface generated by a computer, the selected subset of items selected as a function of the number of times that particular items have been previously shown to other survey participants in one or more prior surveys, comprising the computer-implemented steps of:

storing information regarding the set of items in a memory, the stored information including (a) display information about each of the plurality of items in the set of items for presentation to a survey participant and (b) presentation number information corresponding to the number of times that each particular item in the set of items has been previously shown to survey participants in one or more prior surveys;

selecting a subset of items from the set of items for presenting to a survey participant in accordance with a predetermined selection algorithm that utilizes the presentation number information to select items that are in need of additional data to improve a confidence factor relative to other items in the set of items;

presenting the display information corresponding to the selected subset of items to the survey participant via the survey user interface; and

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the selected subset of items presented.

2. The method of claim 1, further comprising the step of utilizing the rating information input by the survey participant to affect the probability of selection of the items in the selected subset in a subsequent selection of a subset of items for presentation to a subsequent survey participant using the predetermined selection algorithm.

3. The method of claim 1, wherein the predetermined selection algorithm utilizes an adjustment factor to cause items to be selected more or less often as a function of the rating information obtained by previous participants of a survey.

4. The method of claim 1, wherein the predetermined selection algorithm utilizes, at least in part, a random number selection of items in the plurality of items.

5. The method of claim 4, wherein the plurality of items includes a predetermined duplication of items in a set of the plurality of items, with the number of duplications of particular items influenced by the rating information.

6. The method of claim 1, wherein the predetermined selection algorithm is self-adjusting based on previous responses received during a previous survey of the plurality of items.

7. The method of claim 6, wherein the self-adjusting is based on modifying the probability of selecting an item from the plurality of items, the modifying in turn based collection of new items input by survey participants and rating of new items in comparison of previously included items.

8. The method of claim 1, wherein the predetermined selection algorithm is operative initially to randomly select items for the subset of items, and thereafter operative to select items based on utilization of the rating information.

9. The method of claim 8, wherein the probability of selection of a given item is continuously adjusted as items are rated by survey participants.

10. A method for conducting a computer-implemented survey to collect survey information relating to a plurality of items from a survey participant by the display of information relating to a selected subset of a set of items in the survey to a survey participant via a survey user interface generated by a computer, the selected subset of items selected as a function of the number of times that particular items have been previously shown to other survey participants in one or more prior surveys, comprising the computer-implemented steps of:

storing information regarding the set of items in a memory, the stored information including (a) display information about each of the plurality of items for presentation to a survey participant and (b) frequency of display information corresponding to the number of times that each particular item in the set of items has been previously shown to survey participants in one or more prior surveys;

selecting a subset of items from the set of items for presenting to a survey participant in accordance with a predetermined selection algorithm that utilizes the frequency information to select items that are in need of additional data to improve a confidence factor relative to other items in the set of items;

presenting the display information corresponding to the selected subset of items to the survey participant via the survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the selected subset of items presented; and

utilizing the rating information input by the survey participant to affect the probability of selection of items in the set of items for a subsequent selection of a subset of items for presentation to a subsequent survey participant using the predetermined selection algorithm.

11. The method of claim 10, wherein the rating information is selected from the group comprising:

ranking of items relative to each other, ranking of the items on a scale, grading the items, ordering of the items, allocating points among items, scaling the items, choosing an item over other items, categorizing items, and other equivalent methods of indicating a preference of one item over another.

12. The method of claim 10, wherein the step of selecting a subset of items in accordance with the predetermined selection algorithm comprises selecting based on a ranking of items using rating information from previous participants, such that the probability of selection of particular items for presenting in a subsequent survey is influenced by the rating information, whereby certain items that have been rated lower than other items are more likely to be selected for a survey so as to increase the number of presentations of such items.

13. The method of claim 10, further comprising the step of providing the rating information for each item as an output of the method indicative of survey results.

14. The method of claim 10, wherein the memory is a random access memory array.

15. The method of claim 10, wherein information regarding the plurality of items is stored in an ordered array and selected according to a probability index.

16. The method of claim 15, wherein the information about each item in the ordered array is stored in a data field in the ordered array.

17. The method of claim 10, wherein the subset of items selected for presentation to the survey participant is an initial subset, and wherein the subsequent selection of a set of items for presentation comprises a selection from the plurality of items that may include one or more of the items from the initial subset.

18. The method of claim 10, wherein the information regarding the plurality of items includes a unique identifier for each item for use as a primary key to access the item in the memory.

19. The method of claim 10, further comprising the step of storing a users item table for storing information provided by a survey participant relating to an additional item for inclusion in the plurality of items.

20. The method of claim 10, wherein the information regarding the plurality of items includes status information about each item.

21. The method of claim 20, wherein the status information is indicative whether an item has been previously shown to a survey participant or not.

22. The method of claim 21, wherein the predetermined selection algorithm also utilizes the status information in conjunction with the frequency information.

23. The method of claim 10, wherein the frequency information is stored in a times-shown field for each item in the array.

24. The method of claim 10, wherein the predetermined selection algorithm utilizes a selection score in selecting items for presentation.

25. The method of claim 24, wherein the selection score is based upon a confidence score.

26. The method of claim 25, wherein the selection score is further based on an adjusted mean rating determined from the rating information.

27. The method of claim 25, wherein the selection score is further based on a rank influence factor.

28. The method of claim 27, wherein the rank influence factor is an arbitrary number used to adjust the probability of an item being selected based on ranking information.

29. The method of claim 28, wherein the ranking information comprises information corresponding to the actual ranking of an item in relation to other items in the plurality of items.

30. The method of claim 10, wherein the selection algorithm selects an item from the plurality of items based on a computation of a probability index.

31. The method of claim 30, wherein the probability index is determined based on a normalized selection score.

32. The method of claim 31, wherein the normalized selection score is utilized to determine a probability of selection for each item in the plurality of items, the probability of selection of each item is used to determine how many times an item is represented in the plurality of items for selection.

33. The method of claim 10, wherein the plurality of items are represented in a computer system as a pool of selectable items stored in an array of items, with each item in the pool having a high index number and a low index number, with the index numbers representing how many times an item is represented in the pool of selectable items, and wherein the step of selecting a subset of items comprises selecting from the pool of items based on a random number used to index into the array of index numbers.

34. The method of claim 33, wherein the subset of items for presenting to a survey participant is selected by repeating the step of selecting utilizing the random number, until a predetermined number of items corresponding to the size of the selected subset of items has been chosen for presentation.

35. The method of claim 10, wherein a selected subset of items comprises a unique sample of items in the plurality of items of a predetermined sample size that meets predetermined selection criteria according to status information associated with the item.

36. The method of claim 35, wherein the predetermined sample size comprises the maximum number of items presented to a survey participant in the survey.

37. The method of claim 35, wherein the status information comprises information indicative of a condition associated with an item.

38. The method of claim 10, wherein the status information is selected from the group comprising:

whether an item is scheduled, whether an item has been approved, whether an item is implemented, whether an item is active, whether an item is in or under review, whether an item has been submitted, whether an item has been declined, or other equivalent information indicative of a condition of an item.

39. The method of claim 10, wherein the selected subset of items is selected for presentation based at least in part on an indication of interest of a participant.

40. The method of claim 39, wherein the indication of interest of a participant is obtained by input of interest information by a survey participant in response to a query prior to selection of the subset.

41. The method of claim 39, wherein the indication of interest of a participant is obtained by examining items previously submitted by the survey participant, and by selecting other items from the plurality of items based on the topical similarity of other items in the plurality of items.

42. The method of claim 39, wherein the indication of interest of a participant is obtained by executing a query of keywords relating to items submitted by the survey participant.

43. The method of claim 10, wherein the selected subset of items for presentation is a first selected subset of a predetermined small number of items, where “

small”

is relative but determined based on a number that is deemed by a survey manager to be acceptable for purposes of a particular survey, and further comprising the step of selecting additional items for presentation to a survey participant.

44. The method of claim 43, wherein the step of selecting additional items for presentation to a survey participant is based on information provided by a survey participant indicating a desire to view and rate more items.

45. The method of claim 44, wherein the survey participant is provided with a display offering an opportunity to request an additional sample of items for rating, and wherein the information provided by the survey participant indicating a desire to view and rate more items is input by the survey user interface.

46. The method of claim 45, wherein the opportunity to request an additional sample of items for rating is providing during a survey session.

47. A method for conducting a computer-implemented survey to collect survey information relating to a plurality of items from a survey participant by the display of information relating to a selected subset of a set items in the survey to a survey participant via a survey user interface generated by a computer, the selected subset of items selected as a function of the number of times that particular items have been previously shown to other survey participants in one or more prior surveys, comprising the computer-implemented steps of:

arranging information relating to the set of items in a memory as entries in an ordered array;

providing a unique identifier for each entry in the array;

providing display information about each item in the array entry for presentation to a survey participant;

providing a status information field for each item in the array;

providing a times-shown field for each item in the array;

selecting a subset of items from the set of items for presenting to a survey participant in accordance with a predetermined selection algorithm that utilizes the information in the times-shown field of the items in the set of items to select items that are in need of additional data to improve a confidence factor relative to other items in the set of items;

presenting the display information corresponding to the selected subset of items to the survey participant via the survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the subset of items presented; and

utilizing the rating information input by the survey participant to affect the probability of selection of the items in the selected subset of items for a subsequent selection of a subset of items for presentation to a subsequent survey participant using the predetermined selection algorithm.

48. A method for conducting a computer-implemented survey to collect survey information relating to a set of items from a plurality of survey participants by the display of information relating to items in the survey to a survey participant via a survey user interface generated by a computer, comprising the computer-implemented steps of:

storing information regarding each item in the set of items in a memory, the stored information including display information about each item in the set of items for presentation to a survey participant via the survey user interface;

conducting a selection operation involving selecting a first subset of items for presenting to a survey participant in accordance with a function that utilizes a probability of selection;

presenting the display information corresponding to the selected first subset of items to the survey participant via the survey user interface;

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the first subset of items presented;

determining an adjustment factor for the probability of selection of items for a subset of items as a function of the number of times that the items have previously been selected and presented to previous survey participants; and

applying the adjustment factor to the probability of selection for a subsequent selection operation for selecting a second subset of items for a subsequent survey.

49. A method for conducting a computer-implemented survey to collect survey information relating to a set of items from a survey participant by the display of information relating to a selected subset of the set of items in the survey to a survey participant via a survey user interface generated by a computer, comprising the computer-implemented steps of:

storing information regarding each item in the set of items in a memory, the stored information including display information about each items in the set of items for presentation to a survey participant via the survey user interface and frequency information indicating a number of times that each item in the set of items has been presented in a survey;

biasing the set of items as a function of the frequency information in anticipation of a selection operation;

conducting a probabilistic selection operation comprising a random selection within the set of items to select a first subset of items for presenting to a survey participant;

presenting the display information corresponding to the selected first subset of items to the survey participant via the survey user interface; and

receiving rating information input by the survey participant via the survey user interface indicating the survey participant'"'"'s preferences as to items in the first subset of items presented,

whereby a biased random selection of items to be presented to the survey participant is conducted so as to avoid presenting only newer, less frequently presented items to later survey participants and to ensure presentation of some earlier items to such later survey participants.