Automatic validation of survey results

• US 20040093261A1
• Filed: 11/08/2002
• Published: 05/13/2004
• Est. Priority Date: 11/08/2002
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jain, Vivek, Kothari, Ravi
• Application Number:
  US10/291,169
• Publication Number:
  US 20040093261A1
• Time in Patent Office: Days
• US Class Current: 705/7.32
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 30/0203 Market surveys; Market polls

Abstract

The invention describes a method, system and program product for online automatic validation of the survey results that have been conducted online. The invention uses various libraries and tools to analyse the results and conduct new implicit experiments so as to extract more information from the user in order to validate the results of the original survey.

Claims

1. A method for automatically validating the results of an online survey, comprising the steps of:

identifying a subset of potential conclusions from the set of potential conclusions that can be derived from the results of the online survey for validation,

selecting one or more research design tools for validating the selected subsets of potential conclusions either individually or combination,

defining a group of participants for validating a selected subset of potential conclusions and determining the criteria for validation,

applying the research tools on said group of participants for conducting a controlled experiment in a manner that obtains the desired information implicitly,

analyzing the results of each experiment for determining the degree of validation, and

reporting results of the validation.

2. The method as claimed in claim 1 wherein the validation is conducted while conducting the original survey.

3. The method as claimed in claim 1 wherein determining the set of potential conclusions comprises the steps of:

identifying the sub-sets of the survey that match survey templates and keywords in the library of templates and keywords,

mapping identified subsets to analytical techniques present in a library, and/or

determining cause-effect relationships and the potential conclusions associated with each cause-effect.

4. The method as claimed in claim 1 wherein each conclusion is divided into subsets either automatically using tools including rule decomposition or under guidance of an outside agent.

5. The method as claimed in claim 1 wherein the selection of subsets is derived either automatically or guided by an outside agent on the basis of a score that is assigned against parameters including potential flaws in the original survey, prior validation experiments, and the scope and importance of conclusions derived from the subset.

6. The method as claimed in claim 1 wherein the research tool comprises of implicit experiments, each implicit experiment characterized by the actuation of a stimulus, the possible responses, the procedure for conducting the experiment and the conclusions that can be drawn based on the specific response.

7. The method as claimed in claim 1 wherein participants are selected on the basis of parameters including the value of information obtained from the participant and the cost incurred in including the participant in the validation experiment.

8. The method as claimed in claim 1 wherein preferred validation methodologies are defined for each subset of conclusion to be validated and for each research design selected for validating the conclusion.

9. The method as claimed in claim 1 wherein the subsets of conclusions of the original survey are validated using the metrics determined by the validation methodology.

10. The method as claimed in claim 1 wherein the validation results are presented to the user after aggregation and summarisation.

11. The method as claimed in claim 3 wherein the list of conclusions potentially derivable from an analytical technique associated with a survey template of given format and context of user response is derived from similar previous surveys.

12. The method as claimed in claim 3 wherein cause effect relationships are autonomously determined using tools including rule induction, pattern classification, and regression analysis with optional pruning or augmentation by an external agent.

13. The method as claimed in claim 3 wherein various Bayesian techniques provide confidence value and support for each potential conclusion.

14. The method as claimed in claim 5 wherein individual subsets are scored using analytical tools including correspondence analysis and active learning.

15. The method as claimed in claim 7 wherein the participants are chosen using active learning algorithms based on a Bayesian framework to provide a measure of information gain from each user'"'"'s response.

16. The method as claimed in claim 8 wherein an appropriate methodology for validation is automatically chosen given the subsets of conclusions to be validated and the choice of implicit experiment.

17. The method as claimed in claim 1, wherein an outside agent can accept automatic execution of the steps as enumerated in claim 1 or modify some or all of the steps.

18. The method as claimed in claim 1, wherein said validation results are used to decide the incentive for entities involved with the original survey.

19. The method as claimed in claim 18 wherein said entities include designers of the original survey, participants of the original survey and other agencies associated with conducting the survey, selecting the participants and collecting the response.

20. A system for validation of online survey results comprising:

a validation criterion determination tool for generating a list of potential conclusions and their subsets from the results of the online survey, working in conjunction with a research design tool for selecting appropriate conclusion subset/s for validation,

a library of analytical techniques comprising tools and techniques for data mining, prediction, learning, classification and statistical analysis and other business intelligence tools,

a research design library containing a set of experiments used by the research design tool to decide on the appropriate validation experiment,

a respondent selection tool operating independently or in conjunction with the research design tool to select the participants on which the validation experiment is performed,

a validation metrics determination tool,

an analysis tool for analysing the results of each experiment, and

a reporting tool for providing the results of the analysis.

21. The system as claimed in claim 20 wherein the validation criterion determination tool comprises of a library of templates for online survey design and associated keywords, which are mapped to various analytical techniques available with the library of analytical techniques.

22. The system as claimed in claim 20 wherein the library of analysis techniques includes information about input requirements, the performance and/or predictive accuracy of each tool and an initial map of potential conclusions for a given input and the techniques that can be used to achieve them.

23. The system as claimed in claim 20 wherein the reporting tool is capable of reporting the validation of each subset and the aggregation in a tree format with each node representing an aggregation of its child nodes.

24. The system as claimed in claim 21 wherein the mapping is specified in the form of a look-up table or is learnt over a period of time through repeated usage and represented in the form of a table or set of rules.

25. The system as claimed in claim 22 wherein the mapping is stored in various forms including a database table, logical rules, decision tree and neural network.

26. The system as claimed in claim 20 further comprising of a user profile and secondary data repository.

27. The system as claimed in claim 26 wherein the user profile and secondary data repository comprise of online and/or offline, private and/or public data.

28. The system as claimed in claimed in claim 20 wherein the tools and libraries wholly or partially reside on a computing system comprising of:

a system bus,

a communications unit connected to the system bus,

a memory including a set of instructions connected to the system bus, and

a control unit executing the instructions in the memory for the functioning of the tools.

29. The system as claimed in claim 28 further connected to other similar systems and database systems that may contain the libraries and tools to complement and supplement the already existing tools and libraries present.

30. The system as claimed in claim 20, comprising of an outside agent that can accept automatic execution of the steps enumerated above and comprising of means to modify some or all of the steps.

31. The system as claimed in claim 20 comprising of means to calculate the incentive for entities involved with the original survey using said validation results.

32. A computer program product comprising computer readable program code stores on a computer readable storage medium embodied therein for validation of online survey results comprising of:

computer readable program code means configured for determining the set of potential conclusions from the result of original online survey and for selecting subsets of conclusions to be validate either individually or in combination with each other,

computer readable program code means configured for selecting the research design to validate the chosen subsets of conclusions,

computer readable program code means configured for selecting the participants for said research design experiment and for determining the criteria for validation,

computer readable program code means configured for conducting the implicit experiment, and

computer readable program code means configured for collecting, analysing and reporting the result of the implicit experiment.

33. The computer program product as claimed in claim 32 comprising of computer readable program code means configure for arriving at a set of potential conclusions either automatically or fully or partially guided by an outside agent and means for:

identifying the sub-sets of the original survey that match a survey templates and keywords in the library of templates and keywords, and mapping those to analytical techniques present in a library, and/or

determining cause-effect relationships and the potential conclusion associated with it.

34. The computer program product as claimed in claim 32 comprising of computer readable program code means configured for dividing each conclusion into subsets either automatically using tools including rule decomposition or through partial or full guidance of an outside agent.

35. The computer readable program product as claimed in claim 32 comprising of computer readable program code means configured for automatic or guided selection of subsets and scoring the subsets on the basis of a parameters including potential flaws in the original survey, prior validation experiments, and the scope and importance of conclusions derived from the subset.

36. The computer program product as claimed in claim 32 comprising of computer readable program code means configured for the selection of participants using parameters including the value of information obtained from the participant and the cost incurred by including the participant in the approach design.

37. The computer readable program product as claimed in claim 32 comprising of computer readable program code means configured for validating the subsets of conclusions of the original survey using the metric determined by the validation methodology and means for combining the results of validation of each individual subset, in case of multiple conclusions.

38. The computer program product as claimed in claim 32 comprising of computer readable program code means configured for presenting the validation results to the user after aggregation and summarisation.

39. The computer program product as claimed in claim 32, comprising of computer readable program code means configured for accepting automatic execution of the steps as enumerated above and computer readable program code means configured for modifying some or all of the steps.

40. The computer program product as claimed in claim 32 comprising of computer readable program code means configured for deciding the incentive for entities involved with the original survey using said validation results.