SYSTEMS AND METHODS FOR DETECTION OF SATISFICING IN SURVEYS

• US 20130054497A1
• Filed: 08/26/2011
• Published: 02/28/2013
• Est. Priority Date: 08/26/2011
• Status: Active Grant

• Current Assignee: SurveyMonkey (Momentive Global, Inc.)
• Original Assignee: SurveyMonkey.com, LLC (Momentive Global, Inc.)
• Inventors: Garland, Philip, Chen, Kuang
• Granted Patent: US 8,583,568 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0203 Market surveys; Market polls

Abstract

Response data relating to a plurality of responses to a survey is received. The survey comprises a plurality of questions. The response data for each of the plurality of responses comprises a plurality of answers to at least a subset of the plurality of questions. A questionnaire response model is created using the response data. It is then determined, for each questionnaire response, a respective probability that the respective questionnaire response represents satisficing, such that where the respective probability exceeds a threshold, the respective questionnaire response is identified as an outlier, and where the respective probability does not exceed the threshold, the respective questionnaire response is identified as an inlier. A representation of each questionnaire response is output, each respective representation reflecting the likelihood that the respective response to which the respective representation represents satisficing.

Claims

1. A method comprising:

receiving, by a computing device, response data relating to a plurality of responses to a survey, the survey comprising a plurality of questions, the response data for each of the plurality of responses comprising a plurality of answers to at least a subset of the plurality of questions;

creating, by the computing device, using the response data, a questionnaire response model;

determining, by the computing device, for each questionnaire response of the plurality of questionnaire responses, a respective probability that the respective questionnaire response represents satisficing, such that

where the respective probability is not less than a threshold, the respective questionnaire response is identified as an outlier, the threshold relating to a level of probability reflecting a confidence that where the respective probability is not less than the threshold, the respective questionnaire response represents satisficing, and

where the respective probability does not exceed the threshold, the respective questionnaire response is identified as an inlier; and

outputting, by the computing device, a representation of each questionnaire response of the plurality of questionnaire responses, each respective representation reflecting the likelihood that the respective response to which the respective representation represents satisficing.

2. The method of claim 1 wherein the creating and determining are repeated using a subset of the plurality of questionnaire responses identified as inliers as input to the creating a questionnaire response model.

3. The method of claim 1 wherein the creating and determining are iteratively repeated, wherein for each iteration, a respective subset of the plurality of questionnaire responses are identified as inliers and the respective subset of the plurality of questionnaire responses are input to the creating of a next iteration.

4. The method of claim 1 wherein the questionnaire response model comprises a plurality of parameters, each parameter comprising a probability that a respective value of an answer to a respective one question of the plurality of questions represents satisficing.

5. The method of claim 4 wherein for each questionnaire response of the plurality of responses, for at least some of the answers in the respective questionnaire response, a probability that a value of the respective answer represents satisficing is determined using a respective one of the plurality of parameters that relates to the value of the respective answer, wherein the respective probability that a value of the respective answer represents satisficing for at least some of the answers in the respective response are used to determine the respective probability that the respective questionnaire response represents satisficing.

6. The method of claim 4 wherein the questionnaire response model is created by a method comprising:

creating, by the computing device, using the response data, the questionnaire response model using structure learning techniques, wherein the questionnaire response model comprises a probabilistic graphical model reflecting a learned structure for the response data; and

estimating, by the computing device, the plurality of parameters of the questionnaire response model.

7. The method of claim 6, wherein the structure learning techniques searches through a space of possible structures using simulated annealing, and chooses a best structure according to Bayesian Dirichlet Equivalence criterion, wherein a best structure is used to create the questionnaire response model.

8. The method of claim 6, wherein the plurality of parameters of the questionnaire response model are estimated using a method comprising:

creating, by the computing device, using the questionnaire response model, a Bayesian network;

estimating, by the computing device, the plurality of parameters of the questionnaire response model using the Bayesian network.

9. The method of claim 8 comprising:

modifying, by a computing device, the response data such that, for each response of the plurality of responses, the value of each respective answer of the plurality of answers is transformed to a respective discrete value, the respective discrete value comprising a respective one integer of a plurality of integers;

wherein the modified response data is input to the creating a questionnaire response model.

10. The method of claim 9 wherein the estimating the plurality of parameters of the questionnaire response model using the Bayesian network comprises determining, for each question of the plurality of questions, a respective probability mass function for the respective question F_i with m possible discrete values, conditioned on a set of parent nodes ρ

(F_i) from the Bayesian network, using the equation;

11. The method of claim 10 wherein for a respective response of the plurality of responses, where a respective answer is not present for a respective one question of the plurality of questions, an answer to the respective question is added to the response for the respective one question comprising a null value.

12. The method of claim 1 comprising:

receiving, by a computing device, an input data model comprising a plurality of question definitions, such that each question definition of the plurality of question definitions relates to a respective one question of the plurality questions, each question definition comprising a respective expected answer data type; and

processing, by a computing device, using the input data model, the response data, creating modified response data comprising at least some of the plurality of responses, such that where a respective one response of the plurality of responses comprises a respective answer to a respective one question of the plurality of questions and a data type of a value of the respective answer does not match the respective expected answer data type of a respective question definition of the plurality of question definitions that relates to the respective one question, the respective one response is not included in the modified response data,

wherein the modified response data is input to the creating a questionnaire response model.

13. The method of claim 9 comprising:

receiving, by a computing device, an input data model comprising a plurality of question definitions, such that each question definition of the plurality of question definitions relates to a respective one question of the plurality questions, each question definition comprising, each question definition of the plurality of question definitions comprising a respective maximum index value, the respective maximum index value comprising a maximum discrete value for answers to the respective question;

processing, by a computing device, using the input data model, the response data, creating modified response data comprising at least some of the plurality of responses, such that where a respective one response of the plurality of responses comprises a respective answer to a respective one question of the plurality of questions and a value of the respective answer exceeds the respective maximum index value of a respective question definition of the plurality of question definitions that relates to the respective one question, the respective one response is not included in the modified response data,

wherein the modified response data is input to the creating a questionnaire response model.

14. The method of claim 1 additionally comprising:

determining, by a computing device, a time required to process the input data;

causing, by the computing device, the time required to process the input data to be displayed to a user on a user computing device, wherein the user is presented with an option to terminate processing;

receiving, by the computing device, an indication from the user that the user has selected the option to terminate processing; and

in response to receiving the indication from the user that the user has selected the option to terminate processing, terminating, by the computing device, processing of the input data.

15. The method of claim 1 wherein the creating and determining are iteratively repeated until no outliers are identified in the respective subset of the plurality of questionnaire responses.

16. The method of claim 1 wherein the creating and determining are iteratively repeated until a number of inliers in the respective subset of the plurality of questionnaire responses does not change for two iterations.

17. The method of claim 6 additionally comprising:

receiving, by the computing device, over a network, from a user, modifications to the questionnaire response model;

modifying, by the computing device, the questionnaire response model in response to modifications to the questionnaire response model received from a user,

wherein the modified response model is input to the estimating computing the plurality of parameters of the questionnaire response model.

18. The method of claim 1 additionally comprising:

outputting, by the computing device, to a first data file, each questionnaire response of the plurality of questionnaire responses that is identified as an inlier;

outputting, by the computing device, to a second data file, each questionnaire response of the plurality of questionnaire responses that is identified as a outlier.

19. The method of claim 18 additionally comprising:

transmitting, by the computing device, the first data file and the second data file, over a network, to a user.

20. The method of claim 1 additionally comprising:

causing, by the computing device, over a network, a user device associated with a user to display a representation of the number of inliers and the number of outliers.

21. The method of claim 1 wherein the response data is received in the form of a batched data file.

22. The method of claim 1 wherein the response data is received in real-time from an online survey system.

23. A computer readable storage media for tangibly storing thereon computer readable instructions for a method comprising:

receiving response data relating to a plurality of responses to a survey, the survey comprising a plurality of questions, the response data for each of the plurality of responses comprising a plurality of answers to at least a subset of the plurality of questions;

creating, by the computing device, using the response data, a questionnaire response model;

determining. for each questionnaire response of the plurality of questionnaire responses, a respective probability that the respective questionnaire response represents satisficing, such that

where the respective probability is not less than a threshold, the respective questionnaire response is identified as an outlier, the threshold relating to a level of probability reflecting a confidence that where the respective probability is not less than the threshold, the respective questionnaire response represents satisficing, and

where the respective probability does not exceed the threshold, the respective questionnaire response is identified as an inlier; and

outputting, a representation of each questionnaire response of the plurality of questionnaire responses, each respective representation reflecting the likelihood that the respective response to which the respective representation represents satisficing.

24. A computing device comprising:

a processor;

a storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising;

response data receiving logic executed by the processor for receiving response data relating to a plurality of responses to a survey, the survey comprising a plurality of questions, the response data for each of the plurality of responses comprising a plurality of answers to at least a subset of the plurality of questions;

questionnaire response model creation logic executed by the processor for creating, using the response data, a questionnaire response model;

satisficing determination logic executed by the processor for determining, for each questionnaire response of the plurality of questionnaire responses, a respective probability that the respective questionnaire response represents satisficing, such that

where the respective probability is not less than a threshold, the respective questionnaire response is identified as an outlier, the threshold relating to a level of probability reflecting a confidence that where the respective probability is not less than the threshold, the respective questionnaire response represents satisficing, and

where the respective probability does not exceed the threshold, the respective questionnaire response is identified as an inlier; and

data output logic executed by the processor for outputting a representation of each questionnaire response of the plurality of questionnaire responses, each respective representation reflecting the likelihood that the respective response to which the respective representation represents satisficing.