Operation and method for prediction and management of the validity of subject reported data

• US 7,873,589 B2
• Filed: 08/24/2007
• Issued: 01/18/2011
• Est. Priority Date: 04/02/2001
• Status: Expired due to Term

• Current Assignee: eResearchTechnology, Inc. (Explorer Holdings, Inc.)
• Original Assignee: Invivodata Incorporated (Explorer Holdings, Inc.)
• Inventors: Engfer, Douglas R., Shiffman, Saul, Paty, Jean
• Primary Examiner(s): Holmes; Michael B.
• Assistant Examiner(s): Bharadwaj; Kalpana
• Application Number:
  US11/844,632
• Publication Number:
  US 20080052259A1
• Time in Patent Office: 1,243 Days
• Field of Search: 706/46, 703/2, 709/224
• US Class Current: 706/47
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

A system for developing and implementing empirically derived algorithms to generate decision rules to predict invalidity of subject reported data and fraud with research protocols in surveys allows for the identification of complex patterns of variables that detect or predict subject invalidity of subject reported data and fraud with the research protocol in the survey. The present invention may also be used to monitor invalidity of subject reported data within a research protocol to determine preferred actions to be performed. Optionally, the invention may provide a spectrum of invalidity, from minor invalidity needing only corrective feedback, to significant invalidity requiring subject removal from the survey. The algorithms and decision rules can also be domain-specific, such as detecting invalidity or fraud among subjects in a workplace satisfaction survey, or demographically specific, such as taking into account gender or age. The algorithms and decision rules may be optimized for the specific sample of subjects being studied.

Claims

1. A computer implemented method of predicting the validity of subject reported data, comprising the steps of:

a) analyzing validity markers wherein said validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

b) analyzing historical protocol data wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

c) generating at least one predictive algorithm for predicting the invalidity of subject reported data by quantitative analysis of said validity markers and said historical protocol data; and

d) translating said at least one predictive algorithm into at least one prediction rule for use with a survey.

2. The method of claim 1, further comprising the step of

comparing said validity markers to said at least one prediction rule to determine if action is needed.

3. The method of claim 2, further comprising the step of determining an appropriate action if said step of comparing indicates that action is needed.

4. The method of claim 1, wherein said step of analyzing comprises the step of employing a portable electronic device capable of displaying information and receiving and storing input from a user.

5. The method of claim 1, further composing the step of creating an evaluability database adapted to store data related to said validity of subject reported data.

6. The method of claim 5, wherein said evaluability database is tailored to a condition affecting the subject.

7. The method of claim 1, wherein said analyzing of said validity markers or said historical protocol data comprises analyzing at least one database containing at least one set of data from the group consisting of historical validity markers and historical protocol data.

8. The method of claim 1, wherein said validity markers are historical validity markers from more than one event.

9. The method of claim 1, wherein said validity markers are from at least one event.

10. A computer implemented method of determining the validity of subject reported data, comprising the steps of:

a) analyzing at least one set of data selected from the group consisting of historical validity markers and historical protocol data;

i) wherein said historical validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

ii) wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

b) generating at least one algorithm reflective of said at least one set of data selected from the group consisting of said historical validity markers and said historical protocol data by quantitative analysis of said at least one set of data;

c) translating said at least one algorithm into at least one decision rule for analyzing information on the validity of subject reported data;

d) obtaining validity markers for at least one event; and

e) analyzing said validity markers with said at least one decision rule to determine if corrective action is needed.

11. The method of claim 10, further comprising the step of determining an appropriate corrective action if said step of analyzing indicates that corrective action is needed.

12. The method of claim 10, wherein said step of obtaining comprises using a portable electronic device capable of displaying information and receiving and storing input from a user.

13. The method of claim 10, wherein said step of generating employs at least one of the group consisting of multiple linear regression, discriminant function analysis, logistic regression, neural networks, classification trees and regression trees.

14. The method of claim 10, wherein said analyzing said validity markers or said historical protocol data comprises, analyzing at least one database containing at least one data set selected from the group consisting of historical validity markers and historical protocol data.

15. A computer implemented method of determining the validity of subject reported data, comprising the steps of:

a) analyzing historical validity markers and historical protocol data;

i) wherein said validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

ii) wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

b) generating a spectrum of invalidity representative of said historical validity markers or said historical protocol data by quantitative analysis of the historical validity markers and the historical protocol data;

c) obtaining validity markers for at least one event; and

d) comparing said spectrum of invalidity to the historical validity markers to determine if corrective action is needed.

16. The method of claim 15, further comprising the step of determining an appropriate corrective action if said step of comparing indicates that corrective action is needed.

17. A computer implemented method of detecting subject fraud, comprising the steps of:

a) analyzing historical validity markers and historical protocol data;

i) wherein said historical validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

ii) wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

b) generating at least one fraud detection algorithm for detecting subject fraud by quantitative analysis of the historical validity markers and the historical protocol data; and

c) translating said at least one fraud detection algorithm into at least one fraud detection rule for use with a survey.

18. A method of detecting subject fraud, comprising the steps of:

a) analyzing information on the validity of subject reported data and historical protocol data;

i) wherein said validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

ii) wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

b) generating at least one fraud detection algorithm for detecting subject fraud by quantitative analysis of said information on the validity of subject reported data and said historical protocol data; and

c) translating said at least one fraud detection algorithm into at least one fraud detection rule for use with a survey.

19. The method of claim 18, further comprising the steps of:

comparing said validity markers to said at least one fraud detection rule to determine if action is needed.

20. The method of claim 19, further comprising the step of determining an appropriate action if said step of comparing indicates that action is needed.

21. The method of claim 19, wherein said analyzing comprises the use of a portable electronic device capable of displaying information and receiving and storing input from a user.

22. The method of claim 19, further comprising the step of creating an evaluability database adapted to store data related to subject fraud.

23. The method of claim 22, wherein said evaluability database is tailored to a condition affecting said subject.

24. The method of claim 19, wherein said analyzing employs at least one database containing at least one data set selected from the group consisting of historical validity markers and historical protocol data.

25. A medium suitable for use in an electronic device, comprising instructions for execution on the electronic device, said instructions comprising the steps of:

a) analyzing at least one data set selected from the of the group consisting of validity markers and historical protocol data;

i) wherein said validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

ii) wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

b) generating at least one predictive algorithm for predicting invalid subject reported data by quantitative analysis of said at least one data set; and

c) translating said at least one predictive algorithm into at least one prediction rule for use with a survey.

26. The medium of claim 25 comprising instructions further comprising the step of

comparing validity markers to at least one prediction rule to determine if an action is needed.

27. The medium of claim 25, wherein said data set is obtained using a portable electronic device capable of displaying information and receiving and storing input from a user.

28. The medium of claim 25 comprising instructions further comprising the step of creating an evaluability database adapted to store data related to information on the validity of subject reported data.

29. The medium of claim 25, wherein said validity markers are historical information on the validity of subject reported data, from more than one event.

30. The medium of claim 25, wherein said validity markers are from at least one event.

31. The medium of claim 25, wherein said historical protocol data is from more than one survey.

32. The medium of claim 25, wherein said protocol data is from at least one survey.

33. A medium suitable for use in an electronic device, comprising instructions for execution on said electronic device, said instructions comprising the steps of:

a) analyzing at least one data set selected from the group consisting of validity markers and historical protocol data;

i) wherein said validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

ii) wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

b) generating at least one algorithm reflective of said at least one data set by quantitative analysis of said at least one data set;

c) translating said at least one algorithm into at least one decision rule for analyzing information on the validity of subject response data;

d) obtaining validity markers for at least one event; and

e) analyzing said validity markers with said at least one decision rule to determine if corrective action is needed.

34. The medium of claim 33 comprising instructions, further comprising the step of determining an appropriate corrective action if said step e) indicates that corrective action is needed.

35. The medium of claim 33, wherein said step of obtaining comprises using a portable electronic device capable of displaying information and receiving and storing input from a user.

36. The medium of claim 33, wherein said step of generating employs at least one of the group consisting of multiple linear regression, discriminant function analysis, logistic regression, neural networks, classification trees and regression trees.

37. A medium suitable for use in an electronic device, comprising instructions for execution on the electronic device, said instructions comprising the steps of:

a) analyzing historical validity markers and historical protocol data;

i) wherein said validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

ii) wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

b) generating a spectrum of invalidity representative of said historical validity markers or said historical protocol data by quantitative analysis of said historical validity markers and the historical protocol data;

c) obtaining validity markers for at least one event; and

d) comparing the spectrum of invalidity to the validity markers to determine if corrective action is needed.

38. The medium of claim 37 having instructions further comprising the step of determining an appropriate corrective action if said step of comparing indicates that corrective action is needed.

39. The medium of claim 38, wherein said step of obtaining comprises using a portable electronic device capable of displaying information and receiving and storing input from a user.

40. A medium suitable for use in an electronic device, comprising instructions for execution on the electronic device, said instructions comprising the steps of:

a) analyzing validity markers and historical protocol data;

i) wherein said validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

ii) wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

b) generating at least one fraud detection algorithm for detecting subject fraud by quantitative analysis of said validity markers and said protocol data; and

c) translating said at least one fraud detection algorithm into at least one fraud detection rule for use with a survey.

41. The medium of claim 40 comprising instructions, further comprising the step of determining an appropriate corrective action if fraud is detected in a survey.

42. The medium of claim 40, wherein said electronic device is a portable electronic device capable of displaying information and receiving and storing input from a user.

43. The medium of claim 40, wherein said step of generating employs at least one of the group consisting of multiple linear regression, discriminant function analysis, logistic regression, neural networks, classification trees and regression trees.

44. The medium of claim 40, wherein said validity markers are historical information on said validity of subject reported data, from more than one event.

45. The medium of claim 40, wherein said validity markers are from at least one event.

46. The medium of claim 40, wherein said historical protocol data is from more than one survey.

47. The medium of claim 40, wherein said historical protocol data is from at least one survey.

48. A medium suitable for use in an electronic device, comprising instructions for execution on the electronic device, said instructions comprising the steps of:

a) analyzing information on the validity of subject reported data and historical protocol data, wherein said information on the validity of subject reported data comprises one or more validity markers;

i) wherein said validity markers comprise a subject'"'"'s timeliness in responding to a question or responding to a prompt, a subject'"'"'s length of response to a question, consistency of a subject'"'"'s response with other collected data, proximity of data on a subject'"'"'s response to an expected data range, completeness of data, a subject'"'"'s voice stress levels, historical information on a subject'"'"'s veracity, failure of a subject to complete a survey, a subject'"'"'s response to a question about morale, a subject'"'"'s preference of a particular good, service or media outlet, a survey environment'"'"'s temperature, a survey environment'"'"'s light level, a survey environment'"'"'s ambient noise level, a subject'"'"'s heart rate, a subject'"'"'s blood pressure, a subject'"'"'s temperature, a subject'"'"'s skin electroconductivity, a subject'"'"'s perspiration, or a subject'"'"'s rate of blinking;

ii) wherein said historical protocol data comprises data on the research protocol of an earlier clinical trial or survey, an earlier question posed to a subject, frequency of prompting of a subject during various times of the day or week, time allowed for a subject to respond to a question, requirements of a subject'"'"'s behavior, conditions mandating removal of a subject from certain statistical analyses, or conditions mandating removal of a subject as a participant in a clinical trial or survey;

b) generating at least one fraud detection algorithm for detecting subject fraud by quantitative analysis of said information on the validity of subject reported data; and

c) translating the at least one fraud detection algorithm into at least one fraud detection rule for use with a survey.

49. The medium of claim 48, wherein said validity markers comprise historical information on said validity of subject reported data, from more than one event.

50. The medium of claim 48, wherein said validity markers are from at least one event.

51. The medium of claim 48, wherein said electronic device is a portable electronic device capable of displaying information and receiving and storing input from a user.