METHOD AND APPARATUS FOR EVALUATION OF A SUBJECT'S EMOTIONAL, PHYSIOLOGICAL AND/OR PHYSICAL STATE WITH THE SUBJECT'S PHYSIOLOGICAL AND/OR ACOUSTIC DATA

• US 20120116186A1
• Filed: 07/20/2010
• Published: 05/10/2012
• Est. Priority Date: 07/20/2009
• Status: Abandoned Application

• Current Assignee: University of Florida Research Foundation, Incorporated (State University System of Florida)
• Original Assignee: University of Florida Research Foundation, Incorporated (State University System of Florida)
• Inventors: Shrivastav, Rahul, Lin, Jenshan, Patel, Sona, Zawoy, Karl R.
• Application Number:
  US13/384,329
• Publication Number:
  US 20120116186A1
• Time in Patent Office: Days
• US Class Current: 600/301
• CPC Class Codes: A61B 5/0507 using microwaves or terahertz waves
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: A61B 5/7285 for synchronising or triggering a physiological measurement or image acquisition with a physiological event or waveform, e.g. an ECG signal A61B6/5205, A61B6/541, A61B8/5207, A61B8/543 take precedence
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals

Abstract

Embodiments of the subject invention relate to a method and apparatus for remote evaluation of a subject'"'"'s emotive and/or physiological state. Embodiments can utilize a device that can be used to determine the emotional and/or physiological state of a subject through the measurement and analysis of vital signs and/or speech. A specific embodiment relates to a device capable of remotely acquiring a subject'"'"'s physiological and/or acoustic data, and then correlating and analyzing the data to provide an assessment of a subject'"'"'s emotional and/or physiological state. In a further specific embodiment, the device can acquire such data, correlate and analyze the data, and provide the assessment of the subject'"'"'s emotional state and/or physiological in real time.

Claims

1. A method of determining an emotional state of a subject, comprising:

measuring one or more physiological characteristics of a subject;

measuring one or more acoustic characteristics of acoustic output of the subject; and

processing the measured one or more physiological characteristics and the measured one or more acoustic characteristics to determine an emotional state of the subject.

2. The method according to claim 1, wherein processing the measured one or more physiological characteristics and the measured one or more acoustic characteristics to determine an emotional state of the subject comprises:

processing the measured one or more acoustic characteristics to determine a predicted emotional state of the subject; and

refining and/or verifying the predicted emotional state of the subject based on the measured one or more physiological characteristics to determine the emotional state of the subject.

3. The method according to claim 2, further comprising:

providing an acoustic space having one or more dimensions, wherein each dimension of the one or more dimensions of the acoustic space corresponds to at least one baseline acoustic characteristic; and

comparing each acoustic characteristic of the measured one or more acoustic characteristics to a corresponding one or more baseline acoustic characteristics,

wherein processing the measured one or more acoustic characteristics to determine a predicted emotional state of the subject, comprises predicting an emotional state of the subject based on the comparison, wherein the emotional state of the subject comprises at least one magnitude along a corresponding at least one of the one or more dimensions within the acoustic space.

4. The method according to claim 1, wherein processing the measured one or more physiological characteristics and the measured one or more acoustic characteristics to determine an emotional state of the subject comprises:

processing the measured one or more physiological characteristics to determine a predicted emotional state of the subject; and

refining and/or verifying the predicted emotional state of the subject based on the measured one or more acoustic characteristics to determine the emotional state of the subject.

5. The method according to claim 4, wherein the measured one or more physiological characteristics comprises a heartbeat of the subject, wherein the heartbeat of the subject is processed to determine that the predicted emotional state of the subject comprises excitement, and the predicted emotional state of the subject is refined based on the measured one or more acoustic characteristics to determine that the emotional state of the subject comprises fear.

6. The method according to claim 1, wherein processing the measured one or more physiological characteristics and the measured one or more acoustic characteristics to determine an emotional state of the subject comprises:

determining at least one baseline physiological characteristic based on the measured one or more acoustic characteristics;

processing the measured one or more physiological characteristics with the determined at least one baseline physiological characteristic; and

determining the emotional state of the subject based on processing the measured one or more physiological characteristics with the determined at least one baseline physiological characteristic.

7. The method according to claim 6, wherein determining the at least one baseline physiological characteristic based on the measured one or more acoustic characteristics comprises:

predicting an age and/or gender of the subject based on the measured one or more acoustic characteristics; and

determining the at least one baseline physiological characteristic based on the predicated age and/or gender of the subject.

8. The method according to claim 1, wherein processing the measured one or more physiological characteristics and the measured one or more acoustic characteristics to determine an emotional state of the subject comprises:

determining at least one baseline acoustic characteristic based on the measured one or more physiological characteristics;

processing the measured one or more acoustic characteristics with the determined at least one baseline acoustic characteristic; and

determining an emotional state of the subject based on processing the measured one or more acoustic characteristics with the determined at least one baseline acoustic characteristic.

9. The method according to claim 8, wherein determining the at least one baseline acoustic characteristic based on the measured one or more physiological characteristics comprises:

measuring a respiration rate of the subject; and

determining the at least one baseline acoustic characteristic based on the measured respiration rate of the subject.

10. The method according to claim 9, wherein the at least one baseline acoustic characteristic comprises a speaking rate.

11. The method according to claim 8, further comprising:

providing an acoustic space having one or more dimensions, wherein each at least one baseline acoustic characteristic corresponds to one or more dimensions in the acoustic space, wherein the determined emotional state of the subject comprises at least one magnitude along a corresponding at least one of the one or more dimensions within the acoustic space.

12. The method according to claim 1, wherein processing the measured one or more physiological characteristics and the measured one or more acoustic characteristics to determine an emotional state of the subject comprises:

selecting at least one measured physiological characteristic of the measured one or more physiological characteristics, wherein the selected at least one measured physiological characteristic corresponds to a corresponding at least one measured physiological segment of time within a particular segment of time;

selecting at least one measured acoustic characteristic of the measured one or more acoustics characteristics, wherein the selected at least one measured acoustic characteristic corresponds to a corresponding at least one measured acoustic segment of time within the particular segment of time; and

processing the selected at least one measured physiological characteristic and the selected at least one measured acoustic characteristic to determine the emotional state of the subject.

13. The method according to claim 12, wherein the emotional state of the subject corresponds to the particular segment of time.

14. The method according to claim 13, wherein the particular segment of time corresponds to a 10 to 20 second segment of time.

15. The method according to claim 13, wherein the particular segment of time occurs after a stimulus is provided to the subject.

16. The method according to claim 13, wherein the particular segment of time begins within 3 seconds of the stimulus.

17. The method according to claim 15, wherein the stimulus is a question posed to the subject.

18. The method according to claim 15, further comprising providing the stimulus to the subject.

19. The method according to claim 13, wherein the corresponding at least one measured acoustic segment of time is one measured acoustic segment of time within the particular segment of time, wherein each of the selected at least one measured acoustic characteristics corresponds to the measured acoustic segment of time.

20. The method according to claim 13, wherein the corresponding at least one measured acoustic segment of time comprises a plurality of measured acoustic segments of time within the particular segment of time, wherein each of the selected at least one measured acoustic characteristics corresponds to one of the plurality of measured acoustic segments of time.

21. The method according to claim 19, wherein the corresponding at least one measured physiological segment of time is one measured physiological segment of time within the particular segment of time, wherein each of the selected at least one measured physiological characteristics corresponds to the measured physiological segment of time.

22. The method according to claim 19, wherein the corresponding at least one measured physiological segment of time comprises a plurality of measured physiological segments of time within the particular segment of time, wherein each of the selected at least one measured physiological characteristics corresponds to one of the plurality of measured physiological segments of time.

23. The method according to claim 20, wherein the corresponding at least one measured physiological segment of time is one measured physiological segment of time within the particular segment of time, wherein each of the selected at least one measured physiological characteristics corresponds to the measured physiological segment of time.

24. The method according to claim 20, wherein the corresponding at least one measured physiological segment of time comprises a plurality of measured physiological segments of time within the particular segment of time, wherein each of the selected at least one measured physiological characteristics corresponds to one of the plurality of measured physiological segments of time.

25. The method according to claim 13, further comprising:

selecting an additional at least one measured physiological characteristic of the measured one or more physiological characteristics, wherein the additional at least one measured physiological characteristic corresponds to a corresponding additional at least one measured physiological segment of time within an additional segment of time;

selecting an additional at least one measured acoustic characteristic of the measured one or more acoustics characteristics, wherein the additional at least one measured acoustic characteristic corresponds to a corresponding additional at least one measured acoustic segment of time within the additional segment of time; and

processing the additional at least one measured physiological characteristic and the additional at least one measured acoustic characteristic to determine an additional emotional state of the subject corresponding to the additional segment of time.

26. The method according to claim 25, wherein the particular segment of time and the additional segment of time overlap.

27. The method according to claim 25, wherein the particular segment of time and the additional segment of time do not overlap.

28. The method according to claim 1, where at least one of the one or more physiological characteristics is selected from the group consisting of:

heartbeat, respiration, temperature, and galvanic skin response.

29. The method according to claim 1, wherein the acoustic output of the subject comprises speech of the subject and at least one of the one or more acoustic characteristics is an acoustic characteristic of the speech of the subject.

30. The method according to claim 29, wherein at least one of the one or more acoustic characteristics comprises a suprasegmental property of the speech of the subject.

31. The method according to claim 29, wherein the at least one of the one or more acoustic characteristics is selected from the group consisting of:

fundamental frequency, pitch, intensity, loudness, and speaking rate.

32. The method according to claim 29, wherein the at least one of the one or more acoustic characteristics is selected from the group consisting of:

number of peaks in the pitch, intensity contour, loudness contour, pitch contour, fundamental frequency contour, attack of the intensity contour, attack of the loudness contour, attack of the pitch contour, attack of the fundamental frequency contour, fall the intensity contour, fall of the loudness contour, fall of the pitch contour, fall of the fundamental frequency contour, duty cycle of the peaks in the pitch, normalized minimum pitch, normalized maximum of pitch, cepstral peak prominence (CPP), and spectral slope.

33. The method according to claim 1, wherein measuring the one or more physiological characteristics is accomplished in a non-contact manner.

34. The method according to claim 33, wherein measuring the one or more physiological characteristics comprises:

transmitting an RF signal towards the subject;

receiving a reflected RF signal from the subject;

identifying different orders of harmonics caused by a non-linear effect in the reflected RF signal; and

determining an amplitude of a periodic movement of the target from the identified different orders of harmonics.

35. The method according to claim 33, wherein measuring the one or more physiological characteristics comprises:

transmitting a signal towards the subject;

receiving a reflected signal from the subject;

reconstructing a complex signal from an I channel and a Q channel for the received reflected signal;

applying a Fourier transform to the reconstructed signal to obtain the detected spectrum;

extracting angular information of the reconstructed complex signal; and

obtaining original vibration information by analyzing the angular information.

36. The method according to claim 1, wherein measuring the one or more acoustic characteristics is accomplished in a non-contact manner.

37. The method according to claim 1, wherein processing the measured one or more physiological characteristics and the measured one or more acoustic characteristics comprises correlating the measured one or more physiological characteristics with the measured one or more acoustic characteristics.

38. The method according to claim 1, wherein processing the measured one or more physiological characteristics and the measured one or more acoustic characteristics comprises normalizing the measured one or more physiological characteristics based on the measured one or more acoustic characteristics.

39. The method according to claim 1, wherein processing the measured one or more physiological characteristics and the measured one or more acoustic characteristics comprises normalizing the measured one or more acoustic characteristics based on the measured one or more physiological characteristics.

40. A method of determining a physiological state of a subject, comprising:

measuring one or more physiological characteristics of a subject;

creating a corresponding one or more predicted physiological characteristics of the subject based on the measured one or more physiological characteristics of the subject;

measuring one or more acoustic characteristics of acoustic output of the subject;

refining the corresponding one or more predicted physiological characteristics based on the measured one or more acoustic characteristics; and

determining a physiological state of the subject based on the refined one or more physiological characteristics of the subject.

41. The method according to claim 40, wherein refining the one or more physiological characteristics based on the measured one or more acoustic characteristics comprises normalizing the predicted one or more physiological characteristics based on the measured one or more acoustic characteristics.

42. A method of determining physiological characteristics of a subject, comprising:

measuring one or more physiological characteristics of a subject;

creating a corresponding one or more predicted physiological characteristics of the subject based on the measured one or more physiological characteristics of the subject;

measuring one or more acoustic characteristics of acoustic output of the subject; and

normalizing the corresponding one or more predicted physiological characteristics based on the measured one or more acoustic characteristics.

43. A method of determining a subject'"'"'s emotional state, comprising:

measuring one or more acoustic characteristics of acoustic output of a subject, wherein the measured one or more acoustic characteristics corresponds to a corresponding one or more measured acoustic segments of time within a particular segment of time, wherein the particular segment of time occurs after a stimulus; and

processing the measured one or more acoustic characteristics to determine an emotional state of the subject, wherein the emotional state of the subject corresponds to the particular segment of time.

44. A method of determining a subject'"'"'s emotional state, comprising:

measuring one or more physiological characteristics a subject, wherein the measured one or more physiological characteristics corresponds to a corresponding one or more measured physiological segments of time within a particular segment of time, wherein the particular segment of time occurs after a stimulus; and

processing the measured one or more physiological characteristics to determine an emotional state of the subject, wherein the emotional state of the subject corresponds to the particular segment of time.

45. An apparatus for determining a subject'"'"'s emotional state, comprising:

a physiological data acquisition unit, wherein the physiological data acquisition unit acquires physiological data of a subject;

an acoustic data acquisition unit, wherein the acoustic data acquisition unit acquires acoustic data of a subject; and

an information processing unit, wherein the information processing unit receives and processes the physiological data and/or acoustic data, and outputs an indication of the subject'"'"'s emotional state.

46. The apparatus according to claim 45, wherein the physiological data acquisition unit acquires physiological data in a non-contact manner.

47. The apparatus according to claim 45, wherein the acoustic data acquisition unit acquires acoustic data in a non-contact manner.

48. The apparatus according to claim 45, wherein the physiological data acquisition unit comprises a non-contact physiological data detection radar.

49. The apparatus according to claim 45, wherein the acoustic data acquisition unit comprises a transducer and a storage device.

50. The apparatus according to claim 49, wherein the transducer is a microphone.

51. The apparatus according to claim 45, where at least one of the physiological data acquired is selected from the group consisting of:

heartbeat, respiration, temperature, and galvanic skin response.

52. The apparatus according to claim 45, wherein the acoustic data acquired is speech.

53. The apparatus according to claim 45, wherein the apparatus measures physiological data and acoustic data simultaneously.