System and method for recognizing user's emotional state using short-time monitoring of physiological signals

• US 20030139654A1
• Filed: 01/23/2003
• Published: 07/24/2003
• Est. Priority Date: 01/23/2002
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Kyung-Hwan, Bang, Seok-Won, Kong, Dong-Geon
• Granted Patent: US 7,547,279 B2
• Time in Patent Office: Days
• US Class Current: 600/300
• CPC Class Codes: A61B 5/0002 Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network A61B5/07 takes precedence; transmission systems for measured values G08C; transmission H04B; transmission of digital information H04L; wireless communication networks H04W
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: Y10S 128/92 Computer assisted medical diagnostics

Abstract

Disclosed is an emotion recognition apparatus and method thereof which enables perception of a user'"'"'s emotional state by monitoring one or more of his or her physiological signals. The emotion recognition apparatus comprises a feature analyzer adapted to analyze features of the physiological signals acquired from a user and generate feature values, a subtractor adapted to obtain differences between the feature values generated by the feature analyzer and feature values used as a standard to perceive an emotional state of the user, and an emotion classifier adapted to analyze the differences obtained by the subtractor and classify an emotion into a plurality of emotional categories so as to perceive an emotional category exhibiting the greatest intensity to be the emotional state of the user.

Claims

1. An emotion recognition system, comprising:

a physiological signal acquiring unit adapted to acquire one or more physiological signals from a user and wirelessly transmit the acquired physiological signals, the physiological signal acquiring unit being worn on the body of the user; and

an emotion recognition unit adapted to receive the physiological signals transmitted from the physiological signal acquiring unit and monitor them for a short time period to perceive an emotional state of the user.

2. The emotion recognition system as claimed in claim 1, wherein the physiological signal acquiring unit comprises:

one or more sensors adapted to non-invasively sense one or more physiological signals of the user;

an amplifier-filter adapted to amplify and filter the one or more physiological signals applied thereto from the one or more sensors and output the one or more amplified and filtered signals;

a multiplexer adapted to classify the amplified and filtered signals applied thereto from the amplifier-filter with respect to each physiological signal and output the classified physiological signals in the form of analog signals;

an A/D converter adapted to convert the classified analog physiological signals applied thereto from the mulitplexer into the form of digital signals and output the converted digital signals; and

an RF transmitter adapted to wirelessly transmit the converted digital signals applied thereto from the A/D converter to the emotion recognition unit in the form of RF signals.

3. The emotion recognition system as claimed in claim 1, wherein the physiological signals comprise photoplethysmogram (PPG), electrocardiogram (ECG), electrodermal activity (EDA) and skin temperature (SKT) signals.

4. The emotion recognition system as claimed in claim 1, wherein the emotion recognition unit is an arithmetic and logical unit (ALU) having an algorithm stored therein for analyzing the physiological signals of the user transmitted from the physiological signal acquiring unit and recognizing the user'"'"'s emotional state.

5. The emotion recognition system as claimed in claim 1, wherein the emotion recognition unit comprises:

a feature analyzer adapted to analyze features of the physiological signals and generate feature values corresponding to the analyzed results;

a subtractor adapted to obtain differences between the feature values of the analyzed results generated from the feature analyzer and feature values used as a standard to perceive an emotional state of the user; and

an emotion classifier adapted to analyze the differences obtained by the subtractor to classify an emotion into a plurality of emotional categories so as to perceive an emotional category exhibiting the greatest intensity to be an emotional state of the user.

6. The emotion recognition system as claimed in claim 5, wherein the feature analyzer comprises:

a heart rate analyzer adapted to receive either one of an ECG signal and a PPG signal to detect a heart beat signal and extract the feature values associated with the detected heart beat signal;

a skin conductance response (SCR) analyzer adapted to receive an EDA signal to extract the feature values associated with an SCR; and

an SKT analyzer adapted to receive an SKT signal to extract feature values associated with SKT.

7. The emotion recognition system as claimed in claim 6, wherein the heart rate analyzer comprises:

a heart rate detector adapted to receive any one of the ECG and PPG signals to detect a heart beat signal and adapted to convert the detected heart beat signal into time series of heart rate variability (HVR);

a spectrum analyzer adapted to analyze a spectrum of the heart beat signal in response to the time series of HRV; and

a mean/standard deviation calculator adapted to calculate a mean value and a standard deviation value of the detected heart beat signal in response to the time series of HRV.

8. The emotion recognition system as claimed in claim 7, wherein the heart rate detector comprises:

a bandpass filter adapted to pass only frequency components within the same frequency band as the ECG signal of input signals applied to the heart rate analyzer;

a median filter (MF) adapted to estimate a low-frequency noise component existing in the bandpass-filtered resultant signal;

an adder adapted to add an inverse number of the median-filtered resultant signal to the bandpass-filtered resultant signal to eliminate the low-frequency noise component from the bandpass-filtered resultant signal; and

a Teager energy operator (TEO) section adapted to detect the heart beat signal by applying a TEO to an output signal from the adder.

9. The emotion recognition system as claimed in claim 7, wherein the heart rate detector comprises:

a bandpass filter adapted to pass only frequency components within the same frequency band as the PPG signal of input signals applied to the heart rate analyzer;

an MF adapted to estimate a low-frequency noise component existing in the bandpass-filtered resultant signal;

an adder adapted to add an inverse number of the median-filtered resultant signal to the bandpass-filtered resultant signal to eliminate the low-frequency noise component from the bandpass-filtered resultant signal;

a matched filter adapted to extract the heart beat signal from a signal output from the adder; and

a zero clipper adapted to perform a zero clipping process with respect to the extracted heart beat signal from the matched filter.

10. The emotion recognition system as claimed in claim 5, wherein the subtractor uses feature values representing an ordinary emotional state of the user as a standard to perceive the emotional state of the user.

11. The emotion recognition system as claimed in claim 5, wherein the emotion classifier comprises:

a support vector machine (SVM) classifier adapted to analyze the differences obtained by the subtractor and classify an emotion of the user into a plurality of emotional categories to output the result of classifying the emotion in the form of a plurality of intensities each corresponding to an associated one of a plurality of emotions; and

a comparator adapted to compare the plurality of intensities with respect to the plurality of emotions classified by the SVM classifier with each other so as to perceive an emotion exhibiting the greatest intensity among the classified emotions to be a resultant emotional state of the user and output it.

12. The emotion recognition system as claimed in claim 11, wherein the emotion classifier further comprises a database adapted to store a plurality of emotional data used to train the SVM classifier and the trained result of the SVM classifier according to the plurality of emotional data.

13. The emotion recognition apparatus, comprising:

a feature analyzer adapted to analyze features of the physiological signals acquired from a user and generate feature values corresponding to the analyzed results;

a subtractor adapted to obtain differences between the feature values of the analyzed results generated from the feature analyzer and feature values used as a standard to perceive an emotional state of the user; and

an emotion classifier adapted to analyze the differences obtained by the subtractor to classify an emotion into a plurality of emotional categories so as to perceive an emotional category exhibiting the greatest intensity to be the emotional state of the user.

14. The emotion recognition apparatus as claimed in claim 13, wherein the physiological signals comprise PPG, ECG, EDA and SKT signals.

15. The emotion recognition apparatus as claimed in claim 14, wherein the feature analyzer comprises:

a heart rate analyzer adapted to receive either one of the ECG and PPG signals to detect a heart beat signal and extract the feature values associated with the detected heart beat signal;

an SCR analyzer adapted to receive the EDA signal to extract the feature values associated with SCR; and

an SKT analyzer adapted to receive the SKT signal to extract the feature values associated with an SKT.

16. The emotion recognition apparatus as claimed in claim 15, wherein the heart rate analyzer comprises:

a heart rate detector adapted to receive either one of the ECG and PPG signals to detect a heart beat signal and adapted to convert the detected heart beat signal into time series of HVR;

a spectrum analyzer adapted to analyze a spectrum of the heart beat signal in response to the time series of HRV; and

a mean/standard deviation calculator adapted to calculate a mean value and a standard deviation value of the detected heart beat signal in response to the time series of HRV.

17. The emotion recognition apparatus as claimed in claim 16, wherein the heart rate detector comprises:

a bandpass filter adapted to pass only frequency components within the same frequency band as the ECG signal of input signals applied to the heart rate analyzer;

an MF adapted to estimate a low-frequency noise component existing in the bandpass-filtered resultant signal;

an adder adapted to add an inverse number of the median-filtered resultant signal to the bandpass-filtered resultant signal to eliminate the low-frequency noise component from the bandpass-filtered resultant signal; and

a TEO section adapted to detect the heart beat signal by applying a TEO to an output signal from the adder.

18. The emotion recognition apparatus as claimed in claim 16, wherein the heart rate detector comprises:

a bandpass filter adapted to pass only frequency components within the same frequency band as the PPG signal of input signals applied to the heart rate analyzer;

an MF adapted to estimate a low-frequency noise component existing in the bandpass-filtered resultant signal;

an adder adapted to add an inverse number of the median-filtered resultant signal to the bandpass-filtered resultant signal to eliminate the low-frequency noise component from the bandpass-filtered resultant signal;

a matched filter adapted to extract the heart beat signal from a signal output from the adder; and

a zero clipper adapted to perform a zero clipping process with respect to the extracted heart beat signal from the matched filter.

19. The emotion recognition apparatus as claimed in claim 13, wherein the subtractor uses feature values representing an ordinary emotional state of the user as a standard to perceive the emotional state of the user.

20. The emotion recognition apparatus as claimed in claim 13, wherein the emotion classifier comprises:

an SVM classifier adapted to analyze the differences obtained by the subtractor and classify an emotion of the user into a plurality of emotional categories to output the classified results in the form of a plurality of intensities each corresponding to an associated one of a plurality of emotions; and

a comparator adapted to compare the plurality of intensities with respect to the plurality of emotions classified by the SVM classifier with each other so as to perceive an emotion exhibiting the greatest intensity among the classified emotions to be a resultant emotional state of the user and output it.

21. The emotion recognition apparatus as claimed in claim 20, wherein the emotion classifier further comprises a database adapted to store a plurality of emotional data used to train the SVM classifier and the trained result of the SVM classifier according to the plurality of emotional data.

22. An emotion recognition method, comprising the acts of:

receiving one or more physiological signals from a user;

analyzing the received physiological signals and extracting a plurality of feature values to be used for recognition of an emotional state of the user in response to the result of analyzing the received physiological signals;

calculating differences between the extracted plurality of feature values and predetermined feature values used as a standard to perceive an emotional state of the user;

classifying an emotional state of the user into a plurality of emotional categories in response to the calculated differences; and

selecting an emotion exhibiting the greatest intensity among the classified emotional categories as a resultant emotional state of the user and outputting the selected emotion.

23. The emotion recognition method as claimed in claim 22, wherein the physiological signals comprise PPG, ECG, EDA and SKT signals which are acquired non-invasively.

24. The emotion recognition method as claimed in claim 22, wherein the feature values used as a standard to perceive the emotional state of the user are feature values representing an ordinary emotional state of the user.

25. The emotion recognition method as claimed in claim 22, wherein the classifying step uses an SVM classifier for classifying the emotional state of the user into the plurality of emotional categories based on a statistical learning theory.