Apparatus and method for perceiving physical and emotional state

• US 6,656,116 B2
• Filed: 08/20/2001
• Issued: 12/02/2003
• Est. Priority Date: 09/02/2000
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Jay-woo, Lee, Hyoung-ki, Kim, Jeong-hwan, Bang, Seok-won
• Primary Examiner(s): Winakur, Eric F.
• Assistant Examiner(s): McCrosky, David J.
• Application Number:
  US09/931,767
• Publication Number:
  US 20030078505A1
• Time in Patent Office: 834 Days
• Field of Search: 600/300, 600/301, 600/26-28, 600/547, 600/549, 128/903, 128/905, 128/897, 128/898, 128/920, 128/923-925
• US Class Current: 600/300
• CPC Class Codes: A61B 5/0008 Temperature signals
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/0816 Measuring devices for examining respiratory frequency measuring frequency of electric signals G01R23/00
• CPC Class Codes: A61B 5/1455 using optical sensors, e.g. spectral photometrical oximeters
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/441 Skin evaluation, e.g. for skin disorder diagnosis
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

An apparatus and method for perceiving a physical and emotional state, which allows easy attachment to and detachment from a human body, and through which a bio-signal is simply detected, are provided. The apparatus includes a bio-signal detection part attached to a predetermined portion of a body for performing analog signal processing on at least one bio-signal detected from the body and outputting the processed bio-signal, and a bio-signal recognizing part for performing digital signal processing on the processed bio-signal received from the bio-signal detection part, perceiving the physical and emotional state from the result of the digital signal processing, and representing the physical and emotional state. Accordingly, the apparatus can be conveniently attached to a predetermined portion of a user'"'"'s body, a bio-signal transmitted wirelessly or through a wire can be easily detected, a physical and emotional state which is perceived based on the detected bio-signal can be reported to the user, and a rapidly changing emotional state or an emotional state which remains for a long time can be perceived in real time.

Claims

1. An apparatus for perceiving a physical and emotional state, comprising:

a bio-signal detection part for attaching to a predetermined portion of a user'"'"'s body, and for performing analog signal processing on at least one bio-signal detected from the body and outputting the processed bio-signal; and

a bio-signal recognizing part for performing digital signal processing on each processed bio-signal received from the bio-signal detection part, perceiving the physical and emotional state of the user from the result of the digital signal processing, and

reporting the physical and emotional state to the user.

2. The apparatus of claim 1, wherein the bio-signal detection part comprises:

a sensor for detecting the bio-signal from the body in analog form;

an analog signal processor for amplifying and filtering the analog bio-signal detected by the sensor; and

an analog-to-digital converter for converting the amplified and filtered analog bio-signal into a digital bio-signal, and

the bio-signal recognizing part comprises;

a digital signal processor for performing data processing on the digital bio-signal;

a controller for perceiving the physical and emotional state from data processed by the digital signal processor and outputting the perceived result as a control signal; and

a data output unit for outputting the data processed by the digital signal processor in predetermined form in response to the control signal.

3. The apparatus of claim 2, wherein the sensor comprises:

a blood pressure sensor for measuring blood pressure;

a skin temperature sensor for measuring skin temperature; and

a skin resistivity sensor for measuring skin resistivity.

4. The apparatus of claim 3, wherein the blood sensor is a Piezo sensor or a strain gauge pressure sensor.

5. The apparatus of claim 3, wherein the skin temperature sensor measures skin temperature using an electrode and a thermocouple.

6. The apparatus of claim 3, wherein the skin resistivity sensor measures skin resistivity using an electrode and a comparator.

7. The apparatus of claim 2, wherein the bio-signal detection part further comprises a wireless signal transmitter for converting the digital bio-signal output from the analog-to-digital converter into a wireless signal and transmitting the wireless signal, and the bio-signal recognizing part further comprises a wireless signal receiver for receiving the wireless signal and outputting it to the digital signal processor as the digital bio-signal.

8. The apparatus of claim 2, wherein the controller stores at least one reference value, compares the stored reference value with the data processed by the digital signal processor, and perceives the physical and emotional state from the result of the comparison.

9. The apparatus of claim 8, wherein the controller calculates an average value of at least one bio-signal detected by the sensor from data previously processed by the digital signal processor, selects one of the average value and the reference value in response to the difference between the average value and the stored reference value, and compares the selected average value or the selected reference value with the data currently processed by the digital signal processor, to perceive the physical and emotional state.

10. The apparatus of claim 1, wherein the bio-signal detection means is a strap-on type for removably attaching to a predetermined portion of a body.

11. The apparatus of claim 1, wherein the bio-signal recognizing part comprises logic circuitry for perceiving at least one emotional state when at least two comparison values exceed a respective predetermined threshold value associated with a comparison, wherein each comparison value is based on a sum of, or a difference between, one of the digitally processed bio-signals and a respective reference value or a respective average value of the bio-signal.

12. The apparatus of claim 11, wherein the reported emotional state comprises an indication representing the probability of the perceived emotional state.

13. A method of perceiving a physical and emotional state, comprising the steps of:

(a) performing analog signal processing on at least one bio-signal detected from a user'"'"'s body; and

(b) performing digital signal processing on the analog signal processed bio-signal, perceiving the physical and emotional state of the user from the result of the digital signal processing, and reporting the physical and emotional state to the user.

14. The method of claim 13, wherein step (a) comprises the steps of:

(a1) detecting the bio-signal from the body;

(a2) amplifying and filtering the detected analog bio-signal; and

(a3) converting the amplified and filtered analog bio-signal into a digital bio-signal, and

step (b) comprises the steps of;

(b1) performing data processing on the digital bio-signal;

(b2) perceiving the physical and emotional state from the resultant data of the data processing of step (b1); and

(b3) providing the data to the user in response to the perceived result.

15. The method of claim 14, wherein step (a) further comprises the step of converting the digital bio-signal into a wireless signal and transmitting the wireless signal after step (a3), and step (b) further comprises the step of receiving the wireless signal before step (b1).

16. The method of claim 14, wherein step (b2) comprises the steps of:

storing at least one reference value; and

comparing the stored reference value with the resultant data of step (b1) and perceiving the physical and emotional state based on the result of the comparison.

17. The method of claim 14, wherein step (b2) comprises the steps of:

storing at least one reference value;

calculating an average value of least one bio-signal detected in the step (a1) from data which is previously processed in the step (b1);

obtaining the difference between the average value and the reference value;

determining whether the difference is greater than a predetermined allowable error;

comparing the average value with the resultant data of step (b1) and perceiving the physical and emotional state from the result of the comparison, when it is determined that the difference is greater than the predetermined allowable error; and

comparing the reference value with the resultant data of step (b1) and perceiving the physical and emotional state from the result of the comparison, when it is determined that the difference is not greater than the predetermined allowable error.

18. The method of claim 13, wherein perceiving the emotional state in step (b) comprises:

calculating an average value of at least one of the digitally processed bio-signals detected from a body;

determining whether a difference between the average value and a reference value associated with the digitally processed bio-signal exceeds a predetermined threshold value;

perceiving the emotional state based on a comparison of the average value and the digitally processed bio-signal when the threshold is exceeded; and

perceiving the emotional state based on a comparison of the reference value and the digitally processed bio-signal when the threshold value is not exceeded.

19. The method of claim 18, wherein the perceived emotional state is based on at least two digitally processed bio-signals.

20. The method of claim 19, wherein perceiving the emotional state further comprises determining whether a result of each of the comparisons of each of the at least two digitally processed bio-signals with the respective associated reference or average value exceeds a second predetermined threshold value respectively associated with each bio-signal.

21. The method of claim 19, wherein the reporting the emotional state comprises providing an indication representing the probability of the perceived emotional state.

22. A method of perceiving an emotional state, comprising the steps of:

(c) extracting at least one bio-parameter representing a characteristic of a body from at least one bio-signal which is detected from the body of a user for a predetermined time; and

(d) determining an amount of variation of the extracted bio-parameter;

(e) determining a current emotional state using the determined amount of variation, wherein the value of the bio-parameter varies with a change in the emotional state; and

(f) reporting the determined emotional state to the user.

23. The method of claim 22, wherein in step (c), bio-parameters representing the frequency of an electrocardiogram, number of heartbeats and an electrocardiogram attendant upon respiration are extracted from a bio-signal having information as to blood pressure, and bio-parameters representing skin temperature and skin resistivity are extracted from a bio-signal having information as to skin temperature and a bio-signal having information as to skin resistivity.

24. The method of claim 23, wherein step (d) comprises the steps of:

(d11) perceiving happiness using the bio-parameters representing the number of heartbeats, the skin temperature and the frequency of an electrocardiogram;

(d12) perceiving stress using the bio-parameters representing the number of heartbeats and the frequency of an electrocardiogram;

(d13) perceiving sadness using the bio-parameters representing the number of heartbeats and the skin temperature;

(d14) perceiving boredom using the bio-parameters representing the skin temperature and the skin resistivity; and

(d15) perceiving anger using the bio-parameters representing the skin resistivity and the frequency of an electrocardiogram, and

the predetermined time is a minimum time taken for a changed emotion to be represented by a bio-signal when there is a change in the emotion.

25. The method of claim 23, wherein the step (d) comprises the steps of:

(d21) perceiving boredom using the bio-parameters representing the skin temperature, the skin resistivity, the frequency of an electrocardiogram and the electrocardiogram attendant upon respiration;

(d22) perceiving anger using the bio-parameters representing the skin resistivity and the electrocardiogram attendant upon respiration;

(d23) perceiving happiness using the bio-parameters representing the number of heartbeats, the frequency of an electrocardiogram and a rate of the respiration;

(d24) perceiving sadness using the bio-parameters representing the number of heartbeats and the skin resistivity; and

(d25) perceiving stress using the bio-parameters representing the skin resistivity, and

the predetermined time is longer than a minimum time taken for a changed emotion to be represented by a bio-signal when there is a change in the emotion, and the rate of the respiration is extracted from the bio-signal having information as to the blood pressure in step (c).

26. The method of claim 23, the bio-parameters representing the frequency of an electrocardiogram are a low frequency (LF) indicating a value obtained by integrating a power spectrum in a low frequency range over the Fast Fourier Transform (FFT) of the bio-signal having the information as to the blood pressure, a high frequency (HF) indicating a value obtained by integrating a power spectrum in a high frequency range over the FFT of the bio-signal having the information as to the blood pressure, and a LF/HF indicating a value obtained by dividing the LF by the HF;

the bio-parameters representing the number of heartbeats are a RR indicating an average value of the intervals between the positive peaks of the blood pressure and a heart rate (HR) indicating the number of heartbeats for a unit time period;

the bio-parameter representing the electrocardiogram attendant upon respiration is a respiratory sinus arrhythmia (RSA) indicating a value obtained by dividing the result of subtracting a minimum interval from a maximum interval among the intervals between positive peaks of the blood pressure by the minimum interval;

the bio-parameter representing the skin temperature is a SKT indicating an average skin temperature; and

the bio-parameters representing the skin resistivity are a skin conductive level (SCL) indicating an average of reciprocals of the values of the skin resistivity, a number of skin conductive response (N-SCR) indicating the number of negative slope zero crossings measured from a graph expressing the reciprocals of the values of the skin resistivity, and skin conductive response magnitude (SCRM) equal to the sum of reciprocals of values of skin resistivity between a negative slope zero crossing and a positive slope zero crossing.

27. The method of claim 26, wherein when the predetermined time is the minimum time, step (d11) comprises the steps of:

(d111) determining whether a HR value decreases by at least a HR threshold value;

(d112) determining whether a SKT value increases by at least a SKT threshold value if it is determined that the HR value decreases by at least the HR threshold value;

(d113) determining whether a LF value decreases by at least a LF threshold value or whether a LF/HF value decreases by at least a LF/HF threshold value if it is determined that the SKT value increases by at least the SKT threshold value;

(d114) determining that probability that the emotional state is happiness is a first predetermined percentage if it is determined that the LF value decreases by at least the LF threshold value or that the LF/HF value decreases by at least the LF/HF threshold value;

(d115) determining whether a RR value increases by at least a RR threshold value and going back to step (d114) if it is determined that the RR value does not increase by at least the RR threshold value; and

(d116) determining that probability that the emotional state is happiness is a second predetermined percentage greater than the first predetermined percentage if it is determined that the RR value increases by at least the RR threshold value.

28. The method of claim 27, wherein step (d12) comprises the steps of:

(d121) determining whether the LF value decreases by at least the LF threshold value if it is determined that the SKT value does not increase by at least the SKT threshold value in step (d112); and

(d122) determining that probability that the emotional state is stress is the second predetermined percentage if it is determined that the LF value decreases by at least the LF threshold value.

29. The method of claim 28, wherein step (d13) comprises the steps of:

(d131) determining whether the RR value increases by at least the RR threshold value if it is determined that the LF value does not decrease by at least the LF threshold value and that the LF/HF value does not decrease by at least the LF/HF threshold value in step (d113); and

(d132) determining that probability that the emotional state is sadness is the second predetermined percentage if it is determined that the RR value increases by at least the RR threshold value.

30. The method of claim 29, wherein step (d14) comprises the steps of:

(d141) determining whether a N-SCR value increases by at least a N-SCR threshold value and whether a SCRM value increases by at least a SCRM threshold value if it is determined that the HR value does not decrease by at least the HR threshold value in step (d111);

(d142) determining whether the SKT value increases by at least the SKT threshold value if it is determined that the N-SCR value increases by at least the N-SCR threshold value and that the SCRM value increases by at least the SCRM threshold value; and

(d143) determining that probability that the emotional state is boredom is the second predetermined percentage if it is determined that the SKT value increases by at least the SKT threshold value.

31. The method of claim 30, wherein step (d15) comprises the steps of:

(d151) determining whether the LF/HF value decreases by at least the LF/HF threshold value if it is determined that the SKT value does not increase by at least the SKT threshold value in step (d142) and going back to step (d142) if it is determined that the LF/HF value does not decrease by at least the LF/HF threshold value;

(d152) determining that probability that the emotional state is anger is the first predetermined percentage if it is determined that the LF/HF value decreases by at least the LF/HF threshold value;

(d153) determining whether a SCL value increases by at least a SCL threshold value and going back to step (d152) if it is determined that the SCL value does not increase by at least the SCL threshold value; and

(d154) determining that probability that the emotional state is anger is the second predetermined percentage if it is determined that the SCL value increases by at least the SCL threshold value.

32. The method of claim 31, wherein step (d) further comprises the step of determining that no dominant emotional state exists if it is determined that the LF value does not decrease by at least the LF threshold value in step (d121), if it is determined that the N-SCR value does not increase by at least the N-SCR threshold value or that the SCRM value does not increase by at least the SCRM threshold value in step (d141), or if it is determined that the RR value does not increase by at least the RR threshold value in step (d131).

33. The method of claim 26, wherein when the predetermined time is longer than the minimum time, step (d21) comprises the steps of:

(d211) determining whether a RSA value decreases by at least a RSA threshold value, whether a SCL value increases by at least a SCL threshold value, and whether a SCRM value increases by at least a SCRM threshold value;

(d212) determining whether a N-SCR value increases by at least a N-SCR threshold value if it is determined that the RSA value decreases by at least the RSA threshold value, that the SCL value increases by at least the SCL threshold value, and that the SCRM value increases by at least the SCRM threshold value;

(d213) determining whether a SKT value increases by at least a SKT threshold value if it is determined that the N-SCR value increases by at least the N-SCR threshold value;

(d214) determining that probability that the emotional state is boredom is a third predetermined percentage if it is determined that the SKT value increases by at least the SKT threshold value;

(d215) determining whether a HF value decreases by at least a HF threshold value and going back to step (d214) if it is determined that the HF value does not decrease by at least the HF threshold value;

(d216) determining that probability that the emotional state is boredom is a fourth predetermined percentage greater than the third predetermined percentage if it is determined that the HF value decreases by at least the HF threshold value;

(d217) determining whether a LF value decreases by at least a LF threshold value and going back to step (d216) if it is determined that the LF does not decrease by at least the LF threshold value; and

(d218) determining that probability that the emotional state is boredom is a fifth predetermined percentage greater than the fourth predetermined percentage if it is determined that the LF decreases by at least the LF threshold value.

34. The method of claim 33, wherein step (d22) comprises the steps of:

(d221) determining that probability that the emotional state is anger is the fifth predetermined percentage if it is determined that the N-SCR value does not increase by at least the N-SCR threshold value in step (d212); and

(d222) determining that probability that the emotional state is anger is the third predetermined percentage if it is determined that the SKT value does not increase by at least the SKT threshold value in step (d213).

35. The method of claim 34, wherein step (d23) comprises the steps of:

(d231) determining whether a HR value decreases by at least a HR threshold value if it is determined that the RSA value does not decrease by at least the RSA threshold value, that the SCL value does not increase by at least the SCL threshold value, or that the SCRM value does not increase by at least the SCRM threshold value in step (d211);

(d232) determining whether a RESP value increases by at least a RESP threshold value if it is determined that the HR value decreases by at least the HR value;

(d233) determining that probability that the emotional state is happiness is the third predetermined percentage if it is determined that the RESP value increases by at least the RESP threshold value;

(d234) determining whether a LF/HF value decreases by at least a LF/HF threshold value and going back to step (d233) if it is determined that the LF/HF value does not decrease by at least the LF/HF threshold value;

(d235) determining that probability that the emotional state is happiness is the fourth predetermined percentage if it is determined that the LF/HF value decreases by at least the LF/HF threshold value;

(d236) determining whether the LF value decreases by at least the LF threshold value and going back to step (d235) if it is determined that the LF value does not decrease by at least the LF threshold value; and

(d237) determining that probability that the emotional state is happiness is the fifth predetermined percentage if it is determined that the LF value decreases by at least the LF threshold value.

36. The method of claim 35, wherein step (d24) comprises the steps of:

(d241) determining whether the SCL value decreases by at least the SCL threshold value if it is determined that the RESP value does not increase by at least the RESP threshold value in step (d232); and

(d242) determining that probability that the emotional state is sadness is the fifth predetermined percentage if it is determined that the SCL value decreases by at least the SCL threshold value.

37. The method of claim 36, wherein step (d25) comprises the steps of:

(d251) determining whether the N-SCR value increases by at least the N-SCR threshold value if it is determined that the HR value does not decrease by at least the HR threshold value in step (d231); and

(d252) determining that probability that the emotional state is stress is the fourth predetermined percentage if it is determined that the N-SCR value increases by at least the N-SCR threshold value.

38. The method of claim 37, wherein step (d) further comprises the step of determining that no dominant emotional state exists if it is determined that the SCL value does not decrease by at least the SCL threshold value in step (d241) or if it is determined that the N-SCR value does not increase by at least the N-SCR threshold value in step (d251).

39. The method of claim 22, wherein the reporting the emotional state to the user comprises providing the user with an indication representing the probability of the determined emotional state.

40. The method of claim 22, wherein:

step (d) comprises the steps of;

calculating an average value of at least one of the extracted bio-parameters;

determining whether a difference between the average value and a reference value associated with the extracted bio-parameter exceeds a predetermined threshold value; and

step (e) comprises the steps of;

determining the emotional state based on a comparison of the average value and the extracted bio-parameter when the threshold is exceeded; and

determining the emotional state based on a comparison of the reference value and the extracted bio-parameter when the threshold value is not exceeded.

41. The method of claim 40, wherein the determined emotional state is based on at least two extracted bio-parameters.

42. The method of claim 41, wherein the reporting the determined emotional state to the user comprises providing the user with an indication representing the probability of the determined emotional state.

43. An apparatus for perceiving a physical and emotional state, comprising:

a bio-signal detection part for attaching to a predetermined portion of a body, and for performing analog signal processing on at least one bio-signal detected from the body and for outputting the processed bio-signal; and

a bio-signal recognizing part for performing digital signal processing on the processed bio-signal received from the bio-signal detection part, perceiving the physical and emotional state from the result of the digital signal processing, and reporting the physical and emotional state, wherein the reported emotional state comprises an indication representing the probability of the perceived emotional state.

44. A method of perceiving a physical and emotional state, comprising the steps of:

(a) performing analog signal processing on at least one bio-signal detected from a body; and

(b) performing digital signal processing on the analog signal processed bio-signal, perceiving the physical and emotional state from the result of the digital signal processing, and reporting the physical and emotional state, wherein reporting the emotional state comprises providing to a user an indication representing the probability of the perceived emotional state.

45. An apparatus for perceiving a physical and emotional state, comprising:

a bio-signal detection part for attaching to a predetermined portion of a body, and for performing analog signal processing on at least one bio-signal detected from the body and for outputting the processed bio-signal; and

a bio-signal recognizing part for performing digital signal processing on each processed bio-signal received from the bio-signal detection part, storing at least one reference value associated with the bio-signal, and perceiving the physical and emotional state from the result of the digital signal processing, wherein the bio-signal recognizing part comprises;

logical circuitry for calculating an average value of each digital signal processed signal from data previously processed by the bio-signal recognizing part, selecting one of the average value and the reference value based on a difference between the average value and the stored reference value, and for comparing the selected average value or the selected reference value with data currently digitally processed by the bio-signal recognizing part to perceive the physical and emotional state; and

a reporting mechanism for reporting the physical and emotional state.

46. A method of perceiving a physical and emotional state, comprising the steps of:

(a) performing analog signal processing on at least one bio-signal detected from a body; and

(b) performing digital signal processing on the analog signal processed bio-signal, perceiving the physical and emotional state from the result of the digital signal processing, and reporting the physical and emotional state, wherein perceiving the emotional state comprises;

calculating an average value of at least one of the digitally processed bio-signals detected from a body;

determining whether a difference between the average value and a reference value associated with the digitally processed bio-signal exceeds a predetermined threshold value;

perceiving the emotional state based on a comparison of the average value and the digitally processed bio-signal when the threshold is exceeded;

perceiving the emotional state based on a comparison of the reference value and the digitally processed bio-signal when the threshold value is not exceeded.