Determining Emotions Using Camera-Based Sensing

• US 20180314879A1
• Filed: 04/11/2018
• Published: 11/01/2018
• Est. Priority Date: 05/01/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Khwaja, Ayesha, Young, James, Wortham, Cody, Sadi, Sajid, Jain, Jawahar
• Granted Patent: US 10,874,309 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 2562/04 Arrangements of multiple sensors of the same type
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/02007 Evaluating blood vessel condition, e.g. elasticity, compliance
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/02108 from analysis of pulse wave characteristics
• CPC Class Codes: A61B 5/02125 of pulse wave propagation time
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/0285 Measuring or recording phase velocity of blood waves
• CPC Class Codes: A61B 5/0295 using plethysmography, i.e. measuring the variations in the volume of a body part as modified by the circulation of blood therethrough, e.g. impedance plethysmography
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/349 Detecting specific parameters of the electrocardiograph cycle
• CPC Class Codes: A61B 5/489 Blood vessels
• CPC Class Codes: A61B 5/7203 for noise prevention, reduction or removal
• CPC Class Codes: A61B 5/7221 Determining signal validity, reliability or quality A61B5/7207 and A61B5/7217 take precedence
• CPC Class Codes: A61B 5/7235 Details of waveform analysis A61B5/349 takes precedence
• CPC Class Codes: A61B 5/7242 using integration
• CPC Class Codes: A61B 5/7257 using Fourier transforms
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/7278 : Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06V 40/15 : Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/174 : Facial expression recognition
• CPC Class Codes: G06V 40/176 : Dynamic expression

Abstract

In one embodiment, a computer-readable non-transitory storage medium embodies software that is operable when executed to, in real time, capture, by a single sensor, a number of images of a user; determine, based on the number of images, one or more short-term cardiological signals of the user during a period of time; estimate, based on the cardiological signals, a first short-term emotional state of the user; determine, based on the number of images, one or more short-term neurological signals of the user during the period of time; estimate, based on the neurological signals, a second short-term emotional state of the user; compare the first estimated emotional state to the second estimated emotional state; and in response to a determination that the first estimated emotional state corresponds to the second estimated emotional state, determine a short-term emotion of the user during the period of time.

Claims

1. One or more computer-readable non-transitory storage media embodying software that is operable when executed to, in real time:

capture, by a single sensor, a plurality of images of a user;

determine, based on the plurality of images, one or more short-term cardiological signals of the user during a period of time;

estimate, based on the cardiological signals, a first short-term emotional state of the user;

determine, based on the plurality of images, one or more short-term neurological signals of the user during the period of time;

estimate, based on the neurological signals, a second short-term emotional state of the user;

compare the first estimated emotional state to the second estimated emotional state; and

in response to a determination that the first estimated emotional state corresponds to the second estimated emotional state, determine, based on the first estimated emotional state and second estimated emotional state, a short-term emotion of the user during the period of time.

2. The media of claim 1, wherein the software is further operable to:

detect a face of the user in the plurality of images;

determine, based on photoplethysmogram (PPG) data obtained from the plurality of images, one or more of the short-term cardiological signals of the user;

determine, based on facial action coding system (FACS) data obtained from the plurality of images, one or more of the short-term neurological signals of the user;

determine, one or more neuroanatomical signals of the user based on the plurality of images; and

confirm the short-term emotion of the user based on a consistency between the neuroanatomical signals and the short-term emotion.

3. The media of claim 2, wherein the FACS data is based on detected characteristics of the user'"'"'s eye brow, lip width, mouth, teeth visibility, chin lines, nasolabial lines, nose lines, or cheek-based impression of smile lines.

4. The media of claim 2, wherein one or more of the neuroanatomical signals are detected physical attributes of the user.

5. The media of claim 1, wherein the software is further operable to determine one or more of the cardiological signals using a linear model based on a sympathovagal balance (SVB) of the user.

6. The media of claim 5, wherein the software is further operable to:

determine one or more of the cardiological signals using a non-linear statistical model based on heartrate dynamics; and

determine that the short-term emotion corresponds to a positive emotion based on concurrently sensing the SVB and heartrate dynamics and based on determining that the SVB and the heartrate dynamics are consistent with positive emotion.

7. The media of claim 1 wherein the determination of the cardiological and neurological signals are performed concurrently.

8. A method comprising, in real time:

capturing, by a single sensor, a plurality of images of a user;

determining, based on the plurality of images, one or more short-term cardiological signals of the user during a period of time;

estimating, based on the cardiological signals, a first short-term emotional state of the user;

determining, based on the plurality of images, one or more short-term neurological signals of the user during the period of time;

estimating, based on the neurological signals, a second short-term emotional state of the user;

comparing the first estimated emotional state to the second estimated emotional state; and

in response to a determination that the first estimated emotional state corresponds to the second estimated emotional state, determining, based on the first estimated emotional state and second estimated emotional state, a short-term emotion of the user during the period of time.

9. The method of claim 8, further comprising:

detecting a face of the user in the plurality of images;

determining, based on photoplethysmogram (PPG) data obtained from the plurality of images, one or more of the short-term cardiological signals of the user;

determining, based on facial action coding system (FACS) data obtained from the plurality of images, one or more of the short-term neurological signals of the user;

determining, one or more neuroanatomical signals of the user based on the plurality of images; and

confirming the short-term emotion of the user based on a consistency between the neuroanatomical signals and the short-term emotion.

10. The method of claim 9, wherein the FACS data is based on detected characteristics of the user'"'"'s eye brow, lip width, mouth, teeth visibility, chin lines, nasolabial lines, nose lines, or cheek-based impression of smile lines.

11. The method of claim 9, wherein one or more of the neuroanatomical signals are detected physical attributes of the user.

12. The method of claim 8, wherein the determination of one or more of the cardiological signals comprises using a linear model based on a sympathovagal balance (SVB) of the user.

13. The method of claim 12, further comprising:

determining one or more of the cardiological signals using a non-linear statistical model based on heartrate dynamics; and

determining that the short-term emotion corresponds to a positive emotion based on concurrently sensing the SVB and heartrate dynamics and based on determining that the SVB and the heartrate dynamics are consistent with positive emotion.

14. The method of claim 8, further comprising concurrently performing the determination of the cardiological and neurological signals.

15. A system comprising:

one or more processors; and

a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to, in real-time;

capture, by a single sensor, a plurality of images of a user;

determine, based on the plurality of images, one or more short-term cardiological signals of the user during a period of time;

estimate, based on the cardiological signals, a first short-term emotional state of the user;

determine, based on the plurality of images, one or more short-term neurological signals of the user during the period of time;

estimate, based on the neurological signals, a second short-term emotional state of the user;

compare the first estimated emotional state to the second estimated emotional state; and

in response to a determination that the first estimated emotional state corresponds to the second estimated emotional state, determine, based on the first estimated emotional state and second estimated emotional state, a short-term emotion of the user during the period of time.

16. The system of claim 15, wherein the processors are further operable to:

detect a face of the user in the plurality of images;

determine, based on photoplethysmogram (PPG) data obtained from the plurality of images, one or more of the short-term cardiological signals of the user;

determine, based on facial action coding system (FACS) data obtained from the plurality of images, one or more of the short-term neurological signals of the user;

determine, one or more neuroanatomical signals of the user based on the plurality of images; and

confirm the short-term emotion of the user based on a consistency between the neuroanatomical signals and the short-term emotion.

17. The system of claim 16, wherein the FACS data is based on detected characteristics of the user'"'"'s eye brow, lip width, mouth, teeth visibility, chin lines, nasolabial lines, nose lines, or cheek-based impression of smile lines.

18. The system of claim 16, wherein one or more of the neuroanatomical signals are detected physical attributes of the user.

19. The system of claim 15, wherein the processors are further operable to determine one or more of the cardiological signals using a linear model based on a sympathovagal balance (SVB) of the user.

20. The system of claim 15, wherein the processors further operable to:

determine one or more of the cardiological signals using a non-linear statistical model based on heartrate dynamics; and

determine that the short-term emotion corresponds to a positive emotion based on concurrently sensing the SVB and heartrate dynamics and based on determining that the SVB and the heartrate dynamics are consistent with positive emotion.