Detecting physiological responses based on multispectral data from head-mounted cameras

• US 10,076,250 B2
• Filed: 12/06/2017
• Issued: 09/18/2018
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Thieberger, Gil, Frank, Ari M
• Primary Examiner(s): Porta, David
• Assistant Examiner(s): BOOSALIS, FANI POLYZOS
• Application Number:
  US15/833,025
• Publication Number:
  US 20180092549A1
• Time in Patent Office: 286 Days
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: G01J 2005/0077 Imaging
• CPC Class Codes: G01J 5/0025 Living bodies ear thermometers G01J5/0011; detecting, measuring or recording for diagnostic purposes A61B5/00
• CPC Class Codes: G01J 5/12 using thermoelectric elements, e.g. thermocouples
• CPC Class Codes: G01J 5/485 Temperature profile
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/55 : Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Described herein are systems and methods for detecting a physiological response based on multispectral data. In one embodiment, a system includes an inward-facing head-mounted thermal camera (CAM) that takes thermal measurements of a first region of interest (TH_ROI1) on a user'"'"'s face, and an inward-facing head-mounted visible-light camera (VCAM) that takes images of a second region of interest (IM_ROI2) on the face. The first and second regions of interest overlap, and the system includes a computer that detects the physiological response based on TH_ROI1, IM_ROI2, and a model. Optionally, the model was trained based on previous TH_ROI1 and IM_ROI2 of the user taken during different days. Optionally, the physiological response is indicative of an occurrence of an emotional state of the user, such as joy, fear, sadness or anger.

Claims

1. A system configured to detect a physiological response, comprising:

an inward-facing head-mounted thermal camera (CAM) configured to take thermal measurements of a first region of interest (TH_ROI1) on a user'"'"'s face;

an inward-facing head-mounted visible-light camera (VCAM) configured to take images of a second region of interest (IM_ROI2) on the face;

wherein the first region of interest (ROI₁) and the second region of interest (ROI₂) overlap; and

a computer configured to detect the physiological response based on TH_ROI1 and IM_ROI2.

2. The system of claim 1, wherein the computer is further configured to detect the physiological response by generating feature values based on TH_ROI1 and IM_ROI2, and to utilize the utilizing a machine learning-based model to calculate a value indicative of the extent of the physiological response based on the feature values.

3. The system of claim 2, wherein the machine learning-based model was trained based on previous TH_ROI1 and IM_ROI2 of the user taken during different days, and the physiological response is indicative of an occurrence of at least one of the following emotional states of the user:

joy, fear, sadness, and anger.

4. The system of claim 2, wherein the machine learning-based model was trained based on previous TH_ROI1 and IM_ROI2 of the user taken during different days, and the physiological response is indicative of an occurrence of one or more of the following:

stress, mental workload, an allergic reaction, a headache, dehydration, intoxication, and a stroke.

5. The system of claim 2, wherein the machine learning-based model was trained based on previous TH_ROI1 and IM_ROI2 taken during different days, and the physiological response is a physiological signal selected from among:

a heart rate, a breathing rate, and an extent of frontal lobe brain activity.

6. The system of claim 1, wherein ROI₂ covers at least half of ROI₁, each of CAM and VCAM weighs below 10 g, is physically coupled to a frame configured to be worn on the user'"'"'s head, and is located less than 15 cm from the face;

and further comprising a second thermal camera that is located less than 15 cm from the face, is physically coupled to the frame, and is configured to take thermal measurements of a third region of interest (TH_ROI3) on the face;

the center of ROI₁ is to the right of the center of the third region of interest (ROI₃), and the symmetric overlapping between ROI₁ and ROI₃ is above 50%;

whereby to detect the physiological response, the computer is configured to account for facial thermal asymmetry, based on a difference between TH_ROI1 and TH_ROI3.

7. The system of claim 1, wherein ROI₂ covers at least half of ROI₁, each of CAM and VCAM weighs below 10 g, is physically coupled to a frame configured to be worn on the user'"'"'s head, and is located less than 15 cm from the face;

and further comprising a second visible-light camera (VCAM2) that is physically coupled to the frame and is configured to take images of a third ROI (IM_ROI3) on the face;

wherein VCAM and VCAM2 are located at least 0.5 cm to the right and to the left of the vertical symmetry axis that divides the face, respectively, and the symmetric overlapping between ROI₂ and ROI₃ is above 50%; and

wherein the computer is configured to detect the physiological response also based on IM_ROI3.

8. The system of claim 1, wherein the physiological response is emotional response, the computer is further configured to calculate, based on IM_ROI2, facial skin color changes (FSCC), and to detect the emotional response of the user based on TH_ROI1 and FSCC;

wherein, on average, detections of the emotional responses based on both TH_ROI1 and FSCC are more accurate than detections of the emotional responses based on either TH_ROI1 or FSCC.

9. The system of claim 1, wherein the physiological response is emotional response, the computer is further configured to identify facial expressions from IM_ROI2, and to detect the emotional response of the user based on TH_ROI1 and the identified facial expressions;

wherein, on average, detections of the emotional response based on both TH_ROI1 and the identified facial expressions are more accurate than detections of the emotional response based on either TH_ROI1 or the identified facial expressions.

10. The system of claim 1, wherein the computer is further configured to generate an avatar of the user based on IM_ROI2, and to modify the avatar to shows a synthesized facial expression that corresponds to an emotional response detected based on TH_ROI1, which is not manifested in the user'"'"'s facial expression.

11. The system of claim 1, wherein the computer is further configured to utilize a machine learning-based model to detect the physiological response;

and wherein the machine learning-based model was trained on;

samples generated based on a first set of TH_ROI1 and IM_ROI2 taken after cosmetics were applied to a portion of the overlapping region between ROI₁ and ROI₂, and other samples generated based on a second set of TH_ROI1 and IM_ROI2 taken while the overlapping region was bare of cosmetics;

whereby utilizing the machine learning-based model enables the computer to account for presence of cosmetics on the overlapping region.

12. The system of claim 1, wherein the computer is further configured to utilize a machine learning-based model to detect the physiological response;

and wherein the machine learning-based model was trained on;

samples generated from a first set of TH_ROI1 and IM_ROI2 taken while sweat was detectable on a portion of the overlapping region between ROI₁ and ROI₂, and additional samples generated from a second set of TH_ROI1 and IM_ROI2 taken while sweat was not detectable on the overlapping region;

whereby utilizing the machine learning-based model enables the computer to account for sweat on the overlapping region.

13. The system of claim 1, wherein the computer is further configured to utilize a machine learning-based model to detect the physiological response;

and wherein the machine learning-based model was trained on;

samples generated from a first set of TH_ROI1 and IM_ROI2 taken while hair density on a portion of the overlapping region between ROI₁ and ROI₂ was at a first level, and additional samples generated from a second set of TH_ROI1 and IM_ROI2 taken while hair density on the portion of the overlapping region between was at a second level higher than the first level;

whereby utilizing the machine learning-based model enables the computer to account for hair on the portion of the overlapping region.

14. The system of claim 1, wherein the computer is further configured to utilize a machine learning-based model to detect the physiological response;

and wherein the machine learning-based model was trained on;

samples generated from a first set of TH_ROI1 and IM_ROI2 taken while skin inflammation was detectable on a portion of the overlapping region between ROI₁ and ROI₂, and additional samples generated from a second set of TH_ROI1 and IM_ROI2 taken while skin inflammation was not detectable on the overlapping region;

whereby utilizing the machine learning-based model enables the computer to account for skin inflammation on the overlapping region.

15. The system of claim 1, wherein the computer is further configured to utilize a machine learning-based model to detect the physiological response;

and wherein the machine learning-based model was trained on;

samples generated from a first set of TH_ROI1 and IM_ROI2 taken while detecting that the user touches a portion of the overlapping region between ROI₁ and ROI₂, and additional samples generated from a second set of TH_ROI1 and IM_ROI2 taken while detecting that the user does not touch the overlapping region;

whereby utilizing the machine learning-based model enables the computer to account for touching the overlapping region.

16. The system of claim 1, wherein the computer is further configured to:

(i) identify, based on IM_ROI2, occurrences of one or more of the following disruptive activities that alter TH_ROI1;

talking, eating, and drinking, (ii) generate feature values based on the identified disruptive activity, and (iii) utilize the machine learning-based model to detect the physiological response based on the feature values and TH_ROI1.

17. The system of claim 1, wherein ROI₁ and ROI₂ are on the mouth, and IM_ROI2 are indicative of a change in a facial expression during a certain period that involves a transition from a facial expression in which the lips are in contact to a facial expression with an open mouth;

whereby a detection of the physiological response based on TH_ROI1 taken during the certain period involves attributing a change in TH_ROI1 to opening the mouth rather than a change in the temperature of the lips.

18. The system of claim 1, wherein ROI₁ and ROI₂ are on the user'"'"'s forehead located about 1 cm above at least one of the user'"'"'s eyebrows, and IM_ROI2 are indicative of a change in a facial expression during a certain period that involves a transition from a neutral expression to a facial expression involving raised or lowered eyebrows;

whereby in a detection of the physiological response, based on TH_ROI1 taken during the certain period, involves attributing a change in TH_ROI1 to raising or lowering the eyebrows instead of a change in the temperature of the forehead.

19. The system of claim 1, further comprising at least one of the following:

(i) wherein VCAM comprises a multi-pixel sensor and a lens, and the sensor plane is tilted by more than 2°

relative to the lens plane according to the Scheimpflug principle in order to improve the sharpness of images of ROI₂; and

(ii) wherein CAM comprises a focal-plane array (FPA) sensor and an infrared lens, and the FPA plane is tilted by more than 2°

relative to the infrared lens plane according to the Scheimpflug principle in order to improve the sharpness of images of ROI₁.

20. The system of claim 1, wherein responsive to receiving a first set of TH_ROI1 and IM_ROI2 taken during a first period in which the user expressed a certain facial expression, the computer detects a first emotional response of the user, and responsive to receiving a second set of TH_ROI1 and IM_ROI2 taken during a second period in which the user expressed again the certain facial expression, the computer detects a second emotional response of the user, which is not the same as the first emotional response;

and wherein TH_ROI1 of the first set are indicative of a first physiological response, while TH_ROI1 of the second set are indicative of a second physiological response.

21. A method for detecting a physiological response, comprising:

taking, using an inward-facing head-mounted thermal camera located at most 15 cm from a user'"'"'s face, thermal measurements of a first region of interest (TH_ROI1) on the face;

taking, using an inward-facing head-mounted visible-light camera located at most 15 cm from the face, images of a second region of interest (IM_ROI2) on the face;

wherein ROI₁ and ROI₂ overlap;

generating feature values based on TH_ROI1 and IM_ROI2; and

utilizing a machine learning-based model to detect the physiological response based on the feature values;

wherein the machine learning-based model was trained based on previous TH_ROI1 and IM_ROI2 taken during different days.

22. The method of claim 21, wherein the physiological response is emotional response, and further comprising calculating, based on IM_ROI2, a value indicative of facial skin color changes (FSCC), and utilizing the value indicative of FSCC to generate at least one of the feature values used to detect the physiological response.

23. The method of claim 21, further comprising generating, based on IM_ROI2, feature values indicative of an occurrence of one or more of the following confounding factors on a portion of the overlapping region between ROI₁ and ROI₂:

a presence of cosmetics, a presence of sweat, a presence of hair, and a presence of skin inflammation; and

further comprising utilizing said feature values to detect the physiological response.