Detecting physiological responses based on thermal asymmetry of the face

• US 9,968,264 B2
• Filed: 12/06/2017
• Issued: 05/15/2018
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Thieberger, Gil, Frank, Ari M
• Primary Examiner(s): Porta, David
• Assistant Examiner(s): BOOSALIS, FANI POLYZOS
• Application Number:
  US15/832,826
• Publication Number:
  US 20180103851A1
• Time in Patent Office: 160 Days
• CPC Class Codes: A61B 2560/0242 adapted to measure environmental factors, e.g. temperature, pollution
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/02028 Determining haemodynamic parameters not otherwise provided for, e.g. cardiac contractility or left ventricular ejection fraction
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/168 Evaluating attention deficit, hyperactivity
• CPC Class Codes: A61B 5/4041 Evaluating nerves condition
• CPC Class Codes: A61B 5/4052 efferent nerves, i.e. nerves that relay impulses from the central nervous system
• CPC Class Codes: A61B 5/41 Detecting, measuring or recording for evaluating the immune or lymphatic systems
• CPC Class Codes: A61B 5/411 Detecting or monitoring allergy or intolerance reactions to an allergenic agent or substance
• CPC Class Codes: A61B 5/4824 Touch or pain perception evaluation A61B5/4821 takes precedence
• CPC Class Codes: A61B 5/4884 inducing physiological or psychological stress, e.g. applications for stress testing
• CPC Class Codes: A61B 5/6803 : Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/7264 : Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 : involving training the classification device
• CPC Class Codes: A61B 5/7282 : Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/748 : Selection of a region of interest, e.g. using a graphics tablet
• CPC Class Codes: G01J 2005/0077 : Imaging
• CPC Class Codes: G01J 2005/106 : Arrays
• CPC Class Codes: G01J 3/0254 : Spectrometers, other than colorimeters, making use of an integrating sphere
• CPC Class Codes: G01J 5/0025 : Living bodies ear thermometers G01J5/0011; detecting, measuring or recording for diagnostic purposes A61B5/00
• CPC Class Codes: G01J 5/0275 : Control or determination of height or distance or angle information for sensors or receivers
• CPC Class Codes: G01J 5/04 : Casings
• CPC Class Codes: G01J 5/0806 : Focusing or collimating elements, e.g. lenses or concave mirrors
• CPC Class Codes: G01J 5/10 : using electric radiation detectors
• CPC Class Codes: G02C 11/10 : Electronic devices other than hearing aids
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing

Abstract

Manifestation of some physiological responses (e.g., stress, mental workload, or a headache) may involve the emergence of asymmetric thermal patterns on the face. Thus, thermal measurements of the face that are indicative of thermal asymmetry can be useful to detect such physiological responses. In one embodiment, a system includes first and second inward-facing head-mounted thermal cameras (CAM1 and CAM2, respectively) that are located less than 15 cm from the user'"'"'s face, which take thermal measurements of regions on the right and left sides of the face (TH_ROI1 and TH_ROI2, respectively) of the user. The symmetric overlapping between the regions on the right and left sides (ROI₁ and ROI₂, respectively) is above 60%, and CAM1 and CAM2 do not occlude ROI₁ and ROI₂. Optionally, the system includes a computer that detects a physiological response based on thermal asymmetry between TH_ROI1 and TH_ROI2.

Claims

1. A system configured to collect thermal measurements indicative of facial thermal asymmetry, comprising:

first and second inward-facing head-mounted thermal cameras (CAM1 and CAM2, respectively) configured to be located less than 15 cm from the face of a user and to take thermal measurements of regions on the right and left sides of the face (TH_ROI1 and TH_ROI2, respectively) of the user;

wherein the symmetric overlapping between the regions on the right and left sides (ROI₁ and ROI₂, respectively) is above 60%, and CAM1 and CAM2 do not occlude ROI₁ and ROI₂.

2. The system of claim 1, wherein CAM1 and CAM2 are configured to be located less than 10 cm from the face and at least 0.5 cm to the right and to the left of the vertical symmetry axis that divides the face, respectively;

and wherein CAM1 and CAM2 are configured to be located relative to the face, such that there are angles greater than 20°

between the Frankfort horizontal plane and the optical axes of CAM1 and CAM2.

3. The system of claim 2, wherein at least one of CAM1 and CAM2 comprises a multi-pixel sensor and a lens, and the sensor plane is tilted by more than 2°

relative to the lens plane according to the Scheimpflug principle in order to capture sharper images when the system is worn by the user.

4. The system of claim 1, further comprising third and fourth inward-facing head-mounted thermal cameras configured to be located less than 15 cm from the face and to take thermal measurements of regions on the right and left sides of the upper lip (TH_ROI3 and TH_ROI4, respectively), without occluding the upper lip;

wherein the symmetric overlapping between the regions on the right and left sides of the upper lip is above 60%.

5. The system of claim 4, further comprising a computer configured to:

(i) generate feature values based on TH_ROI1, TH_ROI2, TH_ROI3, and TH_ROI4; and

(ii) utilize a model to detect a physiological response based on the feature values;

wherein the model was trained based on previous TH_ROI1, TH_ROI2, TH_ROI3, and TH_ROI4 taken while the user had a physiological response associated with at least one of the following;

stress, mental workload, fear, sexual arousal, anxiety, pain, a headache, dehydration, intoxication, and a stroke.

6. The system of claim 1, further comprising a computer configured to detect a physiological response based on thermal asymmetry between TH_ROI1 and TH_ROI2, and the physiological response is not body temperature.

7. The system of claim 6, wherein the computer is further configured to (i) generate feature values based on previous TH_ROI1 and TH_ROI2 of the user, which were taken during different days, and (ii) utilize a model to detect the physiological response based on the feature values;

wherein the model was trained based on previous TH_ROI1 and TH_ROI2 taken while the user had a physiological response associated with at least one of the following;

stress, mental workload, fear, sexual arousal, anxiety, pain, a headache, dehydration, intoxication, and a stroke.

8. The system of claim 6, further comprising third and fourth outward-facing head-mounted thermal cameras (CAM3, CAM4), configured to take thermal measurements of the environment to the right and to the left of the face (TH_ENV1 and TH_ENV2, respectively) of the user;

wherein the computer is further configured to utilize TH_ENV1 and TH_ENV2 to identify asymmetry resulting from an environmental factor rather than from the physiological response.

9. The system of claim 8, wherein CAM1, CAM2, CAM3 and CAM4 comprise sensors based on at least one of the following sensor types:

thermopile sensors, pyroelectric sensors, and microbolometer sensors; and

wherein the environmental factor that caused the asymmetry comprises at least one of;

sunlight, air blowing from an air-conditioner, radiation from a heater, and radiation from an oven.

10. The system of claim 1, wherein ROI₁ and ROI₂ cover at least one of:

(i) regions on the right and left superficial temporal arteries, respectively, and (ii) regions on the right and left sides of the face comprising exposed facial skin between the mouth level and the eyebrow level, respectively; and

further comprising a computer configured to utilize a model to detect a stroke based on TH_ROI1 and TH_ROI2;

wherein the model was trained based on previous TH_ROI1 and TH_ROI2 taken during different days.

11. The system of claim 1, wherein ROI₁ and ROI₂ are on at least one of:

the periorbital area around the eyes, the nose, and the mouth; and

further comprising a computer configured to utilize a model to detect nerve damage based on TH_ROI1 and TH_ROI2;

wherein the model was trained based on previous TH_ROI1 and TH_ROI2 taken during different days.

12. The system of claim 1, wherein ROI₁ and ROI₂ cover right and left regions on the forehead, respectively, and further comprising third and fourth inward-facing head-mounted thermal cameras, configured to be located less than 10 cm from the face, and to take thermal measurements of regions on the right and left periorbital areas (TH_ROI3 and TH_ROI4, respectively);

and further comprising a computer configured to utilize a model to detect at least one of an emotional state and a stress level of the user based on TH_ROI1, TH_ROI2, TH_ROI3, and TH_ROI4;

wherein the model was trained based on previous TH_ROI1, TH_ROI2, TH_ROI3, and TH_ROI4 taken during different days.

13. The system of claim 12, further comprising fifth and sixth inward-facing head-mounted thermal cameras, configured to be located less than 10 cm from the face, and to take thermal measurements of regions on the right and left cheeks (TH_ROI5 and TH_ROI6, respectively);

and the computer is further configured to detect the at least one of the emotional state and the stress level also based on TH_ROI5 and TH_ROI6.

14. The system of claim 1, wherein ROI₁ and ROI₂ cover right and left regions on at least one of the following:

the forehead, and the periorbital areas; and

further comprising a computer configured to detect at least one of;

a headache, symptomatic behavior of Attention Deficit Hyperactivity Disorder (ADHD), and anger attack;

wherein the computer utilizes a model to perform the detection based on TH_ROI1 and TH_ROI2, and the model was trained based on previous TH_ROI1 and TH_ROI2 taken during different days.

15. The system of claim 1, wherein ROI₁ and ROI₂ cover right and left regions on at least one of the following:

the forehead, the nose, and the mouth; and

further comprising a computer configured to utilize a model detect a headache based on TH_ROI1 and TH_ROI2;

wherein the model was trained based on previous TH_ROI1 and TH_ROI2 taken during different days.

16. The system of claim 1, further comprising a computer configured to receive training data comprising:

physiological measurements indicative of levels of stress of the user, values indicative of durations during which the user felt stressed, and values indicative of durations during which the user had a headache;

the computer is further configured to utilize a machine learning-based training algorithm to train a model based on the training data, and to utilize the model to detect a headache based on TH_ROI1 and TH_ROI2.

17. The system of claim 1, wherein ROI₁ and ROI₂ are on the right and left sides of at least one of the jaws, respectively;

and further comprising a computer configured to utilize a model detect orofacial pain based on TH_ROI1 and TH_ROI2;

wherein the model was trained based on previous TH_ROI1 and TH_ROI2 taken during different days.

18. The system of claim 1, wherein ROI₁ and ROI₂ are over the right and left anterior sinuses, respectively;

and further comprising a computer configured to utilize a model to detect sinusitis based on TH_ROI1 and TH_ROI2;

wherein the model was trained based on (i) previous TH_ROI1 and TH_ROI2 of the user taken during different days, and (ii) TH_ROI1 and TH_ROI2 taken from other users who suffer from at least one of the following;

maxillary sinusitis, frontal sinusitis, unilateral frontal sinusitis, and unilateral maxillary sinusitis.

19. The system of claim 1, further comprising a computer configured to utilize a model to detect Bell'"'"'s palsy based on TH_ROI1 and TH_ROI2;

wherein the model was trained based on previous TH_ROI1 and TH_ROI2 taken during different days; and

further comprising a visible-light camera configured to take photos of the face;

wherein the computer is further configured to identify asymmetry in the photos in order to improve the probability of correctly identifying Bell'"'"'s palsy.

20. A method for detecting a physiological response that causes a thermal asymmetry on a user'"'"'s face, comprising:

taking, using first and second inward-facing head-mounted thermal cameras (CAM1 and CAM2), thermal measurements of regions on the right and left sides of the face (TH_ROI1 and TH_ROI2, respectively);

wherein the regions on the right and left sides of the face have a symmetric overlapping above 60%, CAM1 and CAM2 are less than 10 cm away from the face, and the angles between the Frankfort horizontal plane and the optical axes of CAM1 and CAM2 are greater than 20°

; and

detecting the physiological response based on TH_ROI1 and TH_ROI2.

21. The method of claim 20, further comprising (i) generating feature values based on TH_ROI1 and TH_ROI2;

and (ii) utilizing a model to detect the physiological response based on the feature values;

wherein the model was trained based on previous TH_ROI1 and TH_ROI2 taken while the user had a physiological response associated with at least one of the following;

stress, mental workload, fear, sexual arousal, anxiety, pain, a headache, dehydration, intoxication, and a stroke; and

further comprising alerting about the physiological response that causes the thermal asymmetry on the face.