Eyeglasses having inward-facing and outward-facing thermal cameras

• US 20180094982A1
• Filed: 12/06/2017
• Published: 04/05/2018
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Thieberger, Gil, Frank, Ari M.
• Granted Patent: US 10,151,636 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 2560/0252 using ambient temperature
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/0531 Measuring skin impedance
• CPC Class Codes: A61B 5/0816 Measuring devices for examining respiratory frequency measuring frequency of electric signals G01R23/00
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/318 Heart-related electrical modalities, e.g. electrocardiography [ECG]
• CPC Class Codes: A61B 5/369 Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/4094 Diagnosing or monitoring seizure diseases, e.g. epilepsy
• CPC Class Codes: A61B 5/411 Detecting or monitoring allergy or intolerance reactions to an allergenic agent or substance
• CPC Class Codes: A61B 5/441 Skin evaluation, e.g. for skin disorder diagnosis
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: A61B 5/4845 Toxicology, e.g. by detection of alcohol, drug or toxic products A61B5/4848 takes precedence
• CPC Class Codes: A61B 5/4875 Hydration status, fluid retention of the body
• CPC Class Codes: A61B 5/6803 : Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6898 : Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/744 : Displaying an avatar, e.g. an animated cartoon character
• CPC Class Codes: A61B 5/746 : Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: A61B 5/748 : Selection of a region of interest, e.g. using a graphics tablet
• CPC Class Codes: G01J 2005/0077 : Imaging
• CPC Class Codes: G01J 5/0025 : Living bodies ear thermometers G01J5/0011; detecting, measuring or recording for diagnostic purposes A61B5/00
• CPC Class Codes: G01J 5/0205 : Mechanical elements; Supports for optical elements
• CPC Class Codes: G01J 5/0265 : Handheld, portable ear thermometers G01J5/049
• CPC Class Codes: G01J 5/0806 : Focusing or collimating elements, e.g. lenses or concave mirrors
• CPC Class Codes: G01J 5/485 : Temperature profile
• CPC Class Codes: G01N 25/005 : by investigating specific heat
• CPC Class Codes: G02B 2027/0138 : comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/0172 : characterised by optical features
• CPC Class Codes: G02B 27/0176 : characterised by mechanical features
• CPC Class Codes: G06T 2207/30201 : Face
• CPC Class Codes: G06T 7/0012 : Biomedical image inspection
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: H04N 23/55 : Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Described herein are systems and methods for detecting a physiological response based on thermal measurements while accounting for effects of the environment. In one embodiment, a system includes an inward-facing head-mounted thermal camera (CAM_in) that takes thermal measurements of a region of interest (TH_ROI) on a user'"'"'s face, and an outward-facing head-mounted thermal camera (CAM_out) that takes thermal measurements of the environment (TH_ENV). CAM_in does not occlude the region of interest, and the system further includes a computer that detects the physiological response based on TH_ROI and TH_ENV. Optionally, the computer generates feature values based on sets of TH_ROI and TH_ENV, and utilizes a machine learning-based model to detect, based on the feature values, the physiological response.

Claims

1. A system configured to detect a physiological response, comprising:

an inward-facing head-mounted thermal camera (CAM_in) configured to take thermal measurements of a region of interest (TH_ROI) on a user'"'"'s face;

wherein CAM_in does not occlude the region of interest (ROI) and is located less than 15 cm from the user'"'"'s face;

an outward-facing head-mounted thermal camera (CAM_out) configured to take thermal measurements of the environment (TH_ENV), and is located less than 15 cm from the face; and

a computer configured to detect the physiological response based on TH_ROI and TH_ENV.

2. The system of claim 1, wherein the computer is further configured to generate feature values based on sets of TH_ROI and TH_ENV, and to utilize a machine learning-based model to detect, based on the feature values, the physiological response;

wherein responsive to receiving a first set of measurements in which TH_ROI reaches a first threshold while TH_ENV does not reach a second threshold, the computer detects the physiological response; and

wherein responsive to receiving a second set of measurements in which TH_ROI reaches the first threshold while TH_ENV reaches the second threshold, the computer does not detect the physiological response.

3. The system of claim 1, wherein the field of view (FOV) of CAM_in is larger than the FOV of CAM_out, and the noise equivalent differential temperature (NEDT) of CAM_in is lower than NEDT of CAM_out.

4. The system of claim 1, wherein, on average, detections of the physiological responses based on TH_ROI and TH_ENV are more accurate compared to detections of the physiological responses based on TH_ROI without TH_ENV;

and wherein CAM_in has a field of view (FOV) smaller than 80°

, CAM_out has a FOV larger than 80°

, and CAM_in has more sensing elements than CAM_out.

5. The system of claim 1, wherein the angle between the optical axes of CAM_in and CAM_out is at least one of the following angles:

45°

, 90°

, 130°

, 170°

, and 180°

; and

wherein CAM_in and CAM_out are based on sensors of the same type and are located less than 5 cm apart.

6. The system of claim 1, further comprising an eyeglasses frame configured to be worn on the user'"'"'s head;

wherein CAM_in and CAM_out each weigh below 5 g and are physically coupled to the frame.

7. The system of claim 1, wherein the region of interest includes a region on the forehead of the user and the physiological response comprises at least one of the following:

stress, a headache, and a stroke.

8. The system of claim 1, wherein the region of interest includes a region on the nose of the user and the physiological response is an allergic reaction.

9. The system of claim 1, further comprising a second outward-facing head-mounted thermal camera (CAM_out2) configured to take thermal measurements of the environment (TH_ENV2);

wherein there is an angle of at least 30°

between the optical axes of CAM_out and CAM_out2;

responsive to receiving a first set of measurements in which TH_ROI reach a first threshold while the difference between TH_ENV and TH_ENV2 does not reach a second threshold, the computer detects the physiological response; and

responsive to receiving a second set of measurements in which TH_ROI reach the first threshold while the difference between TH_ENV and TH_ENV2 reaches the second threshold, the computer does not detect the physiological response.

10. The system of claim 1, further comprising a second outward-facing head-mounted thermal camera (CAM_out2) configured to take thermal measurements of the environment (TH_ENV2) wherein there is an angle of at least 30°

between the optical axes of CAM_out and CAM_out2; and

wherein the computer is configured to detect the physiological response based on a difference between TH_ROI, TH_ENV, and TH_ENV2, while taking into account the angle between the optical axes of CAM_out and CAM_out2 and a graph of responsivity as function of the angle from the optical axes of each of CAM_out and CAM_out2.

11. The system of claim 1, wherein CAM_in and CAM_out are located to the right of the vertical symmetry axis that divides the user'"'"'s face, and the ROI is on the right side of the face;

and further comprising a second inward-facing head-mounted thermal camera (CAM_in2) and a second outward-facing head-mounted thermal camera (CAM_out2) located to the left of the vertical symmetry axis;

CAM_in2 is configured to take thermal measurements of a second ROI (TH_ROI2) on the left side of the face, and does not occlude ROI₂;

CAM_out2 is configured to take thermal measurements of an environment (TH_ENV2) that is more to the left relative to the environment measured by CAM_out;

wherein the computer is configured to detect the physiological response also based on TH_ROI2 and TH_ENV2.

12. The system of claim 1, wherein the optical axes of CAM_in and CAM_out are above the Frankfort horizontal plane;

and further comprising a second inward-facing head-mounted thermal camera (CAM_in2) and a second outward-facing head-mounted thermal camera (CAM_out2), located such that their optical axes are below the Frankfort horizontal plane, and configured to take thermal measurements TH_ROI2 and TH_ENV2, respectively;

wherein the computer is configured to detect the physiological response also based on TH_ROI2 and TH_ENV2.

13. The system of claim 1, further comprising a sensor configured to take measurements (m_conf) that are indicative of at least one of the following:

an extent of the user'"'"'s activity, an orientation of the user'"'"'s head, and a change in a position of the user'"'"'s body;

wherein the computer is further configured to detect the physiological response also based on m_conf.

14. The system of claim 13, wherein the computer is further configured to generate feature values based on TH_ROI, TH_ENV, and m_conf, and to utilize a model to detect the physiological response based on the feature values;

wherein the model was trained based on;

a first set of previous TH_ROI, TH_ENV, and m_conf taken while the user was walking or running, and a second set of previous TH_ROI, TH_ENV, and m_conf taken while the user was sitting or standing.

15. The system of claim 13, wherein m_conf are indicative of bending the head down above a certain degree from the normal to earth;

wherein the computer is further configured to generate feature values based on TH_ROI, TH_ENV, and m_conf, and to utilize a model to detect the physiological response based on the feature values;

wherein the model was trained based on;

a first set of TH_ROI, TH_ENV, and m_conf taken while the user was bending the head down above the certain degree, and a second set of previous TH_ROI, TH_ENV, and m_conf taken while the user was not bending the head down above the certain degree.

16. The system of claim 13, wherein the sensor comprises one or more of the following sensors:

(i) a movement sensor that is physically coupled to a frame worn on the user'"'"'s head or to a wearable device worn by the user, (ii) a visible-light camera configured to take images of the user, and (iii) an active 3D tracking device configured to emit electromagnetic waves and generate 3D images based on received reflections of the emitted electromagnetic waves.

17. The system of claim 1, further comprising an anemometer that is physically coupled to a frame worn on the user'"'"'s head, is located less than 15 cm from the face, and is configured to provide a value indicative of a speed of air directed at the face (m_wind);

wherein the computer is further configured to detect the physiological response also based on m_wind.

18. The system of claim 17, wherein the computer is further configured to generate feature values based on TH_ROI, TH_ENV, and m_wind, and to utilize a model to detect, based on the feature values, the physiological response;

wherein the model was trained based on;

a first set of TH_ROI, TH_ENV, and m_wind taken while being indoors, and a second set of TH_ROI, TH_ENV, and m_wind taken while being outdoors.

19. The system of claim 1, wherein the computer is configured to detect the physiological response based on a difference between TH_ROI and TH_ENV;

whereby detecting the physiological response based on the difference enables the system to operate well in an uncontrolled environment that does not maintain environmental temperature in a range below ±

3°

C. and does not maintain humidity in a range below ±

10%.

20. A method for detecting a physiological response, comprising:

taking thermal measurements of a region of interest (TH_ROI) on a user'"'"'s face using an inward-facing head-mounted thermal camera (CAM_in);

taking thermal measurements of the environment (TH_ENV) utilizing an outward-facing head-mounted thermal camera (CAM_out) worn by the user;

generating feature values based on TH_ROI and TH_ENV; and

utilizing a machine learning-based model to detect the physiological response based on the feature values.

21. The method of claim 20, further comprising:

taking a first set of TH_ROI (first TH_ROI), wherein the first TH_ROI reach a first threshold;

taking a first set of TH_ENV (first TH_ENV), wherein the first set of TH_ENV do not reach a second threshold;

detecting, based on the first set of TH_ROI and the first set of TH_ENV, that the user had the physiological response;

taking a second set of TH_ROI, wherein the second set of TH_ROI reach the first threshold;

taking a second set of TH_ENV, wherein the second set of TH_ENV reach the second threshold; and

detecting, based on the second set of TH_ROI and the second set of TH_ENV, that the user did not have the physiological response.