Eyeglasses that measure facial skin color changes

• US 10,216,981 B2
• Filed: 12/06/2017
• Issued: 02/26/2019
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Thieberger, Gil, Frank, Ari M
• Primary Examiner(s): Lamprecht, Joel
• Application Number:
  US15/833,158
• Publication Number:
  US 20180096198A1
• Time in Patent Office: 447 Days
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/1032 Determining colour for diagnostic purposes measuring colour in general G01J3/46
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/441 Skin evaluation, e.g. for skin disorder diagnosis
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: A61B 5/6821 Eye
• CPC Class Codes: G01J 5/0025 Living bodies ear thermometers G01J5/0011; detecting, measuring or recording for diagnostic purposes A61B5/00
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Described herein are systems and methods for detecting a physiological response based on facial skin color changes (FSCC) recognizable in images taken with an inward-facing head-mounted visible-light camera (VCAM_in). Some examples of physiological responses whose manifestation involves FSCC include emotional responses (which at times may be hidden to the naked eye), and physiological signals such as a heart rate, heart rate variability, and/or a breathing rate. In one embodiment, a system that detects a physiological response based on FSCC includes VCAM_in that takes images of a region of interest (IM_ROI) on a user'"'"'s face, which is illuminated by ambient light, and a computer that detects the physiological response based on FSCC recognizable in IM_ROI. Optionally, the system includes an outward-facing head-mounted visible-light camera (VCAM_out) that takes images of the environment (IM_ENV), and the computer detects the physiological response also based on IM_ENV.

Claims

1. A system configured to detect a physiological response based on facial skin color changes (FSCC), comprising:

an inward-facing head-mounted visible-light camera (VCAM_in) configured to take images of a region of interest (IM_ROI) on a user'"'"'s face;

wherein the region of interest (ROI) covers at least a portion of at least one of the following regions on the user'"'"'s face;

forehead, nose, and cheek; and

wherein the ROI is illuminated by ambient light; and

a computer configured to detect the physiological response based on a photoplethysmographic signal manifested by FSCC recognizable in IM_ROI.

2. The system of claim 1, further comprising an outward-facing head-mounted visible-light camera (VCAM_out) configured to take images of the environment (IM_ENV);

wherein the computer is further configured to utilize a model to detect the physiological response also based on IM_ENV.

3. The system of claim 2, wherein IM_ENV is indicative of illumination towards the face, IM_ROI is indicative of reflections from the face, and the computer is configured to utilize IM_ENV to account, at least in part, for variations in ambient light that cause errors in detections of the physiological response.

4. The system of claim 2, wherein the physiological response comprises an expression of emotional response of the user.

5. The system of claim 2, wherein the physiological response comprises at least one of a heart rate of the user, heart rate variability of the user, and a breathing rate of the user.

6. The system of claim 2, wherein at least one of VCAM_in and VCAM_out comprise sensors configured to capture light rays also in at least one of the following near infrared (NIR) spectrum intervals:

700-800 nm, 700-900 nm, 700-1,000 nm; and

the computer is further configured to detect the physiological response also based on data obtained in the NIR spectrum interval.

7. The system of claim 2, wherein the model was trained on:

a first set of IM_ROI and IM_ENV taken while being indoors and not in direct sunlight, and a second set of IM_ROI and IM_ENV taken while being outdoors in direct sunlight.

8. The system of claim 2, wherein the model was trained on:

a first set of IM_ROI and IM_ENV taken while exercising and moving, and a second set of IM_ROI and IM_ENV taken while sitting and not exercising.

9. The system of claim 1, wherein VCAM_in does not occlude the ROI, and the computer is further configured to utilize a model to detect the physiological response;

and wherein the model was trained on;

a first set of IM_ROI taken while being indoors and not in direct sunlight, and a second set of IM_ROI taken while being outdoors in direct sunlight.

10. The system of claim 1, wherein the computer is further configured to utilize a model to detect the physiological response;

and wherein the model was trained on;

a first set of IM_ROI taken while exercising and moving, and a second set of IM_ROI taken while sitting and not exercising;

whereby the physiological response is still detected by the computer based on FSCC recognizable in IM_ROI that are taken while the user is running.

11. The system of claim 1, wherein the computer is further configured to utilize a model to detect the physiological response;

and wherein the model was trained on;

a first set of IM_ROI taken less than 30 minutes after the user drank an alcoholic beverage, and a second set of IM_ROI taken on a day in which the user did not drink an alcoholic beverage.

12. The system of claim 1, wherein the computer is further configured to utilize a model to detect the physiological response;

wherein the model was trained on samples, each sample comprising;

feature values generated based on IM_ROI of the user, and a label indicative of an emotional response of the user; and

wherein the label is generated based on one or more of the following;

semantic analysis of a communication of the user, analysis of a facial expression made by the user, thermal measurements of regions of the face of the user, and one or more of the following physiological signals of the user;

heart rate, heart rate variability, breathing rate, and galvanic skin response.

13. The system of claim 1, wherein the computer is further configured to receive an indication indicative of the user consuming a confounding substance that affects FSCC, and to detect the physiological response, while the consumed confounding substance affects FSCC, based on:

IM_ROI, the indication, and a model;

wherein the model was trained on;

a first set of IM_ROI taken while the confounding substance affected FSCC, and a second set of IM_ROI taken while the confounding substance did not affect FSCC.

14. The system of claim 1, wherein the ROI is on the forehead, and further comprising a frame configured to be worn on the user'"'"'s head and to hold VCAM_in less than 10 cm from the user'"'"'s face such that the optical axis of VCAM_in is above 20°

from the Frankfort horizontal plane.

15. The system of claim 1, wherein the ROI is on the nose, and further comprising a frame configured to be worn on the user'"'"'s head and to hold VCAM_in less than 10 cm from the user'"'"'s face.

16. The system of claim 1, further comprising another VCAM_in configured to be pointed at an eye of the user;

and the computer is further configured to perform at least one of the following;

(i) select the sclera as a second region of interest (ROI2), and detect the physiological response based on color changes recognizable in images taken by the another VCAM_in, and (ii) select the iris as ROI2, and to detect the physiological response based on color changes recognizable in the iris.

17. The system of claim 1, further comprising an inward-facing head-mounted thermal camera configured to take thermal measurements of a second ROI (TH_ROI2) on the user'"'"'s face;

wherein ROI and ROI₂ overlap, and the computer is further configured to utilize TH_ROI2 to detect the physiological response;

whereby, on average, detecting the physiological response based on both FSCC recognizable in IM_ROI and TH_ROI2 is more accurate than detecting the physiological response based on FSCC recognizable in IM_ROI without TH_ROI2.

18. The system of claim 1, wherein VCAM_in does not occlude the ROI, and VCAM_in comprises a sensor and a lens;

the sensor plane is tilted by a fixed angle greater than 2°

relative to the lens plane according to the Scheimpflug principle in order to capture a sharper image when VCAM_in is worn by the user.

19. The system of claim 1, wherein VCAM_in comprises a sensor, a lens, and a motor;

the motor is configured to tilt the lens relative to the sensor according to the Scheimpflug principle;

whereby the tilt improves the sharpness of IM_ROI when VCAM_in is worn by the user.

20. The system of claim 1, wherein the computer is further configured to generate an avatar of the user based on IM_ROI, and to modify the avatar to show a synthesized facial expression that corresponds to an emotional response detected based on FSCC recognizable in IM_ROI, which is not manifested in the user'"'"'s facial expression.

21. A method for detecting a physiological response, comprising:

taking, utilizing an inward-facing head-mounted visible-light camera (VCAM_in), images of region of interest (IM_ROI) on a user'"'"'s face;

wherein the region of interest (ROI) covers at least a portion of at least one of the following regions on the user'"'"'s face;

forehead, nose, and cheek; and

wherein the ROI is illuminated by ambient light;

taking, utilizing an outward-facing head-mounted visible-light camera (VCAM_out), images of the environment (IM_ENV);

wherein IM_ENV is indicative of illumination towards the face; and

detecting the physiological response based on;

a photoplethysmographic signal manifested by FSCC recognizable in IM_ROI, and IM_ENV;

wherein utilizing IM_ENV accounts, at least in part, for variations in ambient light.

22. The method of claim 21, further comprising generating an avatar of the user based on IM_ROI, and modifying the avatar to show a synthesized facial expression that corresponds to an emotional response detected based on FSCC recognizable in IM_ROI, which is not manifested in the user'"'"'s facial expression.