Detecting stress based on thermal measurements of the face

• US 10,299,717 B2
• Filed: 12/06/2017
• Issued: 05/28/2019
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Thieberger, Gil, Frank, Ari M
• Primary Examiner(s): Lamprecht, Joel
• Application Number:
  US15/832,833
• Publication Number:
  US 20180092586A1
• Time in Patent Office: 538 Days
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/4884 inducing physiological or psychological stress, e.g. applications for stress testing
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: G01J 5/0025 Living bodies ear thermometers G01J5/0011; detecting, measuring or recording for diagnostic purposes A61B5/00
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Described herein are systems and methods for detecting a stress level of a user. In one embodiment, a system includes an inward-facing head-mounted thermal camera (CAM) and a computer. CAM takes thermal measurements of a region on a periorbital area (TH_ROI1) of the user. The computer generates feature values based on TH_ROI1, and utilizes a model to detect the stress level based on the feature values. The model was trained based on: previous TH_ROI1 taken while the user was under elevated stress, and other previous TH_ROI1 taken while the user was not under elevated stress. Some embodiments may utilize additional thermal cameras that take thermal measurements of other regions on the face, which may be utilized to detect the stress level.

Claims

1. A system configured to detect a stress level, comprising:

an inward-facing head-mounted thermal camera (CAM) configured to be held less than 10 cm from a user'"'"'s head and to take thermal measurements of a region on a periorbital area (TH_ROI1) of the user;

wherein the field of view of CAM does not cover a region outside the user'"'"'s face; and

a computer configured to;

generate feature values based on TH_ROI1, utilize a model to detect the stress level based on the feature values, and output said stress level;

wherein the model was trained based on;

previous TH_ROI1 taken while the user was under elevated stress, and other previous TH_ROI1 taken while the user was not under elevated stress.

2. The system of claim 1, wherein the region on the periorbital area is a region of the periorbital area of the right eye;

and further comprising a second inward-facing head-mounted thermal camera (CAM2) configured to be held less than 10 cm from the user'"'"'s face and to take thermal measurements of a region on the periorbital area of the left eye (TH_ROI2);

wherein the computer is further configured to detect the stress level based on TH_ROI1 and TH_ROI2.

3. The system of claim 2, further comprising a frame configured to be worn on the user'"'"'s head;

wherein CAM and CAM2 are physically coupled to the frame, and each weighs below 10 g; and

wherein the frame is configured to hold CAM and CAM2 at least 0.5 cm to the right and to the left of the vertical symmetry axis, respectively, and less than 5 cm from the face.

4. The system of claim 2, wherein temperature changes at the periorbital areas of the user, in response to a stressful event above a certain level, are asymmetric;

and the computer is further configured to account for asymmetry when detecting the stress level.

5. The system of claim 1, further comprising a second inward-facing head-mounted thermal camera configured to take thermal measurements of a region on the user'"'"'s forehead (TH_ROI2);

wherein the computer is further configured to generate additional feature values based on TH_ROI2, and to detect the stress level based on the feature values and the additional feature values.

6. The system of claim 1, further comprising a second inward-facing head-mounted thermal camera configured to take thermal measurements of a region on the user'"'"'s nose (TH_ROI2);

wherein the computer is further configured to generate additional feature values based on TH_ROI2, and to detect the stress level based on the feature values and the additional feature values.

7. The system of claim 1, further comprising a second inward-facing head-mounted thermal camera configured to be located above the user'"'"'s upper lip and to take thermal measurements of a region below the user'"'"'s nostrils (TH_ROI2);

wherein TH_ROI2 is indicative of the user'"'"'s breathing, and the computer is further configured to generate additional feature values based on TH_ROI2, and to detect the stress level based on the feature values and the additional feature values.

8. The system of claim 1, further comprising second and third inward-facing head-mounted thermal cameras, configured to be located less than 10 cm from the user'"'"'s face and at least 1 cm to the right and to the left of the vertical symmetry axis, respectively, and to take thermal measurements of regions on the right and left cheeks, respectively;

wherein the computer is configured to generate additional feature values based on the thermal measurements of the cheeks, and to detect the stress level based on the feature values and the additional feature values.

9. The system of claim 1, wherein the computer is further configured to detect the stress level based on a difference between TH_ROI1 and a baseline value determined based on a set of previous measurements taken by CAM;

and wherein most of the measurements belonging to the set were taken while the user was not under elevated stress.

10. The system of claim 1, further comprising a sensor configured to take additional measurements (m_conf) of at least one of the following:

the user, and the environment in which the user was in while TH_ROI1 were taken;

wherein the computer is further configured to generate, based on m_conf, one or more feature values that are utilized to detect the stress level;

whereby m_conf are indicative of an extent to which a confounding factor occurred while TH_ROI1 were taken.

11. The system of claim 10, wherein the sensor is an inward-facing head-mounted visible-light camera configured to take images of a second region on the user'"'"'s face;

wherein the second region includes at least 25% of the region on the periorbital area, and the confounding factor comprises touching the face.

12. The system of claim 10, wherein the sensor comprises a movement sensor configured to measure a movement of the user and the confounding factor involves the user performing at least one of the following:

walking, running, exercising, bending over, and getting up from a sitting or lying position.

13. The system of claim 1, wherein the computer is further configured to generate at least one feature value based on an additional measurement of the user taken during the same period TH_ROI1 were taken;

wherein the additional measurement is indicative of one or more of the following signals;

a heart rate of the user, heart rate variability of the user, brainwave activity of the user, galvanic skin response of the user, muscle activity of the user, and an extent of movement of the user; and

wherein the computer is further configured to detect the stress level of the user based on the feature values and the at least one feature value.

14. The system of claim 1, further comprising a head-mounted display configured to present digital content to the user;

wherein the display does not prevent CAM from measuring the region on the periorbital area.

15. The system of claim 1, further comprising an eye tracker configured to track gaze of the user, and (i) an augmented reality optical see through head-mounted display configured to operate in cooperation with:

a visible-light camera configured to capture images of objects the user is looking at, and a computer configured to match the objects the user is looking at with the detected stress levels, or (ii) a virtual reality head-mounted display configured to present video comprising objects, and the computer is configured to match the objects the user is looking at with the detected stress levels.

16. The system of claim 1, further comprising a user interface configured to notify the user when the stress level reaches a predetermined threshold, and to provide the user with encouragement not to engage in behavior that causes stress.

17. A system configured to alert about stress, comprising:

first and second inward-facing head-mounted thermal cameras (CAM1, CAM2) configured to be held at least 0.5 cm to the right and to the left of the vertical symmetry axis that divides a user'"'"'s face, respectively;

CAM1 and CAM2 are further configured to take thermal measurements of regions on the periorbital areas of the right and left eyes (TH_ROI1 and TH_ROI2, respectively) of the user;

wherein the fields of view of CAM1 and CAM2 do not cover a region outside the user'"'"'s face; and

a user interface configured to provide an alert about a stress level reaching a threshold;

wherein the stress level is determined based on TH_ROI1 and TH_ROI2.

18. The system of claim 17, further comprising a computer configured to detect the stress level based on TH_ROI1 and TH_ROI2 and to command the user interface to provide the alert;

and further comprising a frame configured to be worn on the user'"'"'s head, wherein each of CAM1 and CAM2 is physically coupled to the frame and the frame is further configured to hold each of CAM1 and CAM2 less than 15 cm from the user'"'"'s face.

19. A method comprising:

taking thermal measurements of regions on the periorbital areas of the right and left eyes (TH_ROI1 and TH_ROI2, respectively) of a user utilizing first and second inward-facing head-mounted thermal cameras (CAM1 and CAM2, respectively) configured to be held at least 0.5 cm to the right and to the left of the vertical symmetry axis that divides the user'"'"'s face, respectively;

wherein the fields of view of CAM1 or CAM2 do not cover a region outside the user'"'"'s face; and

alerting about a stress level that is detected based on TH_ROI1 and TH_ROI2.

20. The method of claim 19, further comprising generating feature values based on TH_ROI1 and TH_ROI2, and utilizing a model for detecting the stress level based on the feature values;

wherein the model was trained based on previous TH_ROI1 and TH_ROI2 of the user, taken during different days, which comprise;

a first set of measurements taken while the user had a first stress level according to a predetermined stress scale, and a second set of measurements taken while the user had a second stress level according to the predetermined stress scale.

21. The method of claim 19, further comprising generating feature values based on TH_ROI1 and TH_ROI2, and utilizing a model for detecting the stress level based on the feature values;

wherein the model was trained based on previous TH_ROI1 and TH_ROI2 of users, taken during different days, which comprise;

a first set of measurements taken while the users had a first stress level according to a predetermined stress scale, and a second set of measurements taken while the users had a second stress level according to the predetermined stress scale.