User state selection based on the shape of the exhale stream

• US 10,092,232 B2
• Filed: 12/06/2017
• Issued: 10/09/2018
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Thieberger, Gil, Frank, Ari M
• Primary Examiner(s): Porta, David
• Assistant Examiner(s): BOOSALIS, FANI POLYZOS
• Application Number:
  US15/832,935
• Publication Number:
  US 20180092589A1
• Time in Patent Office: 307 Days
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/0816 Measuring devices for examining respiratory frequency measuring frequency of electric signals G01R23/00
• CPC Class Codes: A61B 5/0873 using optical means
• CPC Class Codes: A61B 5/091 Measuring volume of inspired or expired gases, e.g. to determine lung capacity
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G01J 5/0025 Living bodies ear thermometers G01J5/0011; detecting, measuring or recording for diagnostic purposes A61B5/00
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: H04N 23/11 : for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/55 : Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

The three-dimensional (3D) shape of the exhale stream from a user'"'"'s nostrils is often correlated with the user'"'"'s state (e.g., an emotional state and/or a certain health condition). Thus, identifying the shape of a user'"'"'s exhale stream can help determine the user'"'"'s state. In one embodiment, a system that selects a state of a user includes at least one inward-facing head-mounted thermal camera that takes thermal measurements of at least three regions below the nostrils (TH_S). TH_S are indicative of shape of the exhale stream (SHAPE). The system also includes a computer that generates feature values based on TH_S, which are indicative of the SHAPE, and utilizes a model to select the state of the user, from among potential states of the user, based on the feature values. Optionally, the system includes a user interface that present the user'"'"'s selected state.

Claims

1. A system configured to select a state of a user, comprising:

at least one inward-facing head-mounted thermal camera (CAM) configured to take thermal measurements of at least three regions below the nostrils (TH_S);

wherein TH_S are indicative of shape of the exhale stream (SHAPE); and

a computer configured to select the state of the user, from among potential states of the user, based on TH_S.

2. The system of claim 1, wherein the computer is further configured to generate feature values based on TH_S, and to utilize a machine learning-based model to select the state of the user, from among potential states of the user, based on the feature values;

whereby the feature values are indicative of the SHAPE; and

wherein, for the same breathing rate, the computer is configured to select different states when TH_S are indicative of different shapes of the exhale stream (SHAPEs) that correspond to different potential states.

3. The system of claim 2, wherein the different potential states comprise different emotional states.

4. The system of claim 2, wherein the different potential states comprise at least one of:

(i) different physiological responses, and (ii) healthy and unhealthy states.

5. The system of claim 2, wherein the machine learning-based model was trained based on:

previous TH_S taken while the user was in a first potential state from among the potential states, and other previous TH_S taken while the user was in a second potential state from among the potential states.

6. The system of claim 2, wherein the machine learning-based model was trained based on:

previous TH_S taken from users while the users were in a first potential state from among the potential states, and other previous TH_S taken while the users were in a second potential state from among the potential states.

7. The system of claim 1, wherein for the same breathing rate, respiration volume, and dominant nostril, the computer is configured to select different states when TH_S are indicative of different SHAPEs that correspond to different potential states.

8. The system of claim 1, wherein the at least one CAM comprises (i) at least three sensing elements (S_V) pointed at different vertical positions below the nostrils where the exhale stream is expected to flow, and (ii) at least three sensing elements (S_H) pointed at different horizontal positions below the nostrils where the exhale stream is expected to flow;

wherein the larger the number of S_V that detect the exhale stream, the longer the length of the exhale stream, and the larger the number of S_H that detect the exhale stream, the wider the exhale stream.

9. The system of claim 1, wherein the at least three regions are located on (i) at least two positions below the nostrils having a vertical distance above 5 mm between their centers, and (ii) at least two positions below the nostrils having an horizontal distance above 5 mm between their centers.

10. The system of claim 1, further comprising a frame configured to be worn on the user'"'"'s head;

wherein the at least one CAM does not occlude any of the user'"'"'s mouth and nostrils, is located less than 15 cm from the user'"'"'s face and above the user'"'"'s upper lip, and comprises at least first and second inward-facing head-mounted thermal cameras (CAM1 and CAM2, respectively) configured to take TH_ROI1 and TH_ROI2, respectively;

CAM1 is physically coupled to the right half of the frame and captures the exhale stream from the right nostril better than it captures the exhale stream from the left nostril, and CAM2 is physically coupled to the left half of the frame and captures the exhale stream from the left nostril better than it captures the exhale stream from the right nostril.

11. The system of claim 1, wherein the at least three regions below the nostrils include a first region on the right side of the user'"'"'s upper lip, a second region on the left side of the user'"'"'s upper lip, and a third region on the mouth of the user;

and wherein thermal measurements of the third region are indicative of the exhale stream from the user'"'"'s mouth.

12. The system of claim 1, wherein the at least three regions below the nostrils include a first region comprising a portion of the volume of the air below the right nostril where the exhale stream from the right nostril flows, a second region comprising a portion of the volume of the air below the left nostril where the exhale stream from the left nostril flows, and a third region comprising a portion of a volume protruding out of the mouth where the exhale stream from the user'"'"'s mouth flows.

13. A method for selecting a state of a user, comprising:

taking, using at least one inward-facing head-mounted thermal camera (CAM), thermal measurements of at least three regions below the nostrils (TH_S);

wherein TH_S are indicative of shape of the exhale stream (SHAPE);

generating feature values based on TH_S;

whereby the feature values are indicative of the SHAPE; and

utilizing a machine learning-based model for selecting the state of the user, from among potential states of the user, based on the feature values.

14. The method of claim 13, further comprising selecting different states, for the same breathing rate, when TH_S are indicative of different SHAPEs that correspond to different potential states.

15. The method of claim 14, further comprising training the machine learning-based model based on:

previous TH_S taken while the user was in a first potential state from among the potential states, and other previous TH_S taken while the user was in a second potential state from among the potential states.

16. A system configured to detect the shape of exhale stream, comprising:

at least one inward-facing head-mounted thermal camera (CAM) configured to take thermal measurements of at least three regions below the nostrils (TH_S) of a user;

wherein TH_S are indicative of the shape of the exhale stream (SHAPE); and

a computer configured to detect the SHAPE based on TH_S and a machine learning-based model;

wherein the machine learning-based model was trained based on previous TH_S of the user.

17. The system of claim 16, wherein, for the same breathing rate, the computer is configured to detect a first SHAPE based on a first TH_S, and to detect a second SHAPE based on a second TH_S;

whereby the first and second TH_S are indicative of different thermal patterns.

18. The system of claim 16, wherein for the same breathing rate, respiration volume and dominant nostril, the computer is configured to detect a first SHAPE based on a first TH_S, and to detect a second SHAPE based on a second TH_S;

whereby the first and second TH_S are indicative of different thermal patterns.

19. A system configured to present a user'"'"'s state based on shape of the exhale stream (SHAPE), comprising:

at least one inward-facing head-mounted thermal camera (CAM) configured to take thermal measurements of at least three regions below the nostrils (TH_S);

wherein TH_S are indicative of the SHAPE; and

a user interface (UI) configured to present the user'"'"'s state based on TH_S;

wherein, for the same breathing rate, the UI presents different states for the user when TH_S are indicative of different SHAPEs that correspond to different potential states.

20. The system of claim 19, wherein each of the at least one CAM does not occlude any of the user'"'"'s mouth and nostrils;

and further comprising a computer configured to generate feature values based on TH_S, and to utilize a machine learning-based model to select the state, from among the potential states, based on the feature values.