Calculating respiratory parameters from thermal measurements

• US 10,130,308 B2
• Filed: 12/06/2017
• Issued: 11/20/2018
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Thieberger, Gil, Frank, Ari M
• Primary Examiner(s): Porta, David
• Assistant Examiner(s): BOOSALIS, FANI POLYZOS
• Application Number:
  US15/832,855
• Publication Number:
  US 20180103913A1
• Time in Patent Office: 349 Days
• CPC Class Codes: A61B 2560/0252 using ambient temperature
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/02438 with portable devices, e.g. worn by the patient
• CPC Class Codes: A61B 5/0816 Measuring devices for examining respiratory frequency measuring frequency of electric signals G01R23/00
• CPC Class Codes: A61B 5/091 Measuring volume of inspired or expired gases, e.g. to determine lung capacity
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/4094 Diagnosing or monitoring seizure diseases, e.g. epilepsy
• CPC Class Codes: A61B 5/411 Detecting or monitoring allergy or intolerance reactions to an allergenic agent or substance
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06V 10/143 : Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/454 : Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/462 : Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 40/165 : using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G16H 50/50 : for simulation or modelling of medical disorders
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Described herein are systems and methods for calculating respiratory parameters based on thermal measurements. In one embodiment, a system includes an inward-facing head-mounted thermal camera (CAM) and a computer. CAM takes thermal measurements of a region below the nostrils (TH_ROI) of a user, and TH_ROI are indicative of the exhale stream. Optionally, CAM is located above the user'"'"'s upper lip and less than 15 cm from the user'"'"'s face, and CAM does not occlude any of the user'"'"'s mouth and nostrils. The computer generates feature values based on TH_ROI and utilizes a model to calculate a respiratory parameter based on the feature values. The model was trained based on previous TH_ROI of the user taken during different days. Optionally, the respiratory parameter is indicative of the user'"'"'s breathing rate. Optionally, the computer detects whether the user breathed primarily through the mouth or through the nose.

Claims

1. A system configured to calculate a respiratory parameter, comprising:

an inward-facing head-mounted thermal camera (CAM) configured to take thermal measurements of a region below the nostrils (TH_ROI) of a user;

wherein TH_ROI are indicative of the exhale stream; and

a computer configured to;

generate feature values based on TH_ROI; and

utilize a machine learning-based model to calculate the respiratory parameter based on the feature values.

2. The system of claim 1, wherein the respiratory parameter is indicative of the user'"'"'s breathing rate;

CAM is located above the user'"'"'s upper lip and less than 15 cm from the user'"'"'s face, and does not occlude any of the user'"'"'s mouth and nostrils;

TH_ROI comprise thermal measurements of at least first and second regions below right and left nostrils (TH_ROI1 and TH_ROI2, respectively) of the user, which are indicative of exhale streams from the right and left nostrils, respectively; and

TH_ROI further comprise thermal measurement of at least one of a region on the mouth and a volume protruding out of the mouth (TH_ROI3) of the user, indicative of exhale stream from the mouth.

3. The system of claim 2, wherein the computer is further configured to detect, based on TH_ROI1, TH_ROI2, and TH_ROI3, whether the user is breathing mainly through the mouth or through the nose.

4. The system of claim 2, wherein the computer is further configured to detect, based on TH_ROI1 and TH_ROI2, whether the user is breathing mainly through the right nostril or through the left nostril.

5. The system of claim 1, further comprising at least one inward-facing head-mounted visible-light camera configured to take images of a region on the mouth (IM_M), wherein IM_M are indicative of whether the mouth is open or closed;

and the computer is further configured to utilize the machine learning-based model to detect, based on TH_ROI and IM_M, whether the user is breathing mainly through the mouth or through the nose;

wherein the machine learning-based model was trained based on;

a first set of TH_ROI taken while IM_M was indicative that the mouth was open, and a second set of TH_ROI taken while IM_M was indicative that the mouth was closed.

6. The system of claim 1, wherein the respiratory parameter is indicative of smoothness of the exhale stream, and the computer is further configured to alert the user responsive to detecting that the smoothness is below a predetermined threshold.

7. The system of claim 1, wherein the computer is further configured to utilize an indication of consuming a substance to detect a respiratory-related attack by training the machine learning-based model based on:

a first set of TH_ROI taken while the user experienced a respiratory-related attack after consuming the substance, and a second set of TH_ROI taken while the user did not experience a respiratory-related attack after consuming the substance; and

wherein the consuming of the substance involves consuming at least one of a certain drug and a certain food item.

8. The system of claim 1, wherein the computer is further configured to utilize an indication of a situation of the user to detect a respiratory-related attack by training the machine learning-based model based on:

a first set of TH_ROI taken while the user was in the situation and experienced a respiratory-related attack, and a second set of TH_ROI taken while the user was in the situation and did not experience a respiratory-related attack; and

wherein the situation involves one or more of the following;

(i) interacting with a certain person, (ii) a type of activity the user is conducting, selected from at least two different types of activities associated with different levels of stress, and (iii) a type of activity the user is about to conduct within thirty minutes, selected from at least two different types of activities associated with different levels of stress.

9. The system of claim 1, further comprising a sensor configured to take measurements indicative of the user'"'"'s movements (m_move), and the computer is further configured to detect a respiratory-related attack based on m_move and TH_ROI.

10. The system of claim 9, wherein the respiratory-related attack involves the user experiencing at least one of the following:

an asthma attack, an epileptic attack, an anxiety attack, a panic attack, and a tantrum; and

further comprising a user interface configured to alert the user responsive to detecting that a probability of the respiratory-related attack reaches a threshold.

11. The system of claim 1, further comprising a sensor configured to take measurements indicative of the user'"'"'s movements (m_move), and the computer is further configured to determine, based on m_move and TH_ROI, whether the user exhaled while making a physical effort above a predetermined threshold.

12. The system of claim 11, wherein the computer is further configured to:

receive a first indication that the user is making or is about to make the physical effort, command a user interface (UI) to suggest the user to exhale while making the physical effort, and command the UI to play a positive feedback in response to determining that the user exhaled while making the physical effort.

13. The system of claim 1, further comprising a sensor configured to take measurements indicative of the user'"'"'s movements (m_move), and the computer is further configured to:

(i) receive from a fitness app an indication that the user should exhale while making a movement, (ii) determine, based on m_move, when the user makes the movement, and (iii) determine, based on TH_ROI, whether the user exhaled while making the movement.

14. The system of claim 1, further comprising a sensor configured to take measurements indicative of the user'"'"'s movements (m_move), and the computer is further configured to:

(i) receive from a fitness app a certain number of breath cycles during which the user should perform a physical exercise, (ii) determine, based on m_move, when the user performs the physical exercise, and (iii) count, based on TH_ROI, the number of breath cycles the user had while performing the physical exercise.

15. The system of claim 1, wherein the machine learning-based model was trained based on previous TH_ROI of the user taken while the user had an asthma attack, the respiratory parameter is indicative of the user'"'"'s breathing rate, and the computer is further configured to command a user interface to alert the user about an imminent asthma attack responsive to identifying an increase in the breathing rate that is associated with an asthma attack according to the machine learning-based model.

16. The system of claim 15, further comprising a microphone configured to record the user;

wherein the computer is further configured to analyze the recording in order to identify at least one of the following body sounds;

asthmatic breathing sounds, asthma wheezing, and coughing;

wherein a first alert provided to the user in response to identifying an increase in the breathing rate, without identifying at least one of the body sounds, is less intense than a second alert provided to the user in response to identifying both the increase in the breathing rate and at least one of the body sounds.

17. The system of claim 15, further comprising a movement sensor configured to be worn by the user and to measure movements of the user;

wherein the computer is further configured to analyze the measurements of the movement sensor in order to identify breathing movements indicative of at least one of the following;

a spasm, shivering, and sagittal plane movement indicative of one or more of asthma wheezing, coughing, and chest tightness;

wherein a first alert provided to the user in response to identifying an increase in the breathing rate, without identifying at least one of the breathing movements, is less intense than a second alert provided to the user in response to identifying both the increase in the breathing rate and at least one of the breathing movements.

18. The system of claim 1, wherein the respiratory parameter is indicative of the ratio between exhale and inhale durations (t_exhale/t_inhale), and the computer is further configured to instruct the user, via a user interface, to prolong the exhale duration when t_exhale/t_inhale falls below a predetermined threshold.

19. The system of claim 1, wherein the computer is further configured to:

(i) receive an indication that the user'"'"'s stress level reaches a first threshold, (ii) identify, based on TH_ROI, that the ratio between exhaling and inhaling durations (t_exhale/t_inhale) is below a second threshold that is below 1.5, and (iii) command a user interface to suggest the user to prolong the exhale until t_exhale/t_inhale reaches a third threshold that is at least 1.5.

20. The system of claim 1, further comprising another inward-facing head-mounted thermal camera configure to take thermal measurements of a region on the forehead (TH_F) of the user;

wherein the computer is further configured to utilize TH_ROI and TH_F to detect a respiratory-related attack;

wherein the machine learning-based model was trained based on;

a first set of TH_ROI and TH_F taken while the user experienced a respiratory-related attack, and a second set of TH_ROI and TH_F taken while the user did not experience a respiratory-related attack.

21. A method for calculating a respiratory parameter, comprising:

taking, using an inward-facing head-mounted thermal camera (CAM), thermal measurements of a region below the nostrils (TH_ROI) of a user;

wherein TH_ROI are indicative of the exhale stream;

generating feature values based on TH_ROI; and

utilizing a machine learning-based model to calculate the respiratory parameter based on the feature values.

22. The method of claim 21, wherein TH_ROI comprise thermal measurements of regions below the right and left nostrils and thermal measurement of a region on the mouth, and further comprising detecting based on TH_ROI whether the user is breathing mainly through the mouth or through the nose.

23. The method of claim 21, further comprising taking, using a sensor, measurements indicative of the user'"'"'s movements (m_move), and determining, based on m_move and TH_ROI, whether the user exhaled while making a physical effort above a predetermined threshold.

24. The method of claim 21, further comprising:

training the machine learning-based model based on previous TH_ROI of the user taken while the user had an asthma attack, calculating the user'"'"'s breathing rate based on TH_ROI, and alerting the user about an imminent asthma attack responsive to identifying an increase in the breathing rate that is associated with an asthma attack according to the machine learning-based model.