Smartglasses for detecting congestive heart failure

• US 10,791,938 B2
• Filed: 03/26/2020
• Issued: 10/06/2020
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Ori, Frank, Ari M, Tzvieli, Arie, Thieberger, Gil
• Primary Examiner(s): Shin, Soo
• Application Number:
  US16/831,413
• Publication Number:
  US 20200221956A1
• Time in Patent Office: 194 Days
• Field of Search: None
• CPC Class Codes: A61B 2560/0252 using ambient temperature
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 2562/0276 comprising a thermosensitive compound
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/02438 with portable devices, e.g. worn by the patient
• CPC Class Codes: A61B 5/0878 using temperature sensing means
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/748 Selection of a region of interest, e.g. using a graphics tablet
• CPC Class Codes: G01J 2005/0077 Imaging
• CPC Class Codes: G01J 5/0025 Living bodies ear thermometers G01J5/0011; detecting, measuring or recording for diagnostic purposes A61B5/00
• CPC Class Codes: G01J 5/0265 Handheld, portable ear thermometers G01J5/049
• CPC Class Codes: G01J 5/12 using thermoelectric elements, e.g. thermocouples
• CPC Class Codes: G01J 5/485 Temperature profile

Abstract

Systems for calculating extent of congestive heart failure (CHF) and/or identifying exacerbation of CHF. In one embodiment, a system includes smartglasses configured to be worn on a user'"'"'s head, and an inward-facing camera and a sensor, both physically coupled to the smartglasses. The inward-facing camera is mounted more than 5 mm away from the head and captures images of an area comprising skin on the user'"'"'s head, which is larger than 4 cm{circumflex over ( )}2. The sensor measures a signal indicative of a respiration rate of the user (respiration signal). The system also includes a computer that calculates the extent of CHF based on: a facial blood flow pattern recognizable in the images, and respiration rate recognizable in the respiration signal.

Claims

1. A system configured to calculate extent of congestive heart failure (CHF), comprising:

smartglasses configured to be worn on a user'"'"'s head;

an inward-facing camera, physically coupled to the smartglasses, configured to capture images of an area comprising skin on the user'"'"'s head;

wherein the area is larger than 4 cm{circumflex over ( 

)}2, and the inward-facing camera is mounted more than 5 mm away from the user'"'"'s head;

a sensor, physically coupled to the smartglasses, configured to measure a signal indicative of a respiration rate of the user; and

a computer configured to calculate the extent of CHF based on;

a facial blood flow pattern recognizable in the images, and the respiration rate of the user recognizable in the signal.

2. The system of claim 1, wherein the images and the signal are measured over a period spanning multiple days, and the computer is further configured to identify an exacerbation of the CHF based on:

a reduction in average facial blood flow recognizable in the images taken during the period, and an increase in an average respiration rate recognizable in the signal measured during the period.

3. The system of claim 1, wherein the area comprises a portion of the lips of the user.

4. The system of claim 1, wherein the images and the signal were measured while the user was at rest and prior to a period during which the user walked;

wherein the computer is further configured to receive;

(i) additional images, taken within ten minutes after the period with the inward-facing camera, and (ii) an additional signal indicative of an additional respiration rate of the user, measured with the sensor within ten minutes after the period; and

wherein the computer is further configured to calculate the extent of CHF based on;

a difference between the facial blood flow pattern recognizable in the images and an additional facial blood flow pattern recognizable in the additional images, and a difference between the respiration rate recognizable in the signal and the additional respiration rate recognizable in the additional signal.

5. The system of claim 4, further comprising a movement sensor, physically coupled to the smartglasses, configured to measure movements of the user;

wherein the computer is further configured to;

calculate a number of steps performed by the user during the period, and to calculate the extent of CHF responsive to the number of steps exceeding a predetermined threshold greater than twenty steps.

6. The system of claim 4, wherein the computer is further configured to:

calculate a value indicative of skin color at different times based on the additional images, and to calculate the extent of CHF based on a length of a duration following the period, in which the difference between the skin color and a baseline skin color, calculated based on the images, was above a threshold.

7. The system of claim 4, wherein the computer is further configured to:

calculate a value indicative of skin color at different times based on the additional images, and to calculate the extent of CHF based on a rate of return of the user'"'"'s skin color to a baseline skin color calculated based on the images.

8. The system of claim 4, wherein the computer is further configured to:

calculate respiration rates of the user at different times based on the additional signal, and to calculate the extent of CHF based on a length of a duration following the period, in which the difference between the respiration rate of the user and a baseline respiration rate, calculated based on the respiration rates, was above a threshold.

9. The system of claim 1, wherein the computer is further configured to calculate, based on the images, a value indicative of an extent to which skin in the area is blue and/or gray, and to utilize a difference between the value and a baseline value for the extent to which skin in the area is blue and/or gray to calculate the extent of CHF;

wherein the baseline value was determined while the user experienced a certain baseline extent of CHF.

10. The system of claim 1, wherein the computer is further configured to calculate, based on the images, a value indicative of extent of color changes to skin in the area due to cardiac pulses, and to utilize a difference between the value and a baseline value for the extent of the color changes to calculate the extent of CHF;

wherein the baseline value for the extent of the color changes was determined while the user experienced a certain baseline extent of CHF.

11. The system of claim 1, further comprising a head-mounted sensor configured to measure temperature of a region comprising skin on the user'"'"'s head (T_skin);

and the computer is further configured to utilize T_skin to compensate for effects of skin temperature on the facial blood flow pattern.

12. The system of claim 1, further comprising a head-mounted sensor configured to measure environmental temperature (T_env);

and the computer is further configured to utilize T_env to compensate for effects of physiologic changes related to regulating the user'"'"'s body temperature on the facial blood flow pattern.

13. A system configured to identify exacerbation of congestive heart failure (CHF), comprising:

smartglasses configured to be worn on a user'"'"'s head;

an inward-facing camera, physically coupled to the smartglasses, configured to capture images of an area comprising skin on the user'"'"'s head, which are indicative of a facial blood flow pattern of the user;

wherein the area is larger than 4 cm{umlaut over ( 

)}2, and the inward-facing camera is mounted more than 5 mm away from the user'"'"'s head;

a sensor, physically coupled to the smartglasses, configured to measure a signal indicative of a respiration rate of the user; and

a computer configured to;

receive previous images of the area, which are indicative of a previous facial blood flow pattern while the user had a certain extent of CHF;

receive a previous respiration rate taken while the user had the certain extent of CHF; and

identify exacerbation of the CHF based on;

a difference above a first predetermined threshold between the facial blood flow pattern and the previous facial blood flow pattern, and an increase above a second predetermined threshold in the respiration rate compared to the previous respiration rate.

14. The system of claim 13, further comprising a head-mounted sensor configured to measure temperature of a region comprising skin on the user'"'"'s head (T_skin);

wherein the computer is further configured to utilize T_skin to compensate for effects of skin temperature on the facial blood flow pattern.

15. The system of claim 13, further comprising a head-mounted sensor configured to measure environmental temperature (T_env);

wherein the computer is further configured to utilize T_env to compensate for effects of physiologic changes related to regulating the user'"'"'s body temperature on the facial blood flow pattern.

16. A system configured to calculate extent of congestive heart failure (CHF), comprising:

smartglasses configured to be worn on a user'"'"'s head;

an inward-facing camera, physically coupled to the smartglasses, configured to capture images of an area comprising skin on the user'"'"'s head;

wherein the area is larger than 4 cm{umlaut over ( 

)}2, and the inward-facing camera is mounted more than 5 mm away from the user'"'"'s head; and

a computer configured to;

receive a first set of the images taken while the user was at rest and prior to a period during which the user performed physical activity;

receive a second set of the images taken within ten minutes after the period; and

calculate the extent of CHF based on differences in facial blood flow patterns recognizable in the first and second sets of the images.

17. The system of claim 16, further comprising a movement sensor, physically coupled to the smartglasses, configured to measure movements of the user;

wherein the computer is further configured to detect the period during which the user performed the physical activity based on the movements; and

wherein the physical activity comprises walking at least 20 steps.

18. The system of claim 16, wherein the computer is further configured to calculate first and second series of heart rate values from portions of iPPG signals extracted from the first and second sets of images, respectively;

and wherein the computer is further configured to calculate the extent of the CHF also based on the extent to which heart rate values in the second series were above heart rate values in the first series.

19. The system of claim 18, wherein the computer is further configured to calculate the extent of CHF based on a duration after the period in which the heart rate values in the second series were above the heart rate values in the first series.

20. The system of claim 16, further comprising a head-mounted sensor configured to measure temperature of a region comprising skin on the user'"'"'s head (T_skin);

and the computer is further configured to utilize T_skin to compensate for effects of skin temperature on the facial blood flow pattern.