Utilizing correlations between PPG signals and iPPG signals to improve detection of physiological responses

• US 10,799,122 B2
• Filed: 11/20/2019
• Issued: 10/13/2020
• Est. Priority Date: 06/14/2015
• Status: Active Grant

• Current Assignee: Facense Ltd.
• Original Assignee: Facense Ltd.
• Inventors: Tzvieli, Arie, Frank, Ari M, Tzvieli, Ori, Thieberger, Gil
• Primary Examiner(s): Couso, Jose L
• Application Number:
  US16/689,959
• Publication Number:
  US 20200085312A1
• Time in Patent Office: 328 Days
• CPC Class Codes: A61B 2562/0271 Thermal or temperature sensors
• CPC Class Codes: A61B 2562/0276 comprising a thermosensitive compound
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/015 By temperature mapping of body part
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/14552 Details of sensors specially adapted therefor
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6814 Head
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/748 Selection of a region of interest, e.g. using a graphics tablet
• CPC Class Codes: G01J 2005/0077 Imaging
• CPC Class Codes: G01J 5/0265 Handheld, portable ear thermometers G01J5/049
• CPC Class Codes: G01J 5/12 using thermoelectric elements, e.g. thermocouples
• CPC Class Codes: G01J 5/485 Temperature profile

Abstract

Disclosed herein are systems and methods for detecting a physiological response using different types of photoplethysmography sensors. Examples of physiological responses that may be detected include an allergic reaction, a stroke, a migraine, stress, certain emotional responses, manifestation of pain, and blood pressure. In one embodiment, a head-mounted contact photoplethysmography device measures a signal indicative of photoplethysmogram signal at a first region that includes exposed skin on a user'"'"'s head (PPG signal). A camera, located more than 10 mm away from the user'"'"'s head, captures images of a second region that includes exposed skin on the user'"'"'s head. A computer detects the physiological response based on: (i) imaging photoplethysmogram signals (iPPG signals) recognizable in the images, and (ii) correlations between the PPG signal and the iPPG signals.

Claims

1. A system comprising:

a head-mounted contact photoplethysmography device configured to measure a signal indicative of photoplethysmogram signal (PPG signal) at a first region that comprises exposed skin on a user'"'"'s head;

a camera configured to capture images of a second region that comprises exposed skin on the user'"'"'s head;

wherein the camera is located more than 10 mm away from the user'"'"'s head; and

a computer configured to detect a physiological response based on;

(i) imaging photoplethysmogram signals (iPPG signals) recognizable in the images, and (ii) correlations between the PPG signal and the iPPG signals.

2. The system of claim 1, wherein the camera is an inward-facing head-mounted camera, and both the camera and the head-mounted contact photoplethysmography device are physically coupled to smartglasses or to a smart-helmet, which is designed to measure the user in day-to-day activities, over a duration of weeks, months, and/or years.

3. The system of claim 1, wherein the camera is integrated in a non-wearable device, and the computer is further configured to authenticate that the user is using the non-wearable device by checking whether correlations between the PPG signal and the iPPG signals reach a predetermined threshold.

4. The system of claim 1, wherein the computer is further configured to extract the iPPG signals from the images based on values of time-segments in which the iPPG signals were expected to appear as a function of locations of respective regions of the iPPG signals relative to location of the contact PPG device.

5. The system of claim 1, wherein the computer is further configured to extract from the PPG signal at least one of the following parameters:

systolic peak, dicrotic notch, diastolic peak, interbeat interval, and systolic-diastolic peak-to-peak time; and

the computer is further configured to utilize at least one of the parameters to correlate between the PPG signal and the iPPG signals.

6. The system of claim 1, wherein the computer is further configured to extract from the PPG signal times corresponding to at least one of the following types of fiducial points:

systolic peaks, dicrotic notches, and diastolic peaks; and

the computer is further configured to utilize the times to extract corresponding fiducial points in the iPPG signals by adding to the times an offset in order to determine when the corresponding fiducial points manifest in the iPPG signals.

7. The system of claim 6, wherein the offset is proportional to at least one of the following values:

a heart rate of the user, and a blood pressure value of the user.

8. The system of claim 1, wherein the iPPG signals comprise multiple values for different sub-regions of the second region, and the physiological response is detected based on differences between amplitudes of the values recognizable in the different sub-regions of the second region.

9. The system of claim 8, wherein the physiological response is indicative of an allergic reaction, and the sub-regions of the second region comprise portions of at least two of the following areas on the user'"'"'s face:

nose, upper lip, lips, cheeks, temples, periorbital area around the eyes, and the forehead.

10. The system of claim 8, wherein the physiological response is indicative of a stroke, and the sub-regions of the second region comprise at least one of the following pairs on the user'"'"'s face:

left and right cheeks, left and right temples, left and right sides of the forehead, and left and right sides of the periorbital area around the eyes.

11. The system of claim 8, wherein the physiological response is indicative of a migraine, and the sub-regions of the second region comprise at least one of the following pairs on the user'"'"'s face:

left and right sides of the forehead, left and right temples, left and right sides of the periorbital area around the eyes, and left and right cheeks.

12. The system of claim 8, wherein the physiological response is indicative of a blood pressure value that is calculated based on differences in pulse transit times detectable in the sub-regions of the second region;

wherein the sub-regions comprise at least two of the following areas on the user'"'"'s face;

left temple, right temple, left side of the forehead, right side of the forehead, left check, right cheek, nose, periorbital area around the left eye, and periorbital area around the right eye.

13. The system of claim 8, wherein the physiological response is indicative of at least one of stress, emotional response, and pain, which are calculated based on changes to hemoglobin concentrations observable in the iPPG signals relative to previous measurements of hemoglobin concentrations observable in the iPPG signals of the user;

wherein the sub-regions comprise at least two of the following areas on the user'"'"'s face;

lips, upper lip, chin, left temple, right temple, left side of the forehead, right side of the forehead, left check, right cheek, left ear lobe, right ear lobe, nose, periorbital area around the left eye, and periorbital area around the right eye.

14. The system of claim 8, wherein the camera comprises two or more inward-facing head-mounted cameras, and the second region comprises two or more regions on the user'"'"'s head that are respectively captured by the two or more inward-facing head-mounted cameras.

15. The system of claim 1, wherein the computer is further configured to refrain from detecting the physiological response when a difference between a first heart rate determine based on the PPG signal and a second heart rate determine based on the iPPG signals is greater than 5 beats per minute.

16. The system of claim 1, further comprising a head-mounted sensor configured to measure a signal indicative of timing of the user'"'"'s inhalation phase and/or exhalation phase (respiratory-phase signal);

wherein the computer is further configured to detect the physiological response also based on correlations between timings of the iPPG signals and the respiratory-phase signal.

17. A method comprising:

measuring a signal indicative of a photoplethysmogram signal (PPG signal) at a first region comprising exposed skin on a user'"'"'s head utilizing a head-mounted contact photoplethysmography device;

capturing images of a second region comprising exposed skin on the user'"'"'s head utilizing a camera;

wherein the camera is located more than 10 mm away from the user'"'"'s head; and

detecting a physiological response based on;

(i) imaging photoplethysmogram signals (iPPG signals) recognizable in the images, and (ii) correlations between the PPG signal and the iPPG signals.

18. The method of claim 17, wherein the physiological response is indicative of a value of blood pressure, and is calculated based on differences in pulse transit times detectable in portions of the images that include sub-regions of the second region;

and wherein the sub-regions comprise at least two of the following areas on the user'"'"'s face;

left temple, right temple, left side of the forehead, right side of the forehead, left check, right cheek, nose, periorbital area around the left eye, and periorbital area around the right eye.

19. The method of claim 17, wherein the physiological response is indicative of at least one of stress, an emotional response, and pain, which are calculated based on changes to hemoglobin concentrations observable in the iPPG signals relative to previous measurements of hemoglobin concentrations observable in the iPPG signals of the user;

wherein the sub-regions comprise at least two of the following areas on the user'"'"'s face;

lips, upper lip, chin, left temple, right temple, left side of the forehead, right side of the forehead, left check, right cheek, left ear lobe, right ear lobe, nose, periorbital area around the left eye, and periorbital area around the right eye.

20. A head-mounted imaging photoplethysmography system, comprising:

a head-mounted contact photoplethysmography device configured to measure a signal indicative of a photoplethysmogram signal (PPG signal) at a first region comprising exposed skin on a user'"'"'s head;

a head-mounted camera configured to capture images of a second region comprising exposed skin on the user'"'"'s head;

wherein the camera is located more than 10 mm away from the user'"'"'s head; and

a head-mounted computer configured to;

identify times at which fiducial points appear in the PPG signal;

calculate, based on the times, time-segments in which the fiducial points are expected to appear in imaging photoplethysmogram signals (iPPG signals) recognizable the images; and

detect a physiological response based on values of the iPPG signals during the time-segments.