Remote heart rate estimation

• US 9,852,507 B2
• Filed: 11/10/2015
• Issued: 12/26/2017
• Est. Priority Date: 11/10/2014
• Status: Active Grant

• Current Assignee: Nutritis Ltd.
• Original Assignee: Utah State University
• Inventors: Gunther, Jacob, Ruben, Nathan E.
• Primary Examiner(s): Mistry, Oneal R
• Application Number:
  US14/937,506
• Publication Number:
  US 20160132732A1
• Time in Patent Office: 777 Days
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30048 Heart; Cardiac
• CPC Class Codes: G06T 2207/30088 Skin; Dermal
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

For remote heart rate estimation, a method detects an object of interest (OOI) in each image of a video data and tracks the OOI in each image of the video data. The method identifies a region of interest (ROI) within the OOI and generates a plurality of super pixels from a plurality of pixels in each ROI. The method further generates a super-pixel time series from the plurality of super pixels in each image and removes interfering signals from the super-pixel time series. The method further models the super-pixel time series as a super-pixel model and calculates a heart beat signal from the super-pixel model. The method calculates heart characteristics from the heart beat signal. The heart characteristics include one or more of a heart rate, an inter-beat interval, and a heart rate variability.

Claims

1. A method comprising:

detecting, by use of a processor, an object of interest (OOI) in each image of a video data;

tracking the OOI in each image of the video data;

identifying a region of interest (ROI) within the OOI;

generating a plurality of super pixels from a plurality of pixels in each ROI;

generating a super-pixel time series from the plurality of super pixels in each image;

removing interfering signals from the super-pixel time series;

modeling the super-pixel time series as a super-pixel model;

calculating a heart beat signal from the super-pixel model, wherein an ith super pixel for a time t is modeled as y_i(t)=u_i(t)+h_i(t)+n_i(t), where y_i(t) is a measured pixel value, u_i(t) is a background signal representing contributions to the measured pixel value due to movement and lighting variations, h_i(t) is the heart beat signal, and n_i(t) is a sensor noise signal representing contributions to the measured pixel value due to sensor noise; and

calculating heart characteristics from the heart beat signal, the heart characteristics comprising one or more of a heart rate, an inter-beat interval, and a heart rate variability.

2. The method of claim 1, wherein the sensor noise signal n_i(t) is assumed to be independent, identically distributed Gaussian noise.

3. The method of claim 1, wherein the background signal u_i(t) is assumed to be smooth.

4. The method of claim 3, wherein the background signal u_i(t) is modeled as a first order Markov random process.

5. The method of claim 1, wherein the heart beat is assumed to be the same in each super pixel, wherein h_i(t)=h(t) for all i.

6. The method of claim 1, wherein the heart beat signal and a background signal are further calculated by optimizing

7. The method of claim 1, wherein the heart beat signal and a background signal are further calculated by optimizing

8. The method of claim 1, wherein the OOI is detected using cascaded object detection on red/green/blue (RGB) pixels of the video data and the OOI is tracked with sub-pixel resolution using spatial correlation-based methods.

9. The method of claim 1, wherein the OOI is detected and tracked using infrared band information.

10. The method of claim 1, wherein the video data is captured by one or more of a plurality of cameras, 3-color channel camera, a multispectral n-channel camera, an infrared camera, a depth camera, a 1 pixel sensor, and a servo controlled camera.

11. The method of claim 1, wherein each super pixel comprises one or more measured pixel values representing an average value of pixels in the ROI.

12. The method of claim 1, wherein the heart rate is calculated using one of a machine learning analysis of the heart beat signal, a peak of a Fourier transform of the heart beat signal, a power spectral density of the heart beat signal, a zero crossing rate of the heart beat signal, and a sliding correlation analysis of the heart beat signal.

13. The method of claim 1, wherein removing the interfering signals from the super-pixel time series comprises de-trending by modeling a background signal as a Gaussian process.

14. The method of claim 1, wherein removing the interfering signals from the super-pixel time series comprises de-trending by decorrelating the super-pixel time series with auxiliary signals derived from a position of a facebox and from other regions in the video data.

15. The method of claim 1, wherein removing the interfering signals comprises band pass filtering to remove signals outside a frequency band of a normal heart rate.

16. An apparatus comprising:

a camera;

a processor;

a memory storing code executable by the processor to perform;

detecting an object of interest (OOI) in each image of a video data captured by the camera;

tracking the OOI in each image of the video data;

identifying a region of interest (ROI) within the OOI;

generating a plurality of super pixels from a plurality of pixels in each ROI;

generating a super-pixel time series from the plurality of super pixels in each image;

removing interfering signals from the super-pixel time series;

modeling the super-pixel time series as a super-pixel model;

calculating a heart beat signal from the super-pixel model, wherein an ith super pixel for a time t is modeled as y_i(t)=u_i(t)+h_i(t)+n_i(t), where y_i(t) is a measured pixel value, u_i(t) is a background signal representing contributions to the measured pixel value due to movement and lighting variations, h_i(t) is the heart beat signal, and n_i(t) is a sensor noise signal representing contributions to the measured pixel value due to sensor noise; and

calculating heart characteristics from the heart beat signal, the heart characteristics comprising one or more of a heart rate, an inter-beat interval, and a heart rate variability.

17. The apparatus of claim 16, wherein the heart beat signal and a background signal are further calculated by optimizing

18. The apparatus of claim 16, wherein the heart beat signal and a background signal are further calculated by optimizing

19. A program product comprising a non-transitory computer readable storage medium storing code executable by a processor to perform:

detecting an object of interest (OOI) in each image of a video data captured by the camera;

tracking the OOI in each image of the video data;

identifying a region of interest (ROI) within the OOI;

generating a plurality of super pixels from a plurality of pixels in each ROI;

generating a super-pixel time series from the plurality of super pixels in each image;

removing interfering signals from the super-pixel time series;

modeling the super-pixel time series as a super-pixel model;

calculating a heart beat signal from the super-pixel model, wherein an ith super pixel for a time t is modeled as y_i(t)=u_i(t)+h_i(t)+n_i(t), where y_i(t) is a measured pixel value, u_i(t) is a background signal representing contributions to the measured pixel value due to movement and lighting variations, h_i(t) is the heart beat signal, and n_i(t) is a sensor noise signal representing contributions to the measured pixel value due to sensor noise; and

calculating heart characteristics from the heart beat signal, the heart characteristics comprising one or more of a heart rate, an inter-beat interval, and a heart rate variability.

20. The program product of claim 19, wherein the heart beat signal and a background signal are further calculated by optimizing