Systems and methods for non-contact heart rate sensing

• US 9,020,185 B2
• Filed: 09/28/2011
• Issued: 04/28/2015
• Est. Priority Date: 09/28/2011
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Mestha, Lalit K., Xu, Beilei
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): HEIDEMANN, JASON E
• Application Number:
  US13/247,575
• Publication Number:
  US 20130077823A1
• Time in Patent Office: 1,308 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/7257 using Fourier transforms
• CPC Class Codes: G06F 18/2134 based on separation criteria, e.g. independent component analysis
• CPC Class Codes: G06F 2218/04 Denoising
• CPC Class Codes: G06F 2218/22 Source localisation; Inverse modelling
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30076 Plethysmography
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

An embodiment generally relates to systems and methods for estimating heart rates of individuals using non-contact imaging. A processing module can process multi-spectral video images of individuals and detect skin blobs within different images of the multi-spectral video images. The skin blobs can be converted into time series signals and processed with a band pass filter. Further, the time series signals can be processed to separate pulse signals from unnecessary signals. The heart rate of the individual can be estimated according to the resulting time series signal processing.

Claims

1. A method of processing data, the method comprising:

processing, by a processor, a video of an individual, wherein the video was captured using one or more of a multi-spectral video camera, a multi-spectral near infrared video camera, or a multi-spectral short wave Infrared video camera, and wherein the processing comprises;

separating the video into a sequence of image data cubes each comprising a set of more than three entries, wherein each entry of each set of entries corresponds to a spectral frequency at which the video was captured, wherein the spectral frequency of each entry of the set of entries is in a range of 700 nanometers (nm) to 1700 nm;

identifying, from the image data cubes, a set of skin blobs corresponding to the set of entries;

processing the set of skin blobs to remove non-skin areas;

generating, from the set of skin blobs, a set of time series signals;

processing the set of time series signals to estimate a heart rate of the individual; and

providing the heart rate to a device.

2. The method of claim 1, wherein the processing further comprises:

processing the set of time series signals to identify a source signal that corresponds to a heart beat.

3. The method of claim 1, wherein identifying the set of skin blobs comprises:

identifying the set of skin blobs from a first frame of the image data cubes; and

tracking the set of skin blobs in successive frames of the image data cubes using motion-based video tracking.

4. The method of claim 1, wherein the processing further comprises:

processing the set of time series signals to separate independent source signals that contain a heart beat.

5. The method of claim 1, further comprising:

capturing the video.

6. The method of claim 1, wherein the processing further comprises:

processing the sequence of image data cubes to estimate the heart rate on a continuous basis.

7. The method of claim 1, wherein the processing further comprises:

prefiltering the set of time series signals by a band pass filter.

8. The method of claim 1, wherein generating the set of time series signals comprises:

computing a channel average of pixels in each skin blob of the set of skin blobs;

computing a global channel average based on the channel average of pixels in each skin blob; and

generating the set of time series signals by subtracting the global channel average from the channel average of pixels, and dividing a result of the subtraction by a global channel standard deviation.

9. The method of claim 1, wherein processing the set of time series signals comprises performing an independent component analysis (ICA) on the set of time series signals.

10. The method of claim 1, wherein estimating the heart rate of the individual comprises:

applying a Fast Fourier Transform (FFT) on a source signal from the set of time series signals to generate an FFT source signal; and

calculating a square of the FFT source signal; and

estimating the heart rate of the individual based on the calculation of the square of the FFT source signal.

11. The method of claim 10, wherein estimating the heart rate of the individual based on the calculation of the square of the FFT source signal comprises:

identifying a greatest magnitude of the square of the FFT source signal within a frequency band of interest.

12. The method of claim 1, wherein providing the heart rate to the device comprises:

outputting the heart rate of the individual to a display monitor.

13. The method of claim 1, wherein the processing further comprises:

outputting a source signal used to estimate the heart rate on a display monitor.

14. A system for processing data, the system comprising:

one or more filters configured to filter images of an individual into multi-spectral images;

a capture device configured to capture, from the one or more filters, the multi-spectral images as a video, wherein the capture device comprises one or more of a multi-spectral video camera, a multi-spectral near infrared video camera, or a multi-spectral short wave Infrared video camera; and

a processor coupled to the capture device and configured to perform actions comprising;

processing the video, wherein the processing comprises;

separating the video into a sequence of image data cubes each comprising a set of more than three entries, wherein each entry of each set of entries corresponds to a spectral frequency at which the multi-spectral video was captured, wherein the spectral frequency of each entry of the set of entries is in a range of 700 nanometers (nm) to 1700 nm;

identifying, from the image data cubes, a set of skin blobs corresponding to the set of entries;

processing the set of skin blobs to remove non-skin areas;

generating, from the set of skin blobs, a set of time series signals;

processing the set of time series signals to estimate a heart rate of the individual; and

providing the heart rate to a device.

15. The system of claim 14, wherein identifying the set of skin blobs comprises:

identifying the set of skin blobs from a first frame of the image data cubes; and

tracking the set of skin blobs in successive frames of the image data cubes using motion-based video tracking.

16. The system of claim 14, wherein the processing further comprises:

processing the set of time series signals to identify a source signal that corresponds to a heart beat.

17. The system of claim 14, wherein the processing further comprises:

processing the set of time series signals to separate independent source signals that contain a heart beat.

18. The system of claim 14, wherein the processing further comprises:

processing the sequence of image data cubes to estimate the heart rate on a continuous basis.

19. The system of claim 14, wherein the processing further comprises:

prefiltering the set of time series signals by a band pass filter.

20. The system of claim 14, wherein generating the set of time series signals comprises:

computing a channel average of pixels in each skin blob of the set of skin blobs;

computing a global channel average based on the channel average of pixels in each skin blob; and

generating the set of time series signals by subtracting the global channel average from the channel average of pixels, and dividing a result of the subtraction by a global channel standard deviation.

21. The system of claim 14, wherein processing the set of time series signals comprises performing an independent component analysis (ICA) on the set of time series signals.

22. The system of claim 14, wherein estimating the heart rate of the individual comprises:

applying a Fast Fourier Transform (FFT) on a source signal from the set of time series signals to generate an FFT source signal; and

calculating a square of the FFT source signal; and

estimating the heart rate of the individual based on the calculation of the square of the FFT source signal.

23. The system of claim 22, wherein estimating the heart rate of the individual based on the calculation of the square of the FFT source signal comprises:

identifying a greatest magnitude of the square of the FFT source signal within a frequency band of interest.

24. The system of claim 14, wherein providing the heart rate to the device comprises:

outputting the heart rate of the individual to a display monitor.

25. The system of claim 14, wherein the processing further comprises:

outputting a source signal used to estimate the heart rate on a display monitor.