System and method for camera-based heart rate tracking

• US 10,448,847 B2
• Filed: 10/05/2018
• Issued: 10/22/2019
• Est. Priority Date: 11/14/2016
• Status: Active Grant

• Current Assignee: NuraLogix Corporation
• Original Assignee: NuraLogix Corporation
• Inventors: Lee, Kang, Kabakov, Evgueni, Levy, Phil
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US16/153,214
• Publication Number:
  US 20190038159A1
• Time in Patent Office: 382 Days
• Field of Search: None
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/02427 Details of sensor
• CPC Class Codes: A61B 5/0255 Recording instruments specially adapted therefor
• CPC Class Codes: A61B 5/026 Measuring blood flow A61B3/1233, A61B3/1241 take precedence
• CPC Class Codes: A61B 5/0261 using optical means, e.g. infrared light
• CPC Class Codes: A61B 5/14546 for measuring analytes not otherwise provided for, e.g. ions, cytochromes
• CPC Class Codes: A61B 5/1455 using optical sensors, e.g. spectral photometrical oximeters
• CPC Class Codes: A61B 5/725 using specific filters therefor, e.g. Kalman or adaptive filters A61B5/316 take precedence
• CPC Class Codes: A61B 5/7253 characterised by using transforms
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/7278 Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: A61B 5/7485 Automatic selection of region of interest
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30104 Vascular flow; Blood flow; Perfusion
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A system and method for camera-based heart rate tracking. The method includes: determining bit values from a set of bitplanes in a captured image sequence that represent the HC changes; determining a facial blood flow data signal for each of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images based on the HC changes; applying a band-pass filter of a passband approximating the heart rate to each of the blood flow data signals; applying a Hilbert transform to each of the blood flow data signals; adjusting the blood flow data signals from revolving phase-angles into linear phase segments; determining an instantaneous heart rate for each the blood flow data signals; applying a weighting to each of the instantaneous heart rates; and averaging the weighted instantaneous heart rates.

Claims

1. A method for camera-based heart rate tracking of a human subject, the method comprising:

receiving a captured image sequence of light re-emitted from the skin of the human subject;

determining, using a machine learning model trained with a hemoglobin concentration (HC) changes training set, HC changes of the subject;

determining a blood flow data signal for each of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images based on the HC changes;

determining an instantaneous heart rate for each the blood flow data signals;

applying a weighting to each of the instantaneous heart rates;

averaging the weighted instantaneous heart rates; and

outputting the average heart rate.

2. The method of claim 1, wherein the ROIs are captured from one or more of:

the face, the wrist or the neck of the subject.

3. The method of claim 1, wherein the ROIs are non-overlapping.

4. The method of claim 1, wherein the HC changes of the subject are represented by bit values from a set of bitplanes in the captured image sequence, the set of bitplanes being those that are determined to approximately maximize a signal-to-noise ratio (SNR), the HC changes training set comprising bit values from each bitplane of images captured from a set of subjects for which heart rate is known.

5. The method of claim 4, wherein determining the set of bitplanes that maximize SNR comprises:

performing pixelwise image subtraction and addition of bitplane vectors to maximize signal differences in all ROIs over a predetermined time period;

identifying bit values from bitplanes that increase the signal differentiation and bit values from bitplanes that decrease the signal differentiation or do not contribute to signal differentiation; and

discarding the bit values from the bitplanes that decrease the signal differentiation or do not contribute to signal differentiation.

6. The method of claim 1, wherein the machine learning model comprises a Long Short Term Memory (LSTM) neural network or a non-linear Support Vector Machine.

7. The method of claim 1, wherein the method further comprises applying a band-pass filter of a passband approximating the heart rate to each of the blood flow data signals, the passband being in a range of approximately 0.6 hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1 hertz.

8. The method of claim 1, wherein determining the instantaneous heart rate for each the blood flow data signals comprises adjusting the blood flow data signals from revolving phase-angles into linear phase segments, and applying a differential filter to the linear phase segments to convert phase-angle data into frequency units representing a count value, the count value for each of the ROIs represents the instantaneous heart rate.

9. The method of claim 8, further comprising linearizing and differentiating the revolving phase-angles on a phase continuum scale to determine the instantaneous heart rate.

10. The method of claim 1, wherein the weighting is integrated over an interval in the range of approximately one second to ten seconds.

11. A system for camera-based heart rate tracking of a human subject, the system comprising one or more processors and a data storage device, the one or more processors configured to execute:

a TOI module configured to;

receive a captured image sequence of light re-emitted from the skin of a human subject;

determine, using a machine learning model trained with a hemoglobin concentration (HC) changes training set, HC changes of the subject; and

determine a blood flow data signal for each of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images based on the HC changes;

a derivative module to determine an instantaneous heart rate for each the blood flow data signals;

a weighting module to apply a weighting to each of the instantaneous heart rates;

a summation module to average the weighted instantaneous heart rates; and

an output module to output the average heart rate.

12. The system of claim 11, wherein the ROIs are captured from one or more of:

the face, the wrist or the neck of the subject.

13. The system of claim 11, wherein the ROIs are non-overlapping.

14. The system of claim 11, wherein the HC changes of the subject are represented by bit values from a set of bitplanes in the captured image sequence, the set of bitplanes being those that are determined to approximately maximize a signal-to-noise ratio (SNR), the HC changes training set comprising bit values from each bitplane of images captured from a set of subjects for which heart rate is known.

15. The system of claim 14, wherein the TOI module determines a set of bitplanes that maximize SNR by:

performing pixelwise image subtraction and addition of bitplane vectors to maximize signal differences in all ROIs over a predetermined time period;

identifying bit values from bitplanes that increase the signal differentiation and bit values from bitplanes that decrease the signal differentiation or do not contribute to signal differentiation; and

discarding the bit values from the bitplanes that decrease the signal differentiation or do not contribute to signal differentiation.

16. The system of claim 11 further comprising a band-pass filter having a passband approximating the heart rate applied to each of the blood flow data signals, wherein the passband is in a range of approximately 0.6 hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1 hertz.

17. The system of claim 11, wherein the derivative module determines the instantaneous heart rate for each the blood flow data signals by adjusting the blood flow data signals from revolving phase-angles into linear phase segments, and applying a differential filter to the linear phase segments to convert the phase-angle data into frequency units representing a count value, the count value for each of the ROIs represents the instantaneous heart rate.

18. The system of claim 17, wherein the derivative module linearizes and differentiates the revolving phase-angles on a phase continuum scale to determine the instantaneous heart rate.

19. The system of claim 11, wherein the weighting applied by the weighting module is integrated over an interval in the range of approximately one second to ten seconds.

20. The system of claim 11, wherein the weighting applied by the weighting module is integrated over an interval of approximately five seconds.