SYSTEM AND METHOD FOR CAMERA-BASED HEART RATE TRACKING

• US 20190328247A1
• Filed: 07/09/2019
• Published: 10/31/2019
• Est. Priority Date: 11/14/2016
• Status: Active Grant

• Current Assignee: NuraLogix Corporation
• Original Assignee: NuraLogix Corporation
• Inventors: LEE, Kang, KABAKOV, Evgueni, LEVY, Phil
• Granted Patent: US 10,702,173 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/02427 Details of sensor
• CPC Class Codes: A61B 5/0255 Recording instruments specially adapted therefor
• CPC Class Codes: A61B 5/026 Measuring blood flow A61B3/1233, A61B3/1241 take precedence
• CPC Class Codes: A61B 5/0261 using optical means, e.g. infrared light
• CPC Class Codes: A61B 5/14546 for measuring analytes not otherwise provided for, e.g. ions, cytochromes
• CPC Class Codes: A61B 5/1455 using optical sensors, e.g. spectral photometrical oximeters
• CPC Class Codes: A61B 5/725 using specific filters therefor, e.g. Kalman or adaptive filters A61B5/316 take precedence
• CPC Class Codes: A61B 5/7253 characterised by using transforms
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/7278 Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: A61B 5/7485 Automatic selection of region of interest
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30104 Vascular flow; Blood flow; Perfusion
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A system and method for camera-based heart rate tracking. The method includes: determining bit values from a set of bitplanes in a captured image sequence that represent the HC changes; determining a facial blood flow data signal for each of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images based on the HC changes; applying a band-pass filter of a passband approximating the heart rate to each of the blood flow data signals; applying a Hilbert transform to each of the blood flow data signals; adjusting the blood flow data signals from revolving phase-angles into linear phase segments; determining an instantaneous heart rate for each the blood flow data signals; applying a weighting to each of the instantaneous heart rates; and averaging the weighted instantaneous heart rates.

Claims

1. A method for heart rate tracking of a subject, the method comprising:

receiving a plurality of images captured of the subject;

determining hemoglobin concentration (HC) changes from the plurality of images;

determining one or more blood flow data signals based on the HC changes;

determining a heart rate as an average of the one or more blood flow data signals; and

outputting the determined heart rate.

2. The method of claim 1, wherein determining the heart rate comprises determining an instantaneous heart rate for each the blood flow data signals and averaging the instantaneous heart rates.

3. The method of claim 2, wherein determining the heart rate further comprises assigning a weighting to each of the instantaneous heart rates, and averaging the instantaneous heart rates comprises averaging the weightings.

4. The method of claim 1, wherein each of the blood flow data signals is associated with a separate one of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images.

5. The method of claim 4, wherein the ROIs are captured from one or more of:

the face, the wrist, or the neck of the subject.

6. The method of claim 1, wherein the HC changes of the subject are represented by bit values from a set of bitplanes in the captured image sequence, the set of bitplanes being those that are determined to approximately maximize a signal-to-noise ratio (SNR), and wherein determining the HC changes comprises using a machine learning model trained with an HC changes training set, the HC changes training set comprising bit values from each bitplane of images captured from a set of subjects for which heart rate is known.

7. The method of claim 6, wherein determining the set of bitplanes that maximize SNR comprises:

performing pixelwise image subtraction and addition of bitplane vectors to maximize signal differences in all ROIs over a predetermined time period;

identifying bit values from bitplanes that increase the signal differentiation and bit values from bitplanes that decrease the signal differentiation or do not contribute to signal differentiation; and

discarding the bit values from the bitplanes that decrease the signal differentiation or do not contribute to signal differentiation.

8. The method of claim 6, wherein the machine learning model comprises a Long Short Term Memory (LSTM) neural network or a non-linear Support Vector Machine.

9. The method of claim 1, wherein the method further comprises applying a band-pass filter of a passband approximating the heart rate to each of the blood flow data signals, the passband being in a range of approximately 0.6 hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1 hertz.

10. The method of claim 1, further comprising adjusting the blood flow data signals from revolving phase-angles into linear phase segments, and applying a differential filter to the linear phase segments to convert phase-angle data into frequency units representing a count value, the count value representing the heart rate.

11. A system for heart rate tracking of a subject, the system comprising one or more processors and a data storage device, the one or more processors configured to execute:

a TOI module to;

receive a plurality of images of the subject;

determine hemoglobin concentration (HC) changes from the plurality of images; and

determine one or more blood flow data signals based on the HC changes;

a summation module to determine a heart rate as an average of the one or more blood flow data signals; and

an output module to output the heart rate.

12. The system of claim 11, wherein determining the heart rate comprises using a derivative module to determine an instantaneous heart rate for each the blood flow data signals and a summation module to average the instantaneous heart rates.

13. The system of claim 12, wherein determining the heart rate further comprises using a weighting module to assign a weighting to each of the instantaneous heart rates, and the summation module averaging the instantaneous heart rates comprises averaging the weightings.

14. The system of claim 11, wherein each of the blood flow data signals is associated with a separate one of a plurality of predetermined regions of interest (ROIs) of the subject captured by the images.

15. The system of claim 14, wherein the ROIs are captured from one or more of:

the face, the wrist, or the neck of the subject.

16. The system of claim 11, wherein the HC changes of the subject are represented by bit values from a set of bitplanes in the captured image sequence, the set of bitplanes being those that are determined to approximately maximize a signal-to-noise ratio (SNR), and wherein determining the HC changes comprises using a machine learning model trained with an HC changes training set, the HC changes training set comprising bit values from each bitplane of images captured from a set of subjects for which heart rate is known.

17. The system of claim 16, wherein determining the set of bitplanes that maximize SNR comprises:

performing pixelwise image subtraction and addition of bitplane vectors to maximize signal differences in all ROIs over a predetermined time period;

identifying bit values from bitplanes that increase the signal differentiation and bit values from bitplanes that decrease the signal differentiation or do not contribute to signal differentiation; and

discarding the bit values from the bitplanes that decrease the signal differentiation or do not contribute to signal differentiation.

18. The system of claim 16, wherein the machine learning model comprises a Long Short Term Memory (LSTM) neural network or a non-linear Support Vector Machine.

19. The system of claim 11, further comprising a band-pass filter to apply a band-pass filter of a passband approximating the heart rate to each of the blood flow data signals, the passband being in a range of approximately 0.6 hertz to 1.2 hertz, where 60 heartbeats-per-minute is equivalent to 1 hertz.

20. The system of claim 11, further comprising a derivative module to adjust the blood flow data signals from revolving phase-angles into linear phase segments, and applying a differential filter to the linear phase segments to convert phase-angle data into frequency units representing a count value, the count value representing the heart rate.