OPTICAL HEARTRATE TRACKING

• US 20140316293A1
• Filed: 04/23/2013
• Published: 10/23/2014
• Est. Priority Date: 04/23/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ahmad, Anis, Kennett, Daniel, Walker, Ben, Flavell, Andrew
• Granted Patent: US 9,962,095 B2
• Time in Patent Office: Days
• US Class Current: 600/508
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/6887 mounted on external non-worn devices, e.g. non-medical devices
• CPC Class Codes: A61B 5/7207 of noise induced by motion artifacts
• CPC Class Codes: A61B 5/725 using specific filters therefor, e.g. Kalman or adaptive filters A61B5/316 take precedence
• CPC Class Codes: A61B 5/7257 using Fourier transforms

Abstract

Heartrate tracking is performed entirely optically without the subject being required to wear any monitoring equipment by processing a combination of signals representing frames of video of the sinusoidal motion of a subject'"'"'s facial skin color changes captured by both IR and visible light (e.g., RGB—red/green/blue) cameras. The IR and RGB graphs that result from the processing are perfectly phase-shifted so that when the IR signal is going down in amplitude, the RGB signal is going up. Such phase-shifting enables the optical heartrate tracking to utilize diverse input feeds so that a tracked signal is accepted as the user'"'"'s true heartrate when both IR and RGB signals are well correlated.

Claims

1. A computer-implemented method for tracking a heartrate of a human subject, the method comprising the steps of:

receiving a stream of infrared (IR) video frames captured by an IR camera, one or more of the received IR video frames including a facial image of the subject;

receiving a stream of visible light video frames captured by a visible light camera, one or more of the received visible light video frames including the facial image of the subject;

processing the received IR video frames and visible light video frames to detect changes in reflectivity of the facial image; and

correlating the detected changes from a processed IR video frame with detected changes from a processed visible video frame to identify the heartrate of the subject.

2. The computer-implemented method of claim 1 further including a step of determining a likelihood that pixels in the facial image are non-skin pixels.

3. The computer-implemented method of claim 2 further including a step of eliminating the likely non-skin pixels from processing, the likely non-skin pixels including one of hair, eyes, nostrils, or mouth.

4. The computer-implemented method of claim 3 further including a step of decomposing the processed video frames from multiple channels into a single signal measurement in one channel.

5. The computer-implemented method of claim 4 further including a step of applying a Kalman filter to find an estimated delta between successive frames in the single signal measurement.

6. The computer-implemented method of claim 5 further including a step of successively applying Kalman filters to replace a given video frame with a previous video frame if the noise of the given video frame exceeds a threshold.

7. The computer-implemented method of claim 6 further including a step of generating a time-space representation of low-amplitude deltas in a skin-only facial image.

8. The computer-implemented method of claim 7 further including a step of applying one of time-space smoothing or frequency-space smoothing to the time-space representation.

9. The computer-implemented method of claim 8 further including a step of applying a discrete Fourier transform to transform any remaining signal window into respective frequency-space graphs associated with the IR and visible light video frames.

10. The computer-implemented method of claim 9 further including a step of merging the frequency-space graphs weighted by graph noise level.

11. The computer-implemented method of claim 10 further including steps of identifying a highest peak in the merged frequency-space graphs, measuring a noise level of the highest peak, and bucketing a location of the highest peak, weighted by the noise level into a frequency detector.

12. The computer-implemented method of claim 11 further including steps of storing a group of successive video frame signals, applying a smoothing factor to each stored signal, summing the smoothed signals, graphing the summed smoothed signals, and identifying a heartrate signal from the graph if the peak exceeds a threshold.

13. One or more computer-readable storage media having computer readable instructions stored thereupon that, when executed by a computer, implement:

a face alignment and preparation pipeline arranged for receiving respective streams of infrared video frames and visible light video frames which include a facial image of a human subject and simultaneously processing the received infrared video frames and visible light video frames to output a decomposed, frame-aligned, facial skin-only signal measurement;

a signal-finding pipeline arranged for;

i) receiving the decomposed, frame-aligned, facial skin-only signal measurement, ii) comparing the signal measurement in a current frame to a cached decomposed version of a head image in a previous frame, iii) summing up and averaging pixels having deltas between head images in the current frame and previous frame that trend in a consistent direction, and are within a set number of standard deviations of brightness, iv) generating a measurement of noisiness per pixel, v) applying at least one filter to the summed and averaged pixels, the filter being weighted by the noisiness measurement, vi) creating a time-space representation of low-amplitude deltas in a skin-only facial image; and

a signal-extracting pipeline arranged for a) applying time-space smoothing or frequency-space smoothing to the time-space representation, b) applying a discrete Fourier transform to transform the time-space representation into a frequency-space representation for each of the IR and visible streams, c) merging the frequency-space representations weighted by graph noise level.

14. The one or more computer-readable storage media of claim 13 which further implement a heartrate identification pipeline for identifying a highest peak in the merged frequency-space graphs, measuring a noise level of the highest peak, and bucketing a location of the highest peak, weighted by the noise level into a frequency detector.

15. The one or more computer-readable storage media of claim 14 in which the pipelines operate sequentially on a given video frame.

16. The one or more computer-readable storage media of claim 14 in which the pipelines operate in parallel.

17. A computing platform, comprising:

at least one processor; and

one or more computer-readable storage media storing instructions which, when executed by the at least one processor, perform a method comprising the steps of

obtaining infrared (IR) and visible light video frame feeds of a human subject'"'"'s face captured by respective IR and visible light cameras;

obtaining an estimate of a location of the subject'"'"'s head within a space monitored by the cameras;

applying motion compensation to align the subject'"'"'s head in a current frame with a previous frame on a per-pixel basis;

creating a facial-skin only representation of the subject'"'"'s face;

decomposing the video feeds into a single generic signal feed;

tracking changes in the single generic signal feed, the changes representing sinusoidal motion of the subject'"'"'s facial skin color changes; and

identifying the subject'"'"'s heartrate from the tracked changes.

18. The computing platform of claim 17 further comprising an interface arranged to operatively couple a video capture device including an IR camera and a visible light camera.

19. The computing platform of claim 18 in which the IR camera is configured to operate as a depth camera.

20. The computing platform of claim 17 in which the tracking is performed while the subject is in motion within the monitored space.