Optical heartrate tracking

• US 9,962,095 B2
• Filed: 04/23/2013
• Issued: 05/08/2018
• Est. Priority Date: 04/23/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ahmad, Anis, Walker, Ben, Kennett, Daniel, Flavell, Andrew
• Primary Examiner(s): Patton, Amanda
• Application Number:
  US13/869,016
• Publication Number:
  US 20140316293A1
• Time in Patent Office: 1,841 Days
• Field of Search: 600508
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/6887 mounted on external non-worn devices, e.g. non-medical devices
• CPC Class Codes: A61B 5/7207 of noise induced by motion artifacts
• CPC Class Codes: A61B 5/725 using specific filters therefor, e.g. Kalman or adaptive filters A61B5/316 take precedence
• CPC Class Codes: A61B 5/7257 using Fourier transforms

Abstract

Heartrate tracking is performed entirely optically without the subject being required to wear any monitoring equipment by processing a combination of signals representing frames of video of the sinusoidal motion of a subject'"'"'s facial skin color changes captured by both IR and visible light (e.g., RGB—red/green/blue) cameras. The IR and RGB graphs that result from the processing are perfectly phase-shifted so that when the IR signal is going down in amplitude, the RGB signal is going up. Such phase-shifting enables the optical heartrate tracking to utilize diverse input feeds so that a tracked signal is accepted as the user'"'"'s true heartrate when both IR and RGB signals are well correlated.

Claims

1. One or more hardware-based computer-readable memory devices not consisting of propagated signals having computer readable instructions stored thereupon that, when executed by a computer, implement:

a face alignment and preparation pipeline configured to receive respective streams of infrared video frames and visible light video frames which include a facial image of a human subject and simultaneously process the received infrared video frames and visible light video frames to output a decomposed, frame-aligned, facial skin-only signal measurement comprising spatially-corresponding infrared light and visible light graphs that are phase-shifted so that as values in the infrared light graph are increasing, values in the spatially-corresponding visible light graph are decreasing, and vice-versa;

a signal-finding pipeline configured to;

i) receive the decomposed, frame-aligned, facial skin-only signal measurement, ii) compare the signal measurement in a current frame to a cached decomposed version of a head image in a previous frame, iii) sum up and average pixels having deltas between head images in the current frame and previous frame that trend in a consistent direction, and are within a set number of standard deviations of brightness, iv) generate a measurement of noisiness per pixel, v) apply at least one filter to the summed and averaged pixels, the filter being weighted by the noisiness measurement, vi) create a time-space representation of low-amplitude deltas in a skin-only facial image;

a signal-extracting pipeline configured to a) apply time-space smoothing or frequency-space smoothing to the time-space representation, b) apply a discrete Fourier transform to transform the time-space representation into a frequency-space representation for each of the infrared and visible streams, c) merge the frequency-space representations weighted by graph noise level; and

a heartrate identification pipeline configured to identify a heartrate of the human subject from the merged frequency-space representations.

2. The one or more hardware-based computer-readable memory devices of claim 1 in which the heartrate identification pipeline is configured to identify a highest peak in the merged frequency-space graphs.

3. The one or more hardware-based computer-readable memory devices of claim 2 in which the pipelines operate sequentially on a given video frame.

4. The one or more hardware-based computer-readable memory devices of claim 2 in which the pipelines operate in parallel.

5. The one or more hardware-based computer-readable memory devices of claim 2 further comprising configuring the heartrate identification pipeline to measure a noise level of the highest peak.

6. The one or more hardware-based computer-readable memory devices of claim 5 further comprising configuring the heartrate identification pipeline to bucket a location of the highest peak, weighted by the noise level into a frequency detector.

7. The one or more hardware-based computer-readable memory devices of claim 1 further comprising configuring the heartrate identification pipeline to apply a smoothing factor to nearby signals and sum the signals together.

8. The one or more hardware-based computer-readable memory devices of claim 7 further comprising configuring the heartrate identification pipeline to output a graph of the summed, smoothed signals.

9. The one or more hardware-based computer-readable memory devices of claim 8 further comprising configuring the heartrate identification pipeline to identify a heartrate signal from the graph if a peak result is above a set amplitude threshold.