Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions

• US 10,335,045 B2
• Filed: 06/23/2017
• Issued: 07/02/2019
• Est. Priority Date: 06/24/2016
• Status: Active Grant

• Current Assignee: Fondazione Bruno Kessler, Universita' Degli Studi Di Trento (Repubblica Italiana), The Research Foundation for The State University of New York (State University of New York)
• Original Assignee: Universita' Degli Studi Di Trento (Repubblica Italiana)
• Inventors: Sebe, Niculae, Alameda-Pineda, Xavier, Tulyakov, Sergey, Ricci, Elisa, Yin, Lijun, Cohn, Jeffrey F.
• Primary Examiner(s): Mistry, Oneal R
• Application Number:
  US15/631,346
• Publication Number:
  US 20170367590A1
• Time in Patent Office: 739 Days
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/021 Measuring pressure in heart or blood vessels A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30048 Heart; Cardiac
• CPC Class Codes: G06T 2207/30076 Plethysmography
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

Recent studies in computer vision have shown that, while practically invisible to a human observer, skin color changes due to blood flow can be captured on face videos and, surprisingly, be used to estimate the heart rate (HR). While considerable progress has been made in the last few years, still many issues remain open. In particular, state-of-the-art approaches are not robust enough to operate in natural conditions (e.g. in case of spontaneous movements, facial expressions, or illumination changes). Opposite to previous approaches that estimate the HR by processing all the skin pixels inside a fixed region of interest, we introduce a strategy to dynamically select face regions useful for robust HR estimation. The present approach, inspired by recent advances on matrix completion theory, allows us to predict the HR while simultaneously discover the best regions of the face to be used for estimation. Thorough experimental evaluation conducted on public benchmarks suggests that the proposed approach significantly outperforms state-of-the-art HR estimation methods in naturalistic conditions.

Claims

1. A method of determining heart rate through observation of a human face, comprising:

acquiring with at least one automated camera, a time series of images of a human face, wherein the time series of images are subject to variations between respective images of the time series in illumination and facial movements;

adaptively selecting, with the at least one automated processor, a subset of the regions of interest of respective images of the time series of images of the human face, that exhibit a more statistically reliable heart-rate-determined variation than a non-selected subset of regions of the respective images of the human face;

based at least on the adaptively selected subset of regions of interest of the respective images of the time series of the human face that exhibit the reliable heart-rate-determined variation, determining a heart rate, and updating the adaptively selected subset of the regions of interest that exhibit the reliable heart-rate-determined variation; and

outputting a signal corresponding to the determined heart rate.

2. The method according to claim 1, wherein the regions of interest are selected according to at least matrix completion theory.

3. The method according to claim 1, wherein the heart rate is determined based on at least matrix completion theory.

4. The method according to claim 1, wherein the selected subset is selected dependent on at least a noise parameter of respective features of the time series of images.

5. The method according to claim 1, wherein the selected subset is selected dependent on at least a movement of the human face represented in the time series of images.

6. The method according to claim 1, wherein the selected subset is selected dependent on at least changes represented in the time series of images which represent human facial expressions.

7. The method according to claim 1, further comprising tracking the face in the time series of images to follow rigid head movements.

8. The method according to claim 1, further comprising detecting chrominance features from the time series of images comprising video images, and assessing the heart rate-determined variation based on the detected chrominance features.

9. The method according to claim 1, wherein the adaptively selected subset of the regions of interest exhibit the reliable heart-rate-determined variation through an entire period of heart rate estimation.

10. The method according to claim 1, wherein the reliable heart-rate-determined variation is a variation in chrominance.

11. The method according to claim 1, wherein the heart rate is determined in a process employing a cardiac cycle responsive filter.

12. The method according to claim 1, further comprising simultaneously recovering an unknown low-rank matrix and an underlying data mask, corresponding to most statistically reliable heart-rate-determined variation observations of the human face according to a reliability statistic.

13. A method of determining heart rate from video images, comprising:

processing, with the at least one automated processor, a stream of video images of a face from at least one automated camera, to extract a plurality of face regions;

computing chrominance features of the plurality of face regions, with at least one automated processor;

jointly estimating an underlying low-rank feature matrix and a mask of a selected subset of the plurality of face regions which have a higher statistical reliability than a non-selected subset of the plurality of face regions, using a self-adaptive matrix completion algorithm, with the at least one automated processor; and

computing the heart rate from a signal estimate provided by the self-adaptive matrix completion algorithm, with the at least one automated processor.

14. The method according to claim 13, wherein said processing comprises warping a representation of the face into rectangles using a piece-wise linear warping procedure, and dividing rectangles into a grid containing the plurality of face regions.

15. The method according to claim 14, the selected subset of the plurality of face regions being further selected to be robust to facial movements and expressions, while being sufficiently discriminant to account for changes in skin color responsive to cardiac cycle variation for said computing the heart rate.

16. The method according to claim 13, wherein said computing chrominance features comprises:

for each pixel, computing a chrominance signal C as a linear combination of two signals X_f and Y_f, such that C=X_f−

α

Y_f, where

17. The method according to claim 13, wherein said jointly estimating comprises enforcing a detection of chrominance feature variations that occur within a heart-rate frequency range.

18. The method according to claim 13, wherein said jointly estimating comprises masking extracted regions of the plurality of face regions dependent on at least facial movement dependent changes.

19. The method according to claim 13, wherein said jointly estimating comprises determining a local standard deviation over time of each extracted region of the plurality of face regions.

20. The method according to claim 13, wherein said jointly estimating comprises employing, by the at least one automated processor, an alternating direction method of multipliers (ADMM), which solves an optimization problem by alternating a direction of the optimization while keeping other directions fixed.

21. The method according to claim 20, wherein the solving of the optimization problem comprises repetitively performing the following three steps until convergence:

E/M-step

with fixed F and Z, obtaining optimal values of E and M by solving;

22. The method according to claim 21, further comprising determining the largest singular value of E, which encodes the heart rate information, by the at least one automated processor.

23. A system for determining cardiac contraction timing from video images, comprising:

an input port configured to receive a time sequence of images of a human face from an automated camera;

at least one automated processor, configured to;

process the time sequence of images of the human face to extract a plurality of facial regions;

compute heartbeat-induced time-varying features of the respective plurality of facial regions;

determine a respective statistical parameter for heartbeat-induced time-varying features of the respective plurality of facial regions;

adaptively select, based on the determined respective statistical parameter, a dynamically changing subset of the plurality of facial regions having a higher statistical reliability than a non-selected subset; and

compute a cardiac contraction timing based on at least the respective heartbeat-induced time-varying features of the respective selected subset of the plurality of facial regions; and

an output port configured to convey a signal responsive to the cardiac contraction timing.