Camera based photoplethysmogram estimation

• US 9,936,880 B2
• Filed: 11/25/2015
• Issued: 04/10/2018
• Est. Priority Date: 11/25/2014
• Status: Active Grant

• Current Assignee: Rice University
• Original Assignee: Rice University
• Inventors: Kumar, Mayank, Veeraraghavan, Ashok, Sabharwal, Ashutosh
• Primary Examiner(s): Gedeon, Brian T
• Application Number:
  US14/952,346
• Publication Number:
  US 20160143538A1
• Time in Patent Office: 867 Days
• Field of Search: None
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/02427 Details of sensor
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/7221 Determining signal validity, reliability or quality A61B5/7207 and A61B5/7217 take precedence
• CPC Class Codes: A61B 5/7235 Details of waveform analysis A61B5/349 takes precedence
• CPC Class Codes: A61B 5/7253 characterised by using transforms
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A system for estimating a photoplethysmogram waveform of a target includes an image processor configured to obtain images of the target and a waveform analyzer. The waveform analyzer is configured to determine a weight of a portion of the target. The weight is based on a time variation of a light reflectivity of the portion of the target. The time variation of the light reflectivity of the target is based on the images. The waveform analyzer is further configured to estimate a PPG waveform of the target based on the weight of the portion and the time variation of the light reflectivity of the portion.

Claims

1. A method of estimating a photoplethysmogram (PPG) waveform of a target, comprising:

obtaining a plurality of images, wherein an image of the plurality of images comprises the target;

subdividing the target into a plurality of portions;

determining, based on the plurality of images, a time variation of a light reflectivity of a portion of the plurality of portions;

determining, based on the time variation of the light reflectivity, a weight associated with the portion of the plurality of portions; and

estimating the PPG waveform of the target based on the weight and the image of the plurality of images,

wherein estimating the PPG waveform of the target comprises multiplying the weight of the portion of the plurality of portions by the light reflectivity of the portion of the plurality of portions to form a weighted light reflectivity of the portion.

2. The method of claim 1, wherein obtaining the plurality of images comprises:

activating an illumination source; and

capturing a first image of the target while the illumination source is activated.

3. The method of claim 2, wherein obtaining the plurality of images further comprises:

capturing a second image of the target while the illumination source is activated,

wherein the second image is captured at a time that is different than a time of capture of the first image.

4. The method of claim 1, wherein determining the time variation of the light reflectivity of the portion of the plurality of portions comprises:

identifying the target in each image of the plurality of images;

generating an alignment transformation for each image of the plurality of images based on the location of the target in the image of the plurality of images;

applying the alignment transform to each image of the plurality of images to associate at least one pixel of each image of the plurality of images with the target; and

determining the time variation of the light reflectivity of the portion of the plurality of portions based on the at least one pixel of each image of the plurality of each image and a capture time of each image of the plurality of images.

5. The method of claim 1, wherein determining the weight associated with the portion of the plurality of portions comprises:

generating an estimate of a frequency of the PPG waveform based on the light reflectivity of the portion of plurality of portions;

determining a first spectral power of the time variation of the light reflectivity of the portion of the plurality of portions in a spectral region including the estimate of the frequency of the PPG waveform;

determining a second spectral power of the time variation of the light reflectivity of the portion of the plurality of portions in a spectral region that does not include the estimate of the frequency of the PPG waveform; and

determining the weight based on the first spectral power and the second spectral power.

6. The method of claim 1, further comprising:

generating an estimate of a frequency of the PPG waveform based on the filtered light reflectivity of the portion of plurality of portions;

applying a bandpass filter to the filtered time variation of the light reflectivity of the portion of the plurality of portions to generate a filtered time variation of the light reflectivity, wherein a passband of the bandpass filter comprises the estimate of the frequency of the PPG waveform;

determining a first spectral power of the filtered time variation of the light reflectivity of the portion of the plurality of portions in a spectral region including the estimate of the frequency of the PPG waveform;

determining a second spectral power of the filtered time variation of the light reflectivity of the portion of the plurality of portions in a spectral region that does not include the estimate of the frequency of the PPG waveform; and

determining the weight based on the first spectral power and the second spectral power.

7. The method of claim 6, wherein generating an estimate of a frequency of the PPG waveform comprises:

generating a spectral power density of the filtered time variation of the light reflectivity of the portion of the plurality of portions;

determining a frequency associated with a maximum of the spectral power density; and

generating the estimate based on the frequency associated with the maximum of the spectral power density.

8. The method of claim 1, wherein estimating the PPG waveform of the target further comprises:

multiplying a second weight of a second portion of the plurality of portions by a light reflectivity of the second portion of the plurality of portions to form a weighted light reflectivity of the second portion; and

summing the weighted light reflectivity of the portion with the weighted light reflectivity of the second portion.

9. A non-transitory computer readable medium (CRM) storing instructions for estimating a photoplethysmogram (PPG) waveform of a target, the instructions comprising functionality for:

obtaining a plurality of images, wherein an image of the plurality of images comprises the target;

subdividing the target into a plurality of portions;

determining, based on the plurality of images, a time variation of a light reflectivity of a portion of the plurality of portions;

determining, based on the time variation of the light reflectivity, a weight associated with the portion of the plurality of portions; and

estimating the PPG waveform of the target based on the weight and the image of the plurality of images,

wherein the instructions for determining the time variation of the light reflectivity of the portion of the plurality of portions comprise functionality for;

identifying the target in each image of the plurality of images;

generating an alignment transformation for each image of the plurality of images based on the location of the target in the image of the plurality of images;

applying the alignment transform to each image of the plurality of images to associate at least one pixel of each image of the plurality of images with the target; and

determining the time variation of the light reflectivity of the portion of the plurality of portions based on the at least one pixel of each image of the plurality of each image and a capture time of each image of the plurality of images.

10. The non-transitory CRM of claim 9, wherein the instructions for determining the weight associated with the portion of the plurality of portions comprise functionality for:

generating an estimate of a frequency of the PPG waveform based on the light reflectivity of the portion of plurality of portions;

determining a first spectral power of the time variation of the light reflectivity of the portion of the plurality of portions in a spectral region including the estimate of the frequency of the PPG waveform;

determining a second spectral power of the time variation of the light reflectivity of the portion of the plurality of portions in a spectral region that does not include the estimate of the frequency of the PPG waveform; and

determining the weight based on the first spectral power and the second spectral power.

11. A method of estimating a photoplethysmogram (PPG) waveform of a target, comprising:

obtaining a plurality of images, wherein an image of the plurality of images comprises the target;

subdividing the target into a plurality of portions;

determining, based on the plurality of images, a time variation of a light reflectivity of a portion of the plurality of portions;

determining, based on the time variation of the light reflectivity, a weight associated with the portion of the plurality of portions;

estimating the PPG waveform of the target based on the weight and the image of the plurality of images;

generating an estimate of a frequency of the PPG waveform based on the filtered light reflectivity of the portion of plurality of portions;

applying a bandpass filter to the filtered time variation of the light reflectivity of the portion of the plurality of portions to generate a filtered time variation of the light reflectivity, wherein a passband of the bandpass filter comprises the estimate of the frequency of the PPG waveform;

determining a first spectral power of the filtered time variation of the light reflectivity of the portion of the plurality of portions in a spectral region including the estimate of the frequency of the PPG waveform;

determining a second spectral power of the filtered time variation of the light reflectivity of the portion of the plurality of portions in a spectral region that does not include the estimate of the frequency of the PPG waveform; and

determining the weight based on the first spectral power and the second spectral power.

12. The method of claim 11, wherein obtaining the plurality of images comprises:

activating an illumination source; and

capturing a first image of the target while the illumination source is activated.

13. The method of claim 12, wherein obtaining the plurality of images further comprises:

capturing a second image of the target while the illumination source is activated,

wherein the second image is captured at a time that is different than a time of capture of the first image.

14. The method of claim 11, wherein determining the time variation of the light reflectivity of the portion of the plurality of portions comprises:

identifying the target in each image of the plurality of images;

generating an alignment transformation for each image of the plurality of images based on the location of the target in the image of the plurality of images;

applying the alignment transform to each image of the plurality of images to associate at least one pixel of each image of the plurality of images with the target; and

determining the time variation of the light reflectivity of the portion of the plurality of portions based on the at least one pixel of each image of the plurality of each image and a capture time of each image of the plurality of images.

15. The method of claim 11, wherein determining the weight associated with the portion of the plurality of portions comprises:

generating an estimate of a frequency of the PPG waveform based on the light reflectivity of the portion of plurality of portions;

determining a first spectral power of the time variation of the light reflectivity of the portion of the plurality of portions in a spectral region including the estimate of the frequency of the PPG waveform;

determining a second spectral power of the time variation of the light reflectivity of the portion of the plurality of portions in a spectral region that does not include the estimate of the frequency of the PPG waveform; and

determining the weight based on the first spectral power and the second spectral power.

16. The method of claim 11, wherein generating an estimate of a frequency of the PPG waveform comprises:

generating a spectral power density of the filtered time variation of the light reflectivity of the portion of the plurality of portions;

determining a frequency associated with a maximum of the spectral power density; and

generating the estimate based on the frequency associated with the maximum of the spectral power density.

17. The method of claim 11, wherein estimating the PPG waveform of the target comprises:

multiplying the weight the portion of the plurality of portions by the light reflectivity of the portion of the plurality of portions to form a weighted light reflectivity of the first portion.

18. The method of claim 11, wherein estimating the PPG waveform of the target further comprises:

multiplying a second weight of a second portion of the plurality of portions by a light reflectivity of the second portion of the plurality of portions to form a weighted light reflectivity of the second portion; and

summing the weighted light reflectivity of the first portion with the weighted light reflectivity of the second portion.