Facial skin mask generation for heart rate detection

• US 10,529,071 B2
• Filed: 07/16/2018
• Issued: 01/07/2020
• Est. Priority Date: 10/20/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cheng, Beibei, Walker, Benjamin William, Hoof, Jonathan Ross, Kennett, Daniel, Ahmad, Anis
• Primary Examiner(s): Couso, Yon J
• Application Number:
  US16/036,792
• Publication Number:
  US 20190073767A1
• Time in Patent Office: 540 Days
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30048 Heart; Cardiac
• CPC Class Codes: G06T 2207/30088 Skin; Dermal
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A facial skin mask may be generated based on isolating a head part in a captured image, removing a first pixel that is indicative of non-skin from the head part in the captured image, and removing a second pixel that is indicative of having a high velocity from the head part in the captured image. Heart rate may be detected based on the change of color of the pixels of the generated facial skin mask.

Claims

1. A computing device, comprising:

at least one memory and at least one processor, wherein the at least one memory and the at least one processor are respectively configured to store and execute instructions for causing the computing device to perform operations, and wherein the operations include;

receiving data associated with multiple captured image frames, the data of each of the multiple captured image frames including a plurality of pixels;

determining a subset of the plurality of pixels of a first frame of the multiple captured image frames, wherein the determining of the subset of the first frame is based, at least in part, on whether each particular individual pixel of the plurality of pixels of the first frame has a color value that is indicative of skin of an anatomical head;

determining a subset of the plurality of pixels of a second frame of the multiple captured image frames, wherein the determining of the subset of the second frame is based, at least in part, on whether each particular individual pixel of the plurality of pixels of the second frame has the color value that is indicative of skin of the anatomical head;

determining respective differences between individual color values of the individual pixels of the subset of the first frame and individual color values of the individual pixels of the subset of the second frame; and

determining a heart rate based on the determining of the respective differences.

2. The computing device of claim 1, wherein the determining of the subset of the plurality of pixels of the first frame is also based on positions of each particular individual pixel of the plurality of pixels of the first frame.

3. The computing device of claim 1, wherein the determining of the subset of the plurality of pixels of the first frame is also based on depth of each particular individual pixel of the plurality of pixels of the first frame.

4. The computing device of claim 1, wherein the determining of the heart rate is also based on rates of change between the individual color values of the individual pixels of the subset of the first frame and the individual color values of the individual pixels of the subset of the second frame.

5. The computing device of claim 1, wherein the determining of the heart rate is also based on use of an optical flow technique.

6. The computing device of claim 1, wherein the data associated with multiple captured image frames includes information from a visible portion of a light spectrum.

7. A method, comprising:

receiving data associated with multiple captured image frames, the data of each of the multiple captured image frames including a plurality of pixels;

determining a subset of the plurality of pixels of a first frame of the multiple captured image frames, wherein the determining of the subset of the first frame is based, at least in part, on whether each particular individual pixel of the plurality of pixels of the first frame has a color value that is indicative of skin of an anatomical head;

determining a subset of the plurality of pixels of a second frame of the multiple captured image frames, wherein the determining of the subset of the second frame is based, at least in part, on whether each particular individual pixel of the plurality of pixels of the second frame has the color value that is indicative of skin of the anatomical head;

determining respective differences between individual color values of the individual pixels of the subset of the first frame and individual color values of the individual pixels of the subset of the second frame; and

determining a heart rate based on the determining of the respective differences.

8. The method of claim 7, wherein the determining of the subset of the plurality of pixels of the first frame is also based on positions of each particular individual pixel of the plurality of pixels of the first frame.

9. The method of claim 7, wherein the determining of the subset of the plurality of pixels of the first frame is also based on depth of each particular individual pixel of the plurality of pixels of the first frame.

10. The method of claim 7, wherein the determining of the heart rate is also based on rates of change between the individual color values of the individual pixels of the subset of the first frame and the individual color values of the individual pixels of the subset of the second frame relative to a threshold velocity value.

11. The method of claim 7, wherein the determining of the heart rate is also based on use of an optical flow technique.

12. The method of claim 7, wherein the data associated with multiple captured image frames includes information from a non-visible portion of a light spectrum.

13. The method of claim 7, further comprising:

creating a facial skin map from the subset of the plurality of pixels of a first frame, wherein the determining respective differences is also based on the facial skin map.

14. A computer-readable storage medium, having stored thereon processor-executable code, that, responsive to execution by at least one processor, cause a computing device to perform operations, the operations comprising:

receiving a first data associated with a first captured image including a first subset of data of an anatomical head;

converting a first plurality of representative pixels of the first subset of data of the anatomical head to a color channel, the first plurality of representative pixels having a respective first plurality of pixel values based on the color channel;

determining, for each pixel of the first plurality of representative pixels, whether the pixel has a respective plurality of pixel values within a first threshold value that is indicative of skin;

responsive to determining that the first plurality of representative pixels average a first value that is not within a first skin mask threshold, converting a second plurality of representative pixels of a second subset of data of an anatomical head associated with a second captured image to a color channel, each pixel of the second plurality of representative pixels having a respective second plurality of pixel values based on the color channel;

creating a facial skin mask with each pixel of the second plurality of representative pixels within a threshold value that is indicative of skin based on a comparison of the first plurality of representative pixels that average the first value with the second plurality of representative pixels that average a second value; and

determining a heart rate based on color of the pixels of the facial skin mask.

15. The computer-readable storage medium of claim 14, wherein the operations further comprise:

displaying a graphic overlay based on the facial skin mask.

16. The computer-readable storage medium of claim 14, wherein the creating of the facial skin mask is further based on each pixel of the second plurality of representative pixels having a plurality of respective pixel velocity values within a threshold velocity value.

17. The computer-readable storage medium of claim 16, wherein the plurality of respective pixel velocity values of each pixel of the second plurality of representative pixels is determined using an optical flow technique.

18. The computer-readable storage medium of claim 14, wherein determining the heart rate is further based on a change in the color of pixels of the facial skin mask over time.

19. The computer-readable storage medium of claim 14, wherein the respective first plurality of pixel values is based on a position of the first plurality of representative pixels on the anatomical head.

20. The computer-readable storage medium of claim 14, wherein at least one color channel is for a non-visible portion of a light spectrum.