Facial skin mask generation for heart rate detection

• US 9,582,879 B2
• Filed: 10/20/2014
• Issued: 02/28/2017
• Est. Priority Date: 10/20/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cheng, Beibei, Walker, Benjamin William, Hoof, Jonathan Ross, Kennett, Daniel, Ahmad, Anis
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US14/518,596
• Publication Number:
  US 20160110868A1
• Time in Patent Office: 862 Days
• Field of Search: 382/128
• US Class Current: 1/1
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30048 Heart; Cardiac
• CPC Class Codes: G06T 2207/30088 Skin; Dermal
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A facial skin mask may be generated based on isolating a head part in a captured image, removing a first pixel that is indicative of non-skin from the head part in the captured image, and removing a second pixel that is indicative of having a high velocity from the head part in the captured image. Heart rate may be detected based on the change of color of the pixels of the generated facial skin mask.

Claims

1. A device comprising:

a processor; and

a memory coupled with the processor, the memory having stored thereon executable instructions that when executed by the processor cause the processor to effectuate operations comprising;

receiving data of a captured image, the data of the captured image comprising a subset of data of an anatomical head, the subset of data of the anatomical head comprising a plurality of representative pixels having a respective plurality of pixel values;

determining whether each pixel of the plurality of representative pixels has the respective plurality of pixel values within a threshold value that is indicative of skin;

based on determining that the plurality of representative pixels has the respective plurality of pixel values within the threshold value that is indicative of skin, determining, based on information from multiple captured images comprising the captured image, velocity of each pixel of the plurality of representative pixels that has a plurality of respective pixel velocity values within a threshold velocity value; and

creating a facial skin mask based on each pixel of the plurality of representative pixels having;

the plurality of respective pixel velocity values within the threshold velocity value, and

the respective plurality of pixel values within the threshold value that is indicative of skin.

2. The device of claim 1, the operations further comprising:

determining whether the plurality of representative pixels average a value that is within a facial skin mask threshold.

3. The device of claim 2, the operations further comprising:

responsive to determining that the plurality of representative pixels average a value within the facial skin mask threshold, creating the facial skin mask with each pixel of the plurality of representative pixels within the threshold value that is indicative of skin.

4. The device of claim 1, wherein the plurality of respective pixel velocity values of each pixel is determined using an optical flow technique.

5. The device of claim 1, the operations further comprising:

providing instructions to display a graphic overlay based on the facial skin mask.

6. The device of claim 5, wherein the graphic overlay comprises a depiction of a scar.

7. The device of claim 1, the operations further comprising:

providing instructions to determine a heart rate based on the facial skin mask.

8. The device of claim 1, wherein the respective plurality of pixel values are based on a color channel.

9. The device of claim 1, wherein the captured image is captured by a non-visible light camera based on a detection of lighting.

10. A computer readable storage medium comprising computer executable instructions that when executed by a computing device cause said computing device to perform the computer executable instructions comprising:

receiving data of a captured image, the data of the captured image comprising a subset of data of an anatomical head, the subset of data of the anatomical head comprising a plurality of representative pixels having a respective plurality of pixel values;

determining whether each pixel of the plurality of representative pixels has the respective plurality of pixel values within a threshold value that is indicative of skin;

based on determining that the plurality of representative pixels has the respective plurality of pixel values within the threshold value that is indicative of skin, determining velocity of each pixel of the plurality of representative pixels that has a plurality of respective pixel velocity values within a threshold velocity value; and

creating a facial skin mask based on each pixel of the plurality of representative pixels having;

the plurality of respective pixel velocity values within the threshold velocity value, and

the respective plurality of pixel values within the threshold value that is indicative of skin.

11. The computer readable storage medium of claim 10, further instructions comprising:

determining whether the plurality of representative pixels average a value that is within a facial skin mask threshold.

12. The computer readable storage medium of claim 10, further operations comprising:

providing instructions to display a graphic overlay based on the facial skin mask.

13. The computer readable storage medium of claim 12, wherein the graphic overlay comprises a depiction of a scar.

14. The computer readable storage medium of claim 10, wherein the captured image is captured by a non-visible light camera based on a detection of lighting.

15. A method comprising:

receiving a first data of a first captured image based on detected lighting, the first data of the first captured image comprising a first subset of data of an anatomical head;

converting a first plurality of representative pixels of the first subset of data of the anatomical head to a red, green, and blue (RGB) channel, the first plurality of representative pixels having a respective first plurality of pixel values based on the RGB channel;

determining whether each pixel of the first plurality of representative pixels has a respective plurality of pixel values within a first threshold value that is indicative of skin;

responsive to determining that the first plurality of representative pixels average a first value that is not within a first skin mask threshold, converting a second plurality of representative pixels of a second subset of data of an anatomical head of a second captured image to a red, green, and blue (RGB) channel, each pixel of the second plurality of representative pixels having a respective second plurality of pixel values based on the RGB channel;

comparing the first plurality of representative pixels that average the first value with the second plurality of representative pixels that average a second value; and

creating a facial skin mask with each pixel of the second plurality of representative pixels within the threshold value that is indicative of skin based on the comparing of the first plurality of representative pixels that average the first value with the second plurality of representative pixels that average the second value.

16. The method of claim 15, further comprising:

displaying a graphic overlay based on the facial skin mask.

17. The method of claim 15, further comprising:

responsive to determining that the second plurality of representative pixels of the second subset of data of the anatomical head of the second captured image average the second value within the first skin mask threshold, determining velocity of each pixel of the second plurality of representative pixels that has a plurality of respective pixel velocity values within a velocity threshold; and

creating the facial skin mask based on each pixel of the second plurality of representative pixels having;

the plurality of respective pixel velocity values within the velocity threshold, and

the second respective plurality of pixel values within the threshold value that is indicative of skin.

18. The method of claim 17, wherein the pixel velocity value of each pixel of the second plurality of representative pixels is determined using an optical flow technique.