Image segmentation and modification of a video stream

• US 10,515,454 B2
• Filed: 12/26/2018
• Issued: 12/24/2019
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Kudriashov, Igor, Poliakov, Fedir, Gusarov, Maksim
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US16/232,737
• Publication Number:
  US 20190206061A1
• Time in Patent Office: 363 Days
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Systems, devices, media, and methods are presented for segmenting an image of a video stream with a client device, identifying an area of interest, generating a modified area of interest within one or more image, identifying a first set of pixels and a second set of pixels, and modifying a color value for the first set of pixels.

Claims

1. A computer implemented method for manipulating portions of a video stream, comprising:

determining, by a client device, an approximate location of a mouth within a video stream including a face and comprising a first set of images and a second set of images;

identifying, by the client device, an area of interest comprising a plurality of pixels within the one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the mouth;

generating, by the client device a binarization matrix by performing operations comprising;

for each pixel of the plurality of pixels within the area of interest;

retrieving a set of color values associated with the pixel;

determining a binary value for the pixel by comparing a first value of a first portion of the set of color values with a second value of a second portion of the set of color values;

storing the binary value for the pixel in the binarization matrix;

modifying each pixel in the plurality of pixels within the area of interest using the binarization matrix to create a binarized area of interest;

identifying, by the client device, a set of teeth visible within the mouth in the binarized area of interest;

identifying, by the client device, a first set of pixels and a second set of pixels within the binarized area of interest and adding at least a portion of the first set of pixels as a set of landmark points within a binary mask of the face, the first set of pixels corresponding to the set of teeth within the mouth; and

when the set of teeth are visible within the second set of images, modifying, by the client device, a color value for the first set of pixels within the second set of images of the video stream.

2. The method of claim 1, further comprising:

converting the plurality of pixels within the area of interest to values along a grayscale to generate a grayscale area of interest, the plurality of pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel.

3. The method of claim 2, wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and further comprising:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

4. The method of claim 3, further comprising:

thresholding the equalized area of interest to generate the binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.

5. The method of claim 4, wherein the second set of pixels comprises a plurality of segments interrupted by one or more pixels having the first value and generating the binarized area of interest further comprising:

identifying one or more pixels having the first value positioned between two or more of the plurality of segments having the second value; and

replacing the first value of the identified one or more pixels with the second value.

6. The method of claim 4, further comprising:

determining the first set of pixels have a value greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

7. The method of claim 5, wherein marking the first set of pixels within the second set of images further comprises:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images; and

identifying the color value of the first set of pixels within the first set of images.

8. The method of claim 5, wherein modifying the color value for the first set of pixels within the second set of images further comprises:

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.

9. A system for manipulating portions of a video stream, comprising:

one or more processor; and

a non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising;

determining, by a client device, an approximate location of a mouth within a video stream including a face and comprising a first set of images and a second set of images;

identifying, by the client device, an area of interest comprising a plurality of pixels within the one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the mouth;

generating, by the client device a binarization matrix by performing operations comprising;

for each pixel of the plurality of pixels within the area of interest;

retrieving a set of color values associated with the pixel;

determining a binary value for the pixel by comparing a first value of a first portion of the set of color values with a second value of a second portion of the set of color values;

storing the associated binary value for the pixel in the binarization matrix;

modifying each pixel in the plurality of pixels within the area of interest using the binarization matrix to create a binarized area of interest;

identifying, by the client device, a set of teeth visible within the mouth in the binarized area of interest;

identifying, by the client device, a first set of pixels and a second set of pixels within the binarized area of interest and adding at least a portion of the first set of pixels as a set of landmark points within a binary mask of the face, the first set of pixels corresponding to the set of teeth within the mouth; and

when the set of teeth are visible within the second set of images, modifying, by the client device, a color value for the first set of pixels within the second set of images of the video stream.

10. The system of claim 9, further comprising:

converting the plurality of pixels within the area of interest to values along a grayscale to generate a grayscale area of interest, the plurality of pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel.

11. The system of claim 10, wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and further comprising:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

12. The system of claim 11, further comprising:

thresholding the equalized area of interest to generate the binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.

13. The system of claim 12, wherein the second set of pixels comprises a plurality of segments interrupted by one or more pixels having the first value and generating the binarized area of interest further comprising:

identifying one or more pixels having the first value positioned between two or more of the plurality of segments having the second value; and

replacing the first value of the identified one or more pixels with the second value.

14. The system of claim 12, further comprising:

determining the first set of pixels have a value greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

15. The system of claim 14, wherein marking the first set of pixels within the second set of images further comprises:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images; and

identifying the color value of the first set of pixels within the first set of images.

16. The system of claim 14, wherein modifying the color value for the first set of pixels within the second set of images further comprises:

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.

17. A non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

determining, by a client device, an approximate location of a mouth within a video stream including a face and comprising a first set of images and a second set of images;

identifying, by the client device, an area of interest comprising a plurality of pixels within the one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the mouth;

generating, by the client device a binarization matrix by performing operations comprising;

for each pixel of the plurality of pixels within the area of interest;

retrieving a set of color values associated with the pixel;

determining a binary value for the pixel by comparing a first value of a first portion of the set of color values with a second value of a second portion of the set of color values;

storing the associated binary value for the pixel in the binarization matrix;

modifying each pixel in the plurality of pixels within the area of interest using the binarization matrix to create a binarized area of interest;

identifying, by the client device, a set of teeth visible within the mouth in the binarized area of interest;

identifying, by the client device, a first set of pixels and a second set of pixels within the binarized area of interest and adding at least a portion of the first set of pixels as a set of landmark points within a binary mask of the face, the first set of pixels corresponding to the set of teeth within the mouth; and

when the set of teeth are visible within the second set of images, modifying, by the client device, a color value for the first set of pixels within the second set of images of the video stream.

18. The non-transitory machine-readable storage medium of claim 17, further comprising:

converting the plurality of pixels within the area of interest to values along a grayscale to generate a grayscale area of interest, the plurality of pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel.

19. The non-transitory machine-readable storage medium of claim 18, wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and further comprising:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

20. The non-transitory machine readable storage medium of claim 19, further comprising:

thresholding the equalized area of interest to generate the binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.