Image segmentation and modification of a video stream

• US 10,198,819 B2
• Filed: 11/30/2015
• Issued: 02/05/2019
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Kudriashov, Igor, Poliakov, Fedir, Gusarov, Maksim
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US14/953,726
• Publication Number:
  US 20170262994A1
• Time in Patent Office: 1,163 Days
• Field of Search: None
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Systems, devices, media, and methods are presented for segmenting an image of a video stream with a client device, identifying an area of interest, generating a modified area of interest within one or more image, identifying a first set of pixels and a second set of pixels, and modifying a color value for the first set of pixels.

Claims

1. A computer implemented method for manipulating portions of a video stream, comprising:

determining, using one or more processors of a client device, an approximate location of a mouth within a video stream including a face and comprising a first set of images and a second set of images;

identifying, by the client device, an area of interest within the one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the mouth;

generating, by the client device, a modified area of interest by converting the pixels within the area of interest to values along a grayscale to generate a grayscale area of interest, the pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel;

identifying a set of teeth visible within the mouth in the grayscale area of interest;

identifying, by the client device, a first set of pixels and a second set of pixels within the grayscale area of interest and adding at least a portion of the first set of pixels as a set of landmark points within a binary mask of the face, the first set of pixels corresponding to the set of teeth within the mouth; and

when the teeth are visible within the second set of images, modifying, by the client device, a color value for the first set of pixels within the second set of images of the video stream.

2. The method of claim 1 further comprising:

cropping one or more portions of the one or more images of the first set of images outside of the area of interest.

3. The method of claim 1, wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and generating the modified area of interest further comprises:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

4. The method of claim 3, wherein generating the modified area of interest further comprises:

thresholding the equalized area of interest to generate a binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.

5. The method of claim 4, wherein the second set of pixels comprises a plurality of segments interrupted by one or more pixels having the first value and generating the binarized area of interest further comprising:

identifying one or more pixels having the first value positioned between two or more of the plurality of segments having the second value; and

replacing the first value of the identified one or more pixels with the second value.

6. The method of claim 4, wherein identifying a first set of pixels and a second set of pixels within the modified area of interest further comprises:

determining the first set of pixels have a value greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

7. The method of claim 6, wherein marking the first set of pixels within the second set of images further comprises:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images; and

identifying the color value of the first set of pixels within the first set of images.

8. The method of claim 6, wherein modifying the color value for the first set of pixels within the second set of images further comprises:

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.

9. A system for manipulating portions of a video stream, comprising:

one or more processor; and

a non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising;

determining an approximate location of a mouth within a video stream including a face and comprising a first set of images and a second set of images;

identifying an area of interest within the one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the mouth;

generating, by the client device, a modified area of interest by converting the pixels within the area of interest to values along a grayscale to generate a grayscale area of interest, the pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel;

identifying a set of teeth visible within the mouth in the grayscale area of interest;

identifying a first set of pixels and a second set of pixels within the grayscale area of interest and adding at least a portion of the first set of pixels as a set of landmark points within a binary mask of the face, the first set of pixels corresponding to the set of teeth within the mouth; and

when the teeth are visible within the second set of images, modifying a color value for the first set of pixels within the second set of images of the video stream.

10. The system of claim 9, wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and generating the modified area of interest causes the machine to perform operations comprising:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

11. The system of claim 10, wherein generating the modified area of interest causes the machine to perform operations comprising:

thresholding the equalized area of interest to generate a binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.

12. The system of claim 11, wherein the second set of pixels comprises a plurality of segments interrupted by one or more pixels having the first value and generating the binarized area of interest causes the machine to perform operations comprising:

identifying one or more pixels having the first value positioned between two or more of the plurality of segments having the second value; and

replacing the first value of the identified one or more pixels with the second value.

13. The system of claim 11, wherein identifying a first set of pixels and a second set of pixels within the modified area of interest causes the machine to perform operations comprising:

determining the first set of pixels have a value greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

14. The system of claim 13, wherein marking the first set of pixels within the second set of images causes the machine to perform operations comprising:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images; and

identifying the color value of the first set of pixels within the first set of images.

15. The system of claim 13, wherein modifying the color value for the first set of pixels within the second set of images causes the machine to perform operations comprising:

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.

16. A non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

determining an approximate location of a mouth within a video stream including a face and comprising a first set of images and a second set of images;

identifying an area of interest within the one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the mouth;

generating, by the client device, a modified area of interest by converting the pixels within the area of interest to values along a grayscale to generate a grayscale area of interest, the pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel;

identifying a set of teeth visible within the mouth in the grayscale area of interest;

identifying a first set of pixels and a second set of pixels within the grayscale area of interest and adding at least a portion of the first set of pixels as a set of landmark points within a binary mask of the face, the first set of pixels corresponding to the set of teeth within the mouth; and

when the teeth are visible within the second set of images, modifying a color value for the first set of pixels within the second set of images of the video stream.

17. The non-transitory machine-readable storage medium of claim 16, wherein identifying the first set of pixels and the second set of pixels within the modified area of interest causes the machine to perform operations comprising:

determining the first set of pixels have a value within the modified area of interest greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

18. The non-transitory machine-readable storage medium of claim 17, wherein modifying the color value for the first set of pixels within the second set of images causes the machine to perform operations comprising:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images;

identifying the color value of the first set of pixels within the first set of images;

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.