IMAGE SEGMENTATION AND MODIFICATION OF A VIDEO STREAM

• US 20190206061A1
• Filed: 12/26/2018
• Published: 07/04/2019
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Kudriashov, Igor, Poliakov, Fedir, Gusarov, Maksim
• Granted Patent: US 10,515,454 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Systems, devices, media, and methods are presented for segmenting an image of a video stream with a client device, identifying an area of interest, generating a modified area of interest within one or more image, identifying a first set of pixels and a second set of pixels, and modifying a color value for the first set of pixels.

Claims

1. A computer implemented method for manipulating portions of a video stream, comprising:

determining, using one or more processors of a client device, an approximate location of a mouth within a video stream including a face and comprising a first set of images and a second set of images;

identifying, by the client device, an area of interest within the one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the mouth;

generating, by the client device, a modified area of interest by

removing one or more portions of the one or more images which occur outside of the area of interest to isolate the area of interest, and

generating a modified area of interest by generating a binarized area of interest by binarization of the area of interest;

identifying, by the client device, a set of teeth visible within the mouth in the binarized area of interest;

identifying, by the client device, a first set of pixels and a second set of pixels within the binarized area of interest and adding at least a portion of the first set of pixels as a set of landmark points within a binary mask of the face, the first set of pixels corresponding to the set of teeth within the mouth; and

when the set of teeth are visible within the second set of images, modifying, by the client device, a color value for the first set of pixels within the second set of images of the video stream.

2. The method of claim 1, wherein generating the modified area of interest further comprises:

converting the pixels with the area of interest to values along a grayscale to generate a grayscale area of interest, the pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel.

3. The method of claim 2, wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and generating the modified area of interest further comprises:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

4. The method of claim 3, wherein generating the modified area of interest further comprises:

thresholding the equalized area of interest to generate the binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.

5. The method of claim 4, wherein the second set of pixels comprises a plurality of segments interrupted by one or more pixels having the first value and generating the binarized area of interest further comprising:

identifying one or more pixels having the first value positioned between two or more of the plurality of segments having the second value; and

replacing the first value of the identified one or more pixels with the second value.

6. The method of claim 4, wherein identifying a first set of pixels and a second set of pixels within the modified area of interest further comprises:

determining the first set of pixels have a value greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

7. The method of claim 5, wherein marking the first set of pixels within the second set of images further comprises:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images; and

identifying the color value of the first set of pixels within the first set of images.

8. The method of claim 5, wherein modifying the color value for the first set of pixels within the second set of images further comprises:

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.

9. A system for manipulating portions of a video stream, comprising:

one or more processor; and

a non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising;

determining, using one or more processors of a client device, an approximate location of a mouth within a video stream including a face and comprising a first set of images and a second set of images;

identifying, by the client device, an area of interest within the one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the mouth;

generating, by the client device, a modified area of interest by

removing one or more portions of the one or more images which occur outside of the area of interest to isolate the area of interest, and

generating a modified area of interest by generating a binarized area of interest by binarization of the area of interest;

identifying, by the client device, a set of teeth visible within the mouth in the binarized area of interest;

identifying, by the client device, a first set of pixels and a second set of pixels within the binarized area of interest and adding at least a portion of the first set of pixels as a set of landmark points within a binary mask of the face, the first set of pixels corresponding to the set of teeth within the mouth; and

when the set of teeth are visible within the second set of images, modifying, by the client device, a color value for the first set of pixels within the second set of images of the video stream.

10. The system of claim 9, wherein generating the modified area of interest further comprises:

converting the pixels with the area of interest to values along a grayscale to generate a grayscale area of interest, the pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel.

11. The system of claim 10, wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and generating the modified area of interest further comprises:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

17. The system of claim 11, wherein generating the modified area of interest further comprises:

thresholding the equalized area of interest to generate the binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.

18. The non-transitory machine-readable storage medium of claim 17, wherein generating the modified area of interest further comprises:

converting the pixels with the area of interest to values along a grayscale to generate a grayscale area of interest, the pixels converted to grayscale by, for each pixel, generating a grayscale value by multiplying a fixed intensity value with a quotient of a triplet of color saturation values of the pixel.

19. The non-transitory machine-readable storage medium of claim 18, wherein the pixels within the grayscale area of interest comprise a histogram of values within the grayscale and generating the modified area of interest further comprises:

equalizing the histogram values within the grayscale area of interest to generate an equalized area of interest.

20. The non-transitory machine readable storage medium of claim 19, wherein generating the modified area of interest further comprises:

thresholding the equalized area of interest to generate the binarized area of interest, the first set of pixels within the binarized area of interest having a first value and the second set of pixels within the binarized area of interest having a second value distinct from the first value.

13. The system of claim 12, wherein the second set of pixels comprises a plurality of segments interrupted by one or more pixels having the first value and generating the binarized area of interest further comprising:

identifying one or more pixels having the first value positioned between two or more of the plurality of segments having the second value; and

replacing the first value of the identified one or more pixels with the second value.

14. The system of claim 12, wherein identifying a first set of pixels and a second set of pixels within the modified area of interest further comprises:

determining the first set of pixels have a value greater than a predetermined threshold; and

marking the first set of pixels within the second set of images of the video stream to generate a set of marked pixels.

15. The system of claim 14, wherein marking the first set of pixels within the second set of images further comprises:

identifying locations for the first set of pixels with respect to one or more landmarks of a face depicted within the first set of images; and

identifying the color value of the first set of pixels within the first set of images.

16. The system of claim 14, wherein modifying the color value for the first set of pixels within the second set of images further comprises:

tracking the set of marked pixels across the second set of images of the video stream; and

modifying the color value for the first set of pixels when the color value is presented proximate to the marked pixels.

17-1. A non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

determining, using one or more processors of a client device, an approximate location of a mouth within a video stream including a face and comprising a first set of images and a second set of images;

identifying, by the client device, an area of interest within the one or more images of the first set of images, the area of interest being a portion of the one or more images encompassing the approximate location of the mouth;

generating, by the client device, a modified area of interest by

removing one or more portions of the one or more images which occur outside of the area of interest to isolate the area of interest, and

generating a modified area of interest by generating a binarized area of interest by binarization of the area of interest;

identifying, by the client device, a set of teeth visible within the mouth in the binarized area of interest;

identifying, by the client device, a first set of pixels and a second set of pixels within the binarized area of interest and adding at least a portion of the first set of pixels as a set of landmark points within a binary mask of the face, the first set of pixels corresponding to the set of teeth within the mouth; and

when the set of teeth are visible within the second set of images, modifying, by the client device, a color value for the first set of pixels within the second set of images of the video stream.