IMAGE SEGMENTATION FOR OBJECT MODELING

• US 20190012563A1
• Filed: 09/13/2018
• Published: 01/10/2019
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Gusarov, Maksim Igorevich
• Granted Patent: US 10,268,921 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/20032 Median filtering
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/181 involving edge growing; involving edge linking
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/68 of symmetry
• CPC Class Codes: G06V 10/34 Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Systems, devices, and methods are presented for segmenting an image of a video stream with a client device by accessing a set of images within a video stream, identifying an object of interest within one or more images of the set of images, and detecting a region of interest within the one or more images. The systems, devices, and method identify a first set of median pixels in a first portion of the object of interest and a second set of median pixels in a second portion of the object of interest. The systems, devices, and methods determine a polyline approximating the first and second sets of median pixels and generate a model for the polyline.

Claims

1. A computer implemented method, comprising:

identifying, by a processor, an object of interest in an image;

determining, by the processor, a type associated with the object of interest;

selecting, based on the type associated with the object of interest, a shape from a plurality of shapes for a region of interest to encompass the object of interest;

identifying, by the processor, median pixels within the shape that encompasses the object of interest;

determining, by the processor, a polyline approximating a portion of the object of interest based on the median pixels; and

generating, by the processor, a model for the polyline.

2. The method of claim 1 further comprising:

generating one or more binarization matrices from the image;

determining a pixel radius; and

filtering the one or more binarization matrices to exchange a first pixel value for a second pixel value where a pixel associated with the first pixel value is within a prespecified distance of a set of pixels of the second pixel value, based on the pixel radius.

3. The method of claim 2, wherein the object of interest includes a first portion and a second portion, further comprising identifying a first thickness and a first set of median pixels of the first portion and a second thickness and a second set of median pixels of the second portion.

4. The method of claim 3, wherein determining the polyline comprises:

determining a bend position between the first portion and the second portion of the object of interest;

determining a first line extending in a first direction from the bend position, the first line extending across one or more of the first set of median pixels; and

determining a second line extending in a second direction from the bend position, the second line extending across one or more of the second set of median pixels.

5. The method of claim 4, wherein generating the model for the polyline further comprises:

determining a length for the first line and the second line;

determining a bend angle for the bend position; and

determining a slope angle for the first portion of the object of interest.

6. The method of claim 1, wherein generating the model for the polyline comprises generating an averaged model based on a model generated for each image of a set of images of the video stream, the set of images including the image.

7. The method of claim 1, wherein determining the type comprises determining that the object of interest is an eyebrow, and wherein the region of interest includes the eyebrow and a portion of the face surrounding the eyebrow.

8. The method of claim 1, wherein determining the type comprises determining that the object of interest matches shapes, edges, or landmarks within a range of the type.

9. A system, comprising:

one or more processors; and

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

identifying an object of interest in an image;

determining a type associated with the object of interest;

selecting, based on the type associated with the object of interest, a shape from a plurality of shapes for a region of interest to encompass the object of interest;

identifying median pixels within the shape that encompasses the object of interest;

determining a polyline approximating a portion of the object of interest based on the median pixels; and

generating a model for the polyline.

10. The system of claim 9, wherein the operations further comprise:

generating one or more binarization matrices from the image;

determining a pixel radius; and

filtering the one or more binarization matrices to exchange a first pixel value for a second pixel value where a pixel associated with the first pixel value is within a prespecified distance of a set of pixels of the second pixel value, based on the pixel radius.

11. The system of claim 10, wherein the object of interest includes a first portion and a second portion, further comprising identifying a first thickness and a first set of median pixels of the first portion and a second thickness and a second set of median pixels of the second portion.

12. The system of claim 11, wherein determining the polyline comprises:

determining a bend position between the first portion and the second portion of the object of interest;

determining a first line extending in a first direction from the bend position, the first line extending across one or more of the first set of median pixels; and

determining a second line extending in a second direction from the bend position, the second line extending across one or more of the second set of median pixels.

13. The system of claim 12, wherein generating the model for the polyline further comprises:

determining a length for the first line and the second line;

determining a bend angle for the bend position; and

determining a slope angle for the first portion of the object of interest.

14. The system of claim 9, wherein generating the model for the polyline comprises generating an averaged model based on a model generated for each image of a set of images of the video stream, the set of images including the image.

15. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

identifying an object of interest in an image;

determining a type associated with the object of interest;

selecting, based on the type associated with the object of interest, a shape from a plurality of shapes for a region of interest to encompass the object of interest;

identifying median pixels within the shape that encompasses the object of interest;

determining a polyline approximating a portion of the object of interest based on the median pixels; and

generating a model for the polyline.

16. The non-transitory processor-readable storage medium of claim 15, wherein the operations further comprise:

generating one or more binarization matrices from the image;

determining a pixel radius; and

filtering the one or more binarization matrices to exchange a first pixel value for a second pixel value where a pixel associated with the first pixel value is within a prespecified distance of a set of pixels of the second pixel value, based on the pixel radius.

17. The non-transitory processor-readable storage medium of claim 16, wherein the object of interest includes a first portion and a second portion, further comprising identifying a first thickness and a first set of median pixels of the first portion and a second thickness and a second set of median pixels of the second portion.

18. The non-transitory processor-readable storage medium of claim 17, wherein determining the polyline comprises:

determining a bend position between the first portion and the second portion of the object of interest;

determining a first line extending in a first direction from the bend position, the first line extending across one or more of the first set of median pixels; and

determining a second line extending in a second direction from the bend position, the second line extending across one or more of the second set of median pixels.

19. The non-transitory processor-readable storage medium of claim 18, wherein generating the model for the polyline further comprises:

determining a length for the first line and the second line;

determining a bend angle for the bend position; and

determining a slope angle for the first portion of the object of interest.

20. The non-transitory processor-readable storage medium of claim 15, wherein generating the model for the polyline comprises generating an averaged model based on a model generated for each image of a set of images of the video stream, the set of images including the image.