Image segmentation for object modeling

• US 9,886,647 B1
• Filed: 12/30/2015
• Issued: 02/06/2018
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Gusarov, Maksim Igorevich
• Primary Examiner(s): Liew, Alex
• Application Number:
  US14/983,930
• Time in Patent Office: 769 Days
• CPC Class Codes: G06T 2207/20032 Median filtering
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/181 involving edge growing; involving edge linking
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/68 of symmetry
• CPC Class Codes: G06V 10/34 Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Systems, devices, and methods are presented for segmenting an image of a video stream with a client device by accessing a set of images within a video stream, identifying an object of interest within one or more images of the set of images, and detecting a region of interest within the one or more images. The systems, devices, and method identify a first set of median pixels in a first portion of the object of interest and a second set of median pixels in a second portion of the object of interest. The systems, devices, and methods determine a polyline approximating the first and second sets of median pixels and generate a model for the polyline.

Claims

1. A computer implemented method, comprising:

accessing a set of images within a video stream;

identifying an object of interest in one or more images of the set of images, the object of interest having a first portion and a second portion;

detecting a region of interest within the one or more images;

generating one or more binarization matrices from the one or more images;

determining a pixel radius;

filtering the one or more binarization matrices to exchange a first pixel value for a second pixel value where a pixel associated with the first pixel value is proximate to a set of pixels of the second pixel value;

identifying a first set of median pixels of the first portion and a second set of median pixels of the second portion;

determining a polyline approximating the first portion and the second portion of the object of interest; and

generating a model for the polyline.

2. The method of claim 1, wherein identifying the first set of median pixels for the first portion and the second set of median pixels for the second portion further comprises:

identifying a first set of median pixels for the first portion of the object of interest, the first set of median pixels being a median pixel in each column, of the one or more binarization matrices, having a pixel associated with the first portion of the object of interest;

identifying a second set of median pixels for the second portion of the object of interest, the second set of median pixels being a median pixel in each column, of the one or more binarization matrices, having a pixel associated with the second portion of the object of interest; and

isolating the first set of median pixels and the second set of median pixels within the one or more binarization matrices from the remaining pixels.

3. The method of claim 2, wherein determining the polyline further comprises:

determining a bend position between the first portion and the second portion of the object of interest;

determining a first line extending in a first direction from the bend position, the first line extending across one or more of the first set of median pixels; and

determining a second line extending in a second direction from the bend position, the second line extending across one or more of the second set of median pixels.

4. The method of claim 3, wherein generating the model for the polyline further comprises:

determining a length for the first line and the second line;

determining a bend angle for the bend position; and

determining a slope angle for the first portion of the object of interest.

5. The method of claim 1 further comprising:

determining a first thickness of the first portion of the object of interest; and

determining a second thickness of the second portion of the object of interest.

6. The method of claim 1, wherein generating the model for the polyline further comprises:

generating an averaged model based on a model generated for each image of the set of images of the video stream.

7. A system, comprising:

one or more processors; and

a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

accessing a set of images within a video stream;

identifying an object of interest in one or more images of the set of images, the object of interest having a first portion and a second portion;

detecting a region of interest within the one or more images;

generating one or more binarization matrices from the one or more images;

determining a pixel radius;

filtering the one or more binarization matrices to exchange a first pixel value for a second pixel value where a pixel associated with the first pixel value is proximate to a set of pixels of the second pixel value;

identifying a first set of median pixels of the first portion and a second set of median pixels of the second portion;

determining a polyline approximating the first portion and the second portion of the object of interest; and

generating a model for the polyline.

8. The system of claim 7, wherein identifying the first set of median pixels for the first portion and the second set of median pixels for the second portion further comprises:

identifying a first set of median pixels for the first portion of the object of interest, the first set of median pixels being a median pixel in each column, of the one or more binarization matrices, having a pixel associated with the first portion of the object of interest;

identifying a second set of median pixels for the second portion of the object of interest, the second set of median pixels being a median pixel in each column, of the one or more binarization matrices, having a pixel associated with the second portion of the object of interest; and

isolating the first set of median pixels and the second set of median pixels within the one or more binarization matrices from the remaining pixels.

9. The system of claim 8, wherein determining the polyline further comprises:

determining a bend position between the first portion and the second portion of the object of interest;

determining a first line extending in a first direction from the bend position, the first line extending across one or more of the first set of median pixels; and

determining a second line extending in a second direction from the bend position, the second line extending across one or more of the second set of median pixels.

10. The system of claim 9, wherein generating the model for the polyline further comprises:

determining a length for the first line and the second line;

determining a bend angle for the bend position; and

determining a slope angle for the first portion of the object of interest.

11. The system of claim 7, wherein the operations further comprise:

determining a first thickness of the first portion of the object of interest; and

determining a second thickness of the second portion of the object of interest.

12. The system of claim 7, wherein generating the model for the polyline further comprises:

generating an averaged model based on a model generated for each image of the set of images of the video stream.

13. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

accessing a set of images within a video stream;

identifying an object of interest in one or more images of the set of images, the object of interest having a first portion and a second portion;

detecting a region of interest within the one or more images;

generating one or more binarization matrices from the one or more images;

determining a pixel radius;

filtering the one or more binarization matrices to exchange a first pixel value for a second pixel value where a pixel associated with the first pixel value is proximate to a set of pixels of the second pixel value;

identifying a first set of median pixels of the first portion and a second set of median pixels of the second portion;

determining a polyline approximating the first portion and the second portion of the object of interest; and

generating a model for the polyline.

14. The non-transitory processor-readable storage medium of claim 13, wherein identifying the first set of median pixels for the first portion and the second set of median pixels for the second portion further comprises:

identifying a first set of median pixels for the first portion of the object of interest, the first set of median pixels being a median pixel in each column, of the one or more binarization matrices, having a pixel associated with the first portion of the object of interest;

identifying a second set of median pixels for the second portion of the object of interest, the second set of median pixels being a median pixel in each column, of the one or more binarization matrices, having a pixel associated with the second portion of the object of interest; and

isolating the first set of median pixels and the second set of median pixels within the one or more binarization matrices from the remaining pixels.

15. The non-transitory processor-readable storage medium of claim 14, wherein determining the polyline further comprises:

determining a bend position between the first portion and the second portion of the object of interest;

determining a first line extending in a first direction from the bend position, the first line extending across one or more of the first set of median pixels; and

determining a second line extending in a second direction from the bend position, the second line extending across one or more of the second set of median pixels.

16. The non-transitory processor-readable storage medium of claim 15, wherein generating the model for the polyline further comprises:

determining a length for the first line and the second line;

determining a bend angle for the bend position; and

determining a slope angle for the first portion of the object of interest.

17. The non-transitory processor-readable storage medium of claim 13, wherein the operations further comprise:

determining a first thickness of the first portion of the object of interest; and

determining a second thickness of the second portion of the object of interest.

18. The non-transitory processor-readable storage medium of claim 13, wherein generating the model for the polyline further comprises:

generating an averaged model based on a model generated for each image of the set of images of the video stream.