Image segmentation of a video stream

• US 10,102,634 B1
• Filed: 11/22/2017
• Issued: 10/16/2018
• Est. Priority Date: 10/15/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Poliakov, Fedir
• Primary Examiner(s): Liu, Li
• Application Number:
  US15/821,405
• Time in Patent Office: 328 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification

Abstract

Systems, devices, media, and methods are presented for segmenting an image of a video stream with a client device, binarizing an area of interest within one or more image, identifying an initial pupil location and an initial iris radius, and determining a final pupil location and a final iris radius. Some embodiments enable the client device to perform one or more operations within a user interface based on the image segmentation.

Claims

1. A computer implemented method for segmenting a video stream, comprising:

determining an approximate location of an eye within a video stream comprising a set of images to identify an area of interest within one or more images of the set of images;

binarizing the area of interest within the one or more images to generate a binarized image from the video stream;

identifying an initial pupil location of a pupil of the eye within the binarized image;

determining an initial iris radius for the eye by identifying, within the binarized image, a first distance from a first corner of the eye to an outer edge of the iris and a second distance from a second corner of the eye to the outer edge of the iris and determining the initial iris radius based on the first distance and the second distance; and

determining a final pupil location and a final iris radius based at least in part on the initial pupil location and the initial iris radius.

2. The method of claim 1 further comprising:

cropping one or more images of the video stream to isolate the area of interest within the one or more images of the video stream.

3. The method of claim 1, wherein the area of interest is binarized using a red channel of the one or more images of the set of images.

4. The method of claim 1 further comprising:

determining a set of first pixels and a set of second pixels within the binarized image, the set of first pixels representing the iris and pupil and having a first value, the set of second pixels representing a noise region having a second value and positioned within the set of first pixels, an area of the noise region being smaller than a predetermined percentage of the area of interest; and

converting the second value of the set of second pixels from the noise region to the first value.

5. The method of claim 1, wherein identifying the initial pupil location further comprises:

identifying a set of first pixels within the binarized image, the set of first pixels having a first value and having an area greater than a predetermined percentage of the area of interest; and

determine an approximate center of mass of the set of first pixels.

6. The method of claim 5, wherein determining the final pupil location further comprises:

identifying a perimeter for the set of first pixels, the perimeter including a set of perimeter pixels having a first value;

smoothing the perimeter, by modifying a value of one or more perimeter pixels of the set of perimeter pixels based on values of a set of proximate pixels, to generate a smoothed set of first pixels; and

determining the final pupil location based on the smoothed set of first pixels.

7. The method of claim 1, wherein determining the final iris radius further comprises:

identifying a perimeter for a set of first pixels within the binarized image, the set of first pixels having a first value and representing an area of the iris and the pupil;

smoothing the perimeter by modifying a value of one or more perimeter pixels of the set of perimeter pixels based on values of a set of proximate pixels, to generate a smoothed set of first pixels; and

determining the final iris radius based on the smoothed set of first pixels.

8. The method of claim 1 further comprising:

forming a binary mask for the eye, the binary mask identifying a set of pixels associated with the iris of the eye.

9. The method of claim 1, wherein the video stream is a first portion of a video stream, and further comprising:

tracking at least one of the final pupil location and the final iris radius across one or more images included in a second portion of the video stream.

10. The method of claim 1 further comprising:

detecting an obstruction of the eye within the area of interest in one or more image within the video stream to identify one or more obstructed images and one or more unobstructed images in the video stream; and

binarizing the area of interest within the one or more unobstructed images to generate a binarized image from the video stream.

11. An system, comprising:

one or more processors; and

a non-transitory machine-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

determining an approximate location of an eye within a video stream comprising a set of images to identify an area of interest within one or more images of the set of images;

binarizing the area of interest within the one or more images to generate a binarized image from the video stream;

identifying an initial pupil location of a pupil of the eye within the binarized image;

determining an initial iris radius for the eye by identifying, within the binarized image, a first distance from a first corner of the eye to an outer edge of the iris and a second distance from a second corner of the eye to the outer edge of the iris and determining the initial iris radius based on the first distance and the second distance; and

determining a final pupil location and a final iris radius based at least in part on the initial pupil location and the initial iris radius.

12. The system of claim 11, wherein the processor executable instructions cause the one or more processors to perform operations comprising:

determining a set of first pixels and a set of second pixels within the binarized image, the set of first pixels representing the iris and pupil and having a first value, the set of second pixels representing a noise region having a second value and positioned within the set of first pixels, an area of the noise region being smaller than a predetermined percentage of the area of interest; and

converting the second value of the set of second pixels from the noise region to the first value.

13. The system of claim 11, wherein in identifying the initial pupil location, the processor executable instructions cause the one or more processors to perform operations comprising:

identifying a set of first pixels within the binarized image, the set of first pixels having a first value and having an area greater than a predetermined percentage of the area of interest; and

determine an approximate center of mass of the set of first pixels.

14. The system of claim 13, wherein in determining the final pupil location, the processor executable instructions cause the one or more processors to perform operations comprising:

identifying a perimeter for the set of first pixels, the perimeter including a set of perimeter pixels having a first value;

smoothing the perimeter, by modifying a value of one or more perimeter pixels of the set of perimeter pixels based on values of a set of proximate pixels, to generate a smoothed set of first pixels; and

determining the final pupil location based on the smoothed set of first pixels.

15. The system of claim 11, wherein in determining the final iris radius, the processor executable instructions cause the one or more processors to perform operations comprising:

identifying a perimeter for a set of first pixels within the binarized image, the set of first pixels having a first value and representing an area of the iris and the pupil;

smoothing the perimeter, by modifying a value of one or more perimeter pixels of the set of perimeter pixels based on values of a set of proximate pixels, to generate a smoothed set of first pixels; and

determining the final iris radius based on the smoothed set of first pixels.

16. The system of claim 11, wherein the processor executable instructions cause the one or more processors to perform operations comprising:

tracking at least one of the final pupil location and the final iris radius across one or more images included in a second portion of the video stream.

17. The system of claim 11, wherein the processor executable instructions cause the one or more processors to perform operations comprising:

detecting an obstruction of the eye within the area of interest in one or more image within the video stream to identify one or more obstructed images and one or more unobstructed images in the video stream; and

binarizing the area of interest within the one or more unobstructed images to generate a binarized image from the video stream.

18. A non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

determining an approximate location of an eye within a video stream comprising a set of images to identify an area of interest within one or more images of the set of images;

binarizing the area of interest within the one or more images to generate a binarized image from the video stream;

identifying an initial pupil location of a pupil of the eye within the binarized image;

determining an initial iris radius for the eye by identifying, within the binarized image, a first distance from a first corner of the eye to an outer edge of the iris and a second distance from a second corner of the eye to the outer edge of the iris and determining the initial iris radius based on the first distance and the second distance; and

determining a final pupil location and a final iris radius based at least in part on the initial pupil location and the initial iris radius.

19. The non-transitory machine-readable storage medium of claim 18, wherein the operations further comprise:

determining a set of first pixels and a set of second pixels within the binarized image, the set of first pixels representing the iris and pupil and having a first value, the set of second pixels representing a noise region having a second value and positioned within the set of first pixels, an area of the noise region being smaller than a predetermined percentage of the area of interest; and

converting the second value of the set of second pixels from the noise region to the first value.

20. The non-transitory machine-readable storage medium of claim 18, wherein the operations further comprise:

identifying a set of first pixels within the binarized image, the set of first pixels having a first value and having an area greater than a predetermined percentage of the area of interest;

determine an approximate center of mass of the set of first pixels;

identifying a perimeter for the set of first pixels, the perimeter including a set of perimeter pixels having a first value;

smoothing the perimeter, by modifying a value of one or more perimeter pixels of the set of perimeter pixels based on values of a set of proximate pixels, to generate a smoothed set of first pixels; and

determining the final pupil location based on the smoothed set of first pixels.