System and method for determining pupil location and iris radius of an eye

• US 10,607,347 B1
• Filed: 09/13/2018
• Issued: 03/31/2020
• Est. Priority Date: 10/15/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Poliakov, Fedir
• Primary Examiner(s): Liu, Li
• Application Number:
  US16/130,684
• Time in Patent Office: 565 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification

Abstract

Systems, devices, media, and methods are presented for segmenting an image of a video stream with a client device, binarizing an area of interest within one or more image, identifying an initial pupil location and an initial iris radius, and determining a final pupil location and a final iris radius. Some embodiments enable the client device to perform one or more operations within a user interface based on the image segmentation.

Claims

1. A computer-implemented method comprising:

identifying, by a processor, an initial pupil location of a pupil of an eye within a set of images of a video stream;

identifying, by the processor, a first distance from a first corner of the eye to a first point on an outer edge of an iris of the eye located closest to the first corner and a second distance from a second corner of the eye to a second point on the outer edge of the iris located closest to the second corner;

determining a total distance extending from the first corner of the eye to the second corner of the eye;

determining, by the processor, an initial iris radius based on the first distance, the second distance, and the total distance; and

determining, by the processor, a final pupil location and a final iris radius based on the initial pupil location and the initial iris radius.

2. The method of claim 1 further comprising:

determining a location of the eye within the video stream; and

binarizing at least a portion of the set of images to generate a binarized image from the video stream.

3. The method of claim 1, wherein an area of interest within the set of images is binarized using a red channel of one or more images of the set of images.

4. The method of claim 1 further comprising:

determining a set of first pixels and a set of second pixels within the set of images, the set of first pixels representing the iris and pupil and having a first value, the set of second pixels representing a noise region having a second value and positioned within the set of first pixels, an area of the noise region being smaller than a predetermined percentage of an area of interest; and

converting the second value of the set of second pixels from the noise region to the first value.

5. The method of claim 1, wherein identifying the initial pupil location further comprises:

identifying a set of first pixels within the set of images, the set of first pixels having a first value and having an area greater than a predetermined percentage of an area of interest; and

determining an approximate center of mass of the set of first pixels.

6. The method of claim 5, wherein determining the final pupil location further comprises:

identifying a perimeter for the set of first pixels, the perimeter including a set of perimeter pixels having a first value;

smoothing the perimeter, by modifying a value of one or more perimeter pixels of the set of perimeter pixels based on values of a set of proximate pixels, to generate a smoothed set of first pixels; and

determining the final pupil location based on the smoothed set of first pixels.

7. The method of claim 1, wherein determining the final iris radius further comprises:

identifying a perimeter for a set of first pixels within the set of images, the set of first pixels having a first value and representing an area of the iris and the pupil;

smoothing the perimeter by modifying a value of one or more perimeter pixels of a set of perimeter pixels based on values of a set of proximate pixels, to generate a smoothed set of first pixels; and

determining the final iris radius based on the smoothed set of first pixels.

8. The method of claim 1 further comprising:

determining a gaze direction associated with the eye across multiple of the set of images; and

performing an action based on the determined gaze.

9. The method of claim 8, wherein the action comprises:

selecting a user interface element responsive to detecting a change in the gaze direction; and

launching an application associated with the user interface element.

10. The method of claim 8, wherein the gaze direction is determined by calculating a line of sight extending from the eye to a display device, the line of sight being represented by a cone extending from the eye to the display device, a radius of the cone and a slant height of the cone being determined based on a location of the eye and an angle of the line of sight.

11. The method of claim 1 further comprising:

detecting an obstruction of the eye within an area of interest in one or more images of the set of images of the video stream to identify one or more obstructed images and one or more unobstructed images in the video stream; and

determining gaze based on the obstruction.

12. A system, comprising:

one or more processors; and

a non-transitory machine-readable storage medium storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

identifying an initial pupil location of a pupil of an eye within a set of images of a video stream;

identifying a first distance from a first corner of the eye to a first point on an outer edge of an iris of the eye located closest to the first corner and a second distance from a second corner of the eye to a second point on the outer edge of the iris located closest to the second corner;

determining a total distance extending from the first corner of the eye to the second corner of the eye;

determining an initial iris radius based on the first distance, the second distance and the total distance; and

determining a final pupil location and a final iris radius based on the initial pupil location and the initial iris radius.

13. The system of claim 12, wherein the processor-executable instructions cause the one or more processors to perform operations comprising:

determining a location of the eye within the video stream; and

binarizing at least a portion of the set of images to generate a binarized image from the video stream.

14. The system of claim 12, wherein an area of interest within the set of images is binarized using a red channel of one or more images of the set of images.

15. The system of claim 12, wherein the processor-executable instructions cause the one or more processors to perform operations comprising:

determining a gaze direction associated with the eye across multiple of the set of images; and

performing an action based on the determined gaze.

16. The system of claim 15, wherein the action comprises:

selecting a user interface element responsive to detecting a change in the gaze direction; and

launching an application associated with the user interface element.

17. The system of claim 15, wherein the gaze direction is determined by calculating a line of sight extending from the eye to a display device, the line of sight being represented by a cone extending from the eye to the display device, a radius of the cone and a slant height of the cone being determined based on a location of the eye and an angle of the line of sight.

18. The system of claim 12, wherein the processor-executable instructions cause the one or more processors to perform operations comprising:

detecting an obstruction of the eye within an area of interest in one or more image of the set of images of the video stream to identify one or more obstructed images and one or more unobstructed images in the video stream; and

determining gaze based on the obstruction.

19. A non-transitory machine-readable storage medium storing processor-executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

identifying an initial pupil location of a pupil of an eye within a set of images of a video stream;

identifying a first distance from a first corner of the eye to a first point on an outer edge of an iris of the eye located closest to the first corner and a second distance from a second corner of the eye to a second point on the outer edge of the iris located closest to the second corner;

determining a total distance extending from the first corner of the eye to the second corner of the eye;

determining an initial iris radius based on the first distance, the second distance and the total distance and

determining a final pupil location and a final iris radius based on the initial pupil location and the initial iris radius.