NEURAL NETWORK FOR EYE IMAGE SEGMENTATION AND IMAGE QUALITY ESTIMATION

• US 20180089834A1
• Filed: 05/25/2017
• Published: 03/29/2018
• Est. Priority Date: 09/29/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Spizhevoy, Alexey, Kaehler, Adrian, Badrinarayanan, Vijay
• Granted Patent: US 10,445,881 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification

Abstract

Systems and methods for eye image segmentation and image quality estimation are disclosed. In one aspect, after receiving an eye image, a device such as an augmented reality device can process the eye image using a convolutional neural network with a merged architecture to generate both a segmented eye image and a quality estimation of the eye image. The segmented eye image can include a background region, a sclera region, an iris region, or a pupil region. In another aspect, a convolutional neural network with a merged architecture can be trained for eye image segmentation and image quality estimation. In yet another aspect, the device can use the segmented eye image to determine eye contours such as a pupil contour and an iris contour. The device can use the eye contours to create a polar image of the iris region for computing an iris code or biometric authentication.

Claims

1. A system for eye image segmentation and image quality estimation, the system comprising:

an eye-imaging camera configured to obtain an eye image;

non-transitory memory configured to store the eye image;

a hardware processor in communication with the non-transitory memory, the hardware processor programmed to;

receive the eye image;

process the eye image using a convolution neural network to generate a segmentation of the eye image; and

process the eye image using the convolution neural network to generate a quality estimation of the eye image,

wherein the convolution neural network comprises a segmentation tower and a quality estimation tower,

wherein the segmentation tower comprises segmentation layers and shared layers,

wherein the quality estimation tower comprises quality estimation layers and the shared layers,

wherein a first output layer of the shared layers is connected to a first input layer of the segmentation tower and to a second input layer of the segmentation tower, at least one of the first input layer or the second input layer comprising a concatenation layer,

wherein the first output layer of the shared layers is connected to an input layer of the quality estimation layer, and

wherein the eye image is received by an input layer of the shared layers.

2. The system of claim 1, wherein a second output layer of the shared layers is connected to a third input layer of the segmentation tower, the third input layer comprising a concatenation layer.

3. The system of claim 1, wherein to process the eye image using the convolution neural network to generate the segmentation of the eye image, the hardware processor is programmed to

generate the segmentation of the eye image using the segmentation tower,

wherein an output of an output layer of the segmentation tower comprises the segmentation of the eye image.

4. The system of claim 3, wherein the segmentation of the eye image includes a background, a sclera, an iris, or a pupil of the eye image.

5. The system of claim 4, wherein the hardware processor is further programmed to:

determine a pupil contour of an eye in the eye image using the segmentation of the eye image;

determine an iris contour of the eye in the eye image using the segmentation of the eye image; and

determine a mask for an irrelevant area in the eye image.

6. The system of claim 1, wherein the shared layers are configured to encode the eye image by decreasing a spatial dimension of feature maps and increasing a number of feature maps computed by the shared layers.

7. The system of claim 6, wherein the segmentation layers are configured to decode the eye image encoded by the shared layers by increasing the spatial dimension of the feature maps and reducing the number of feature maps.

8. The system of claim 1, wherein to process the eye image using the convolution neural network to generate the quality estimation of the eye image, the hardware processor is programmed to:

generate the quality estimation of the eye image using the quality estimation tower,

wherein an output of an output layer of the quality estimation tower comprises the quality estimation of the eye image.

9. The system of claim 1, wherein the quality estimation tower is configured to output at least two channels of output, wherein a first of the at least two channels comprises a good quality estimation and a second of the at least two channels comprises a bad quality estimation.

10. The system of claim 1, wherein the shared layers, the segmentation layers, or the quality estimation layers comprise a convolution layer, a brightness normalization layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a concatenation layer, a pooling layer, a fully connected layer, a linear fully connected layer, a softsign layer, or any combination thereof.

11. A system for eye image segmentation and image quality estimation, the system comprising:

an eye-imaging camera configured to obtain an eye image;

non-transitory memory configured to store the eye image;

a hardware processor in communication with the non-transitory memory, the hardware processor programmed to;

receive the eye image;

process the eye image using a convolution neural network to generate a segmentation of the eye image; and

process the eye image using the convolution neural network to generate a quality estimation of the eye image,

wherein the convolution neural network comprises a segmentation tower and a quality estimation tower,

wherein the segmentation tower comprises segmentation layers and shared layers,

wherein the quality estimation tower comprises quality estimation layers and the shared layers,

wherein the segmentation layers are not shared with the quality estimation tower,

wherein the quality estimation layers are not shared with the segmentation tower, and

wherein the eye image is received by an input layer of the shared layers.

12. The system of claim 11, wherein a first output layer of the shared layers is connected to a first input layer of the segmentation tower.

13. The system of claim 12, wherein the first output layer of the shared layers is connected to a second input layer of the segmentation tower,

wherein the first input layer or the second input layer comprises a concatenation layer.

14. The system of claim 12, wherein the first output layer of the shared layers is further connected to an input layer of the quality estimation tower.

15. The system of claim 11,

wherein to process the eye image using the convolution neural network to generate the segmentation of the eye image, the hardware processor is programmed to:

generate the segmentation of the eye image using the segmentation tower,

wherein an output of an output layer of the segmentation tower comprises the segmentation of the eye image.

16. The system of claim 11, wherein the segmentation of the eye image includes a background, a sclera, an iris, or a pupil of the eye image.

17. The system of claim 11, wherein to process the eye image using the convolution neural network to generate the quality estimation of the eye image, the hardware processor is programmed to:

generate the quality estimation of the eye image using the quality estimation tower,

wherein an output of an output layer of the quality estimation tower comprises the quality estimation of the eye image.

18. The system of claim 11, wherein the shared layers, the segmentation layers, or the quality estimation layers comprise a convolution layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a concatenation layer, a pooling layer, a fully connected layer, a linear fully connected layer, or any combination thereof.

19. The system of claim 18, wherein the batch normalization layer is a batch local contrast normalization layer or a batch local response normalization layer.

20. The system of claim 11, wherein the shared layers, the segmentation layers, or the quality estimation layers comprise a brightness normalization layer, a softsign layer, or any combination thereof.