Detailed eye shape model for robust biometric applications

• US 10,521,661 B2
• Filed: 09/01/2017
• Issued: 12/31/2019
• Est. Priority Date: 09/01/2017
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Chen, Jixu, Amayeh, Gholamreza
• Primary Examiner(s): Couso, Yon J
• Application Number:
  US15/693,975
• Publication Number:
  US 20190073533A1
• Time in Patent Office: 851 Days
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification

Abstract

Systems and methods for robust biometric applications using a detailed eye shape model are described. In one aspect, after receiving an eye image of an eye (e.g., from an eye-tracking camera on an augmented reality display device), an eye shape (e.g., upper or lower eyelids, an iris, or a pupil) of the eye in the eye image is calculated using cascaded shape regression methods. Eye features related to the estimated eye shape can then be determined and used in biometric applications, such as gaze estimation or biometric identification or authentication.

Claims

1. A wearable display system comprising:

an infrared light source configured to illuminate an eye of a user;

an image capture device configured to capture an eye image of the eye;

non-transitory memory configured to store the eye image; and

a hardware processor in communication with the non-transitory memory, the hardware processor programmed to;

receive the eye image from the non-transitory memory;

estimate an eye shape from the eye image using cascaded shape regression, the eye shape comprising a pupil shape, an iris shape, or an eyelid shape; and

perform a biometric application based at least in part on the eye shape

wherein to estimate the eye shape from the eye image using cascaded shape regression, the hardware processor is programmed to iterate a regression function for determining a shape increment over a plurality of stages, the regression function comprising a shape-indexed extraction function, and

wherein to iterate the regression function, the hardware processor is programmed to evaluate

Δ

S_t=ƒ

_t(Φ

_t(I,S<sub>t−

1</sub>)),

for a shape increment Δ

S_t at stage t of the iteration, where ƒ

_t is the regression function at stage t, Φ

_t is the shape-indexed extraction function at stage t, I is the eye image, and S<sub>t−

1</sub> is the eye shape at stage t−

1 of the iteration.

2. The wearable display system of claim 1, wherein the hardware processor is further programmed to determine an eye feature based at least in part on the eye shape, wherein the eye feature comprises at least one of a glint from the infrared light source, a blood vessel, an iris feature, or a center of the pupil.

3. The wearable display system of claim 1, wherein the biometric application comprises determination of eye gaze.

4. The wearable display system of claim 3, wherein the eye shape comprises the iris shape, and the hardware processor is programmed to search for glints from the infrared light source that are within the iris shape.

5. The wearable display system of claim 1, wherein the eye shape comprises the pupil shape and the eyelid shape, and the hardware processor is programmed to identify a portion of the pupil that is occluded by the eyelid.

6. The wearable display system of claim 5, wherein the hardware processor is programmed to determine a pupillary boundary based on the pupil shape without the portion of the pupil that is occluded by the eyelid.

7. The wearable display system of claim 1, wherein the eye shape comprises the iris shape and the eyelid shape, and the hardware processor is programmed to identify a portion of the iris that is occluded by the eyelid.

8. The wearable display system of claim 7, wherein the hardware processor is programmed to determine a limbic boundary based on the iris shape without the portion of the iris that is occluded by the eyelid.

9. The wearable display system of claim 1, wherein eye shape comprises the eyelid shape, and the biometric application comprises determination of eye blink.

10. The wearable display system of claim 9, wherein the hardware processor is programmed to reject or assign a lower weight to the eye image if a distance between an upper eyelid and a lower eyelid is less than a threshold.

11. The wearable display system of claim 1, wherein the eye shape comprises a boundary to a pupil, an iris, or an eyelid.

12. The wearable display system of claim 1, wherein the biometric application comprises biometric identification or biometric authentication.

13. The wearable display system of claim 1, wherein the shape-indexed extraction function provides a comparison of eye image values between a pair of pixel locations.

14. A method for training an eye shape calculation engine, the method comprising:

under control of a hardware processor;

receiving a set of annotated training eye images, wherein each image in the set is labeled with an eye shape; and

using a machine learning technique applied to the set of annotated training eye images to learn a regression function and a shape-indexed extraction function, where the regression function and the shape-indexed extraction function learn to recognize the eye shape, and wherein the regression function and the shape-indexed extraction function are learned to recognize eye shape according to an iteration of

Δ

S_t=ƒ

_t(Φ

_t(I,S<sub>t−

1</sub>)),

for a shape increment Δ

S_t at stage t of the iteration, where ƒ

_t is the regression function at stage t,Φ

_t is the shape-indexed extraction function at stage t, I is an unlabeled eye image, and S<sub>t−

1</sub> is the eye shape at stage t−

1 of the iteration.

15. The method of claim 14, wherein the eye shape comprises a shape of a pupil, a shape of an iris, or a shape of an eyelid.

16. The method of claim 14, wherein the shape-indexed extraction function provides a comparison of eye image values between a pair of pixel locations.

17. The method of claim 16, wherein the comparison comprises a binary or Boolean value.

18. A wearable display system comprising:

an infrared light source configured to illuminate an eye of a user;

an image capture device configured to capture an eye image of the eye;

non-transitory memory configured to store the eye image; and

a hardware processor in communication with the non-transitory memory, the hardware processor programmed to;

receive the eye image from the non-transitory memory;

estimate an eye shape from the eye image using cascaded shape regression, the eye shape comprising an eyelid shape;

perform a biometric application comprising determination of eye blink based at least in part on the eye shape; and

reject or assign a lower weight to the eye image if a distance between an upper eyelid and a lower eyelid is less than a threshold.