Eyelid shape estimation using eye pose measurement

• US 10,671,845 B2
• Filed: 11/02/2018
• Issued: 06/02/2020
• Est. Priority Date: 08/21/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Kaehler, Adrian
• Primary Examiner(s): Tsai, Tsung Yin
• Application Number:
  US16/179,541
• Publication Number:
  US 20190087634A1
• Time in Patent Office: 578 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Systems and methods for eyelid shape estimation are disclosed. In one aspect, after receiving an eye image of an eye (e.g., from an image capture device), an eye pose of the eye in the eye image is determined. From the eye pose, an eyelid shape (of an upper eyelid or a lower eyelid) can be estimated using an eyelid shape mapping model. The eyelid shape mapping model relates the eye pose and the eyelid shape. In another aspect, the eyelid shape mapping model is learned (e.g., using a neural network).

Claims

1. A method for estimating pose of an eye, the method comprising:

under control of a hardware processor;

detecting, from an image of the eye, a boundary between an eyelid of an eye and an iris of the eye using an edge detector;

determining an eyelid shape of the eye using the boundary between the eyelid of the eye and the iris of the eye,

wherein an eyelid shape coordinate system of the eyelid shape comprises a horizontal shift, a vertical shift, and a curvature of the eyelid,

wherein a functional relationship between the eyelid shape coordinate system and an eye pose coordinate system comprises a mapping matrix, and

wherein the eye pose coordinate system comprises an azimuthal deflection angle of the eye and a zenithal deflection angle of the eye relative to a resting orientation of the eye; and

estimating an eye pose of the eye based at least in part on the eyelid shape and the functional relationship.

2. The method of claim 1, wherein determining the eyelid shape of the eye comprises fitting a parabolic curve to the boundary between the eyelid of the eye and the iris of the eye.

3. The method of claim 1, further comprising:

applying a morphological filter to the eye image prior to detecting the boundary between the eyelid of the eye and the iris of the eye using the edge detector.

4. The method of claim 1, wherein the edge detector comprises a Canny edge detector.

5. The method of claim 1, wherein the mapping matrix is determined by a regression analysis of eye poses and eyelid shapes.

6. The method of claim 1, further comprising determining, from the eye image, biometric information of the eye.

7. The method of claim 6, wherein the biometric information comprise an iris code of the eye.

8. A head mounted display system comprising:

an image capture device configured to capture an eye image;

non-transitory memory configured to store an eye pose mapping model; and

a hardware processor in communication with the non-transitory memory, the hardware processor programmed to;

receive the eye image captured by the image capture device;

determine an eyelid shape of an eye in the eye image; and

estimate an eye pose of the eye based at least in part on the eyelid shape and the eye pose mapping model,

wherein the eye pose mapping model relates the eyelid shape to the eye pose.

9. The head mounted display system of claim 8, wherein the eye pose comprises a measurement of an azimuthal angle and a zenithal angle of the eye relative to a resting orientation of the eye.

10. The head mounted display system of claim 8, wherein to determine the eyelid shape, the hardware processor is programmed to determine a horizontal shift, a vertical shift, and a curvature of the eyelid.

11. The head mounted display system of claim 8, wherein the eye pose mapping model comprises a parametric relationship between the eyelid shape and the eye pose.

12. The head mounted display system of claim 8, wherein the parametric relationship comprises a relationship between an azimuthal angle and a zenith angle of the eye and eyelid shape parameters.

13. The head mounted display system of claim 12, wherein the eyelid shape parameters comprise a horizontal shift, a vertical shift, and a curvature of the eyelid.

14. The head mounted display system of claim 8, wherein the eye pose mapping model comprises a neural network.

15. The head mounted display system of claim 8, herein the hardware processor is further programmed to:

determine biometric data of the eye using the eye pose.

16. A method for estimating an eye pose from an eyelid shape, the method comprising:

under control of a hardware processor;

determining an eyelid shape of an eye in an eye image; and

estimating an eye pose based at least partly on the eyelid shape.

17. The method of claim 16, wherein estimating the eye pose comprises applying an eye pose mapping model to the eyelid shape.

18. The method of claim 17, wherein the eye pose mapping model comprises a parametric relationship between the eyelid shape and the eye pose.

19. The method of claim 18, wherein the eyelid shape is represented by a polynomial.

20. The method of claim 17, wherein the eye pose mapping model comprises a neural network.

21. The method of claim 16, further comprising:

re-determining the eyelid shape based on the eye pose; and

verifying consistency of the determined eyelid shape and the re-determined eyelid shape.