Eyelid shape estimation

• US 10,089,526 B2
• Filed: 08/16/2016
• Issued: 10/02/2018
• Est. Priority Date: 08/21/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Amayeh, Gholamreza, Kaehler, Adrian
• Primary Examiner(s): Motsinger, Sean
• Application Number:
  US15/238,549
• Publication Number:
  US 20170053166A1
• Time in Patent Office: 777 Days
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Systems and methods for eyelid shape estimation are disclosed. In one aspect, after receiving an eye image of an eye (e.g., from an image capture device), an eye-box is generated over an iris of the eye in the eye image. A plurality of radial lines can be generated from approximately the center of the eye-box to an upper edge or a lower edge of the eye box. Candidate points can be determined to have local maximum derivatives along the plurality of radial lines. From the candidate points, an eyelid shape curve (e.g., for an upper eyelid or a lower eyelid) can be determined by fitting a curve (e.g., a parabola or a polynomial) to the candidate points or a subset of the candidate points.

Claims

1. A method for eye tracking, comprising:

under control of a hardware processor;

generating an eye-box over an eye in an eye image for determining an eye pose of the eye in the eye image, wherein the eye pose comprises a direction toward which the eye is looking,

wherein the eye-box has an upper edge and a lower edge,

wherein the upper edge overlays a portion of an upper eyelid of the eye, and

wherein the lower edge overlays a portion of a lower eyelid of the eye;

generating a plurality of radial lines extending from approximately the center of the eye-box to the upper edge or the lower edge;

determining candidate points on the plurality of radial lines using an edge detector;

determining parameters of a parabolic fit curve from a subset of the candidate points, wherein the parabolic fit curve is an estimation of an eyelid shape of the eye in the eye image;

determining a score of the parabolic fit curve is above a threshold score using the parameters of the parabolic fit curve;

determining that the parabolic fit curve is a preferred estimation of the eyelid shape of the eye in the eye image; and

determining the eye pose of the eye in the eye image using parameters of the preferred estimation of the eyelid shape; and

tracking the eye in the eye image using the eye pose of the eye in the eye image.

2. The method of claim 1, wherein the eye-box is rectangular or square in shape.

3. The method of claim 1, wherein determining the score of the parabolic fit curve is above the threshold score comprises determining a number of the candidate points that intersect the parabolic fit curve.

4. The method of claim 1, wherein determining the score of the parabolic fit curve comprises determining a number of the candidate points that are within a threshold distance of the parabolic fit curve.

5. The method of claim 1, wherein determining the candidate points on the plurality of radial lines using the edge detector comprises determining the candidate points on a subset of the plurality of radial lines using the edge detector.

6. A head mounted display (HMD) system comprising:

an image capture device for tracking an eye pose of an eye of a wearer of the HMD system in an eye image of the eye of the wearer, wherein the eye pose comprises a direction toward which the eye is looking;

non-transitory memory configured to store the eye image and an edge detector;

a display for providing virtual image information to the wearer of the HMD system based on the eye pose of the wearer in the eye image; and

a hardware processor in communication with the non-transitory memory, the hardware processor programmed to;

receive the eye image from the image capture device;

generate an eye-box around an iris of the eye, wherein a first edge of the eye-box is tangent to a limbic boundary of the eye;

generate a plurality of lines extending to a second edge of the eye-box;

determine candidate points on the plurality of lines in the eye image using the edge detector;

determine parameters of an eyelid shape of the eye using a first subset of the candidate points;

determine the eye pose of the eye using the parameters of the eyelid shape of the eye;

determine the virtual image information to be provided to the wearer of the HMD using the eye pose of the eye; and

cause the display to provide the virtual image information to the wearer of the HMD system.

7. The head mounted display system of claim 6, wherein the plurality of lines comprises a plurality of radial lines, and wherein the plurality of radial lines extends from approximately the center of the eye-box to the second edge of the eye-box.

8. The head mounted display system of claim 6, wherein the plurality of lines comprises a plurality of vertical lines, and wherein the plurality of vertical lines extends from a bisector line of the eye-box to the second edge of the eye-box.

9. The head mounted display system of claim 6, wherein the candidate points comprise points with local maximum derivatives along the plurality of the lines.

10. The head mounted display system of claim 6, wherein the candidate points are not along a pupillary boundary or a limbic boundary.

11. The method of claim 6, wherein the hardware processor is further programmed to:

apply a morphological filter to the eye image prior to determining the candidate points.

12. The head mounted display system of claim 6, wherein determine the eyelid shape using the first subset of the candidate points comprises determine a functional form of the eyelid shape using the first subset of the candidate points.

13. The head mounted display system of claim 12, wherein the functional form is a parabola or a polynomial.

14. The head mounted display system of claim 6, wherein determine the eyelid shape using the first subset of the candidate points comprises re-determine the eyelid shape using a second subset of the candidate points if the determined eyelid shape has a negative curvature and the determined eyelid shape is an estimation of the upper eyelid or if the determined eyelid shape has a positive curvature and the determined eyelid shape is an estimation of the lower eyelid.

15. A method for eye tracking, the method comprising:

under control of a hardware processor;

identifying a portion of an eye image comprising at least a portion of a pupil and an iris of an eye;

generating a plurality of lines extending to an upper edge or a lower edge of the portion of the eye image;

determining candidate points on the plurality of lines in the eye image using an edge detector; and

determining parameters of an eyelid shape of the eye in the eye image using a subset of the candidate points comprising;

determining a first functional form of the eyelid shape using a first subset of the candidate points;

determining a second functional form of the eyelid shape using a second subset of the candidate points;

determining a score of the first functional form of the eyelid shape;

determining a score of the second functional form of the eyelid shape; and

determining a preferred estimation of the eyelid shape is the first functional form of the eyelid shape if the score of the first functional form is greater than the score of the second functional form, and the second functional form of the eyelid shape otherwise; and

determining an eye pose of the eye in the eye image using parameters of the preferred estimation of the eyelid shape, wherein the eye pose comprises a direction toward which the eye is looking; and

tracking the eye in the eye image using the eye pose of the eye in the eye image.

16. The method of claim 15, further comprising:

randomly sampling the candidate points to generate the subset of the candidate points.

17. The method of claim 15, wherein the edge detector comprises a Sobel filter, and wherein determining the candidate points on the plurality of lines using the edge detector comprises:

convolving the Sobel filter with the eye image to generate a derivative approximation image result; and

identifying the candidate points using the derivative approximation image result.

18. The method of claim 15, wherein identifying the portion of the eye image comprises generating an eye-box over the eye in the eye image.