Eye pose identification using eye features

• US 10,163,010 B2
• Filed: 10/12/2016
• Issued: 12/25/2018
• Est. Priority Date: 10/16/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Kaehler, Adrian, Klug, Michael Anthony, Amayeh, Gholamreza
• Primary Examiner(s): Dulaney, Kathleen Y
• Application Number:
  US15/291,929
• Publication Number:
  US 20170109580A1
• Time in Patent Office: 804 Days
• Field of Search: 382117, 382103, 345633, 345 8, 345156, 351209, 359630, 396 51
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification

Abstract

Systems and methods for eye pose identification using features of an eye are described. Embodiments of the systems and methods can include segmenting an iris of an eye in the eye image to obtain pupillary and limbic boundaries of the eye, determining two angular coordinates (e.g., pitch and yaw) of an eye pose using the pupillary and limbic boundaries of the eye, identifying an eye feature of the eye (e.g., an iris feature or a scleral feature), determining a third angular coordinate (e.g., roll) of the eye pose using the identified eye feature, and utilizing the eye pose measurement for display of an image or a biometric application. In some implementations, iris segmentation may not be performed, and the two angular coordinates are determined from eye features.

Claims

1. A head mounted display (HMD) system comprising:

an image capture device for tracking an eye pose of an eye of a wearer of the HMD system in an eye image of the eye of the wearer, wherein the eye pose comprises a direction toward which the eye is looking;

non-transitory memory configured to store the eye image;

a display for providing virtual image information to the wearer of the HMD system based on the eye pose of the wearer in the eye image; and

a hardware processor programmed to;

receive the eye image from the image capture device;

map a pupil in the eye image to an equivalent frontal view to provide a remapped eye image;

identify an eye feature based at least partly on the remapped eye image;

determine a pitch and a yaw of the eye based at least partly on the remapped eye image;

determine a roll of the eye based at least partly on the eye feature in the remapped eye image;

determine the eye pose of the eye based at least partly on the pitch, the yaw, and the roll;

determine the virtual image information to be provided to the wearer of the HMD using the eye pose of the eye; and

cause the display to provide the virtual image information to the wearer of the HMD system.

2. The head mounted display system of claim 1, wherein to determine the roll of the eye, the hardware processor is programmed to determine a homography between the eye image and a reference eye image.

3. The head mounted display system of claim 2, wherein the reference eye image comprises an image of the eye of the wearer of the HMD system in a resting eye state.

4. The head mounted display system of claim 3, wherein the resting eye state of the eye of the wearer of the HMD system corresponds to a state in which the eye of the wearer of the HMD system is looking forward.

5. The head mounted display system of claim 1, wherein to determine the roll of the eye, the hardware processor is programmed to utilize a polar coordinate representation of the eye image.

6. The head mounted display system of claim 1, wherein to determine the roll of the eye, the hardware processor is programmed to compare an iris code of the eye image to an iris code from a reference eye image.

7. The head mounted display system of claim 1, wherein the hardware processor is further programmed to:

determine biometric data of the eye using the eye pose of the eye image.

8. The head mounted display system of claim 1, wherein the eye feature comprises an iris feature, wherein the iris feature comprises a texture, a pattern, or a keypoint in the iris.

9. The head mounted display system of claim 1, wherein the eye feature comprises a scleral feature, wherein the scleral feature comprises a blood vessel.

10. The head mounted display system of claim 1, wherein the processor is further programmed to segment an iris of the eye in the eye image.

11. The head mounted display system of claim 1, wherein the direction toward which the eye is looking comprises a direction toward which the eye is looking through the display.

12. The head mounted display system of claim 1, wherein the hardware processor is programmed to map the pupil in the eye image to the equivalent frontal view to provide the remapped eye image based at least in part on one or more intrinsic parameters of the image capture device.

13. The head mounted display system of claim 1, wherein the hardware processor is further programmed to identify one or more predicted areas of occlusion by an eyelid of the eye over an iris of the eye.

14. The head mounted display system of claim 1, wherein the hardware processor is programmed to normalize a radial dimension extending from a pupillary boundary to a limbic boundary.

15. The head mounted display system of claim 1, wherein, the hardware processor is programmed to invert a homography computed from a limbic boundary to map the pupil in the eye image to an equivalent frontal view to provide the remapped eye image.