Eye tracking system with single point calibration

• US 10,115,205 B2
• Filed: 03/10/2017
• Issued: 10/30/2018
• Est. Priority Date: 03/11/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Cavin, Robert Dale, Hunt, Warren Andrew
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/456,383
• Publication Number:
  US 20170263007A1
• Time in Patent Office: 599 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1637 Details related to the display arrangement, including those related to the mounting of the display in the housing
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/514 from specularities
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/421 by analysing segments intersecting the pattern
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

A head mounted display (HMD) comprises an eye tracking system configured to perform a calibration process using an eye tracking system of the HMD that includes determining a pupillary axis and/or determining an angular offset between the pupillary axis and the eye'"'"'s true line of sight. The eye tracking system obtains an eye model captures images of the user'"'"'s pupil while the user is looking at a target or other content displayed on the HMD. In some embodiments, the calibration process is based on a single image of the user'"'"'s eye and is performed only once. For example, the process can be performed the first time the user uses the HMD, which stores the calibration data for the user in a memory for future use.

Claims

1. A method comprising:

obtaining cornea information for an eye of a user, the cornea information including a radius of a corneal sphere of the eye and location of a center of the corneal sphere;

capturing one or more reflections of a pupil of the eye of the user while the user is viewing content displayed on a head-mounted display (HMD);

determining a shape of the pupil based on the one or more captured pupil reflections;

identifying a three-dimensional plane parallel to the pupil, the three-dimensional plane identified based on the obtained cornea information and the identified pupil shape;

identifying a ray originating from the three-dimensional plane and perpendicular to a surface of the corneal sphere;

determining a pupillary axis for the pupil of the user based on the identified ray; and

performing an optical action based in part on the determined pupillary axis.

2. The method of claim 1, further comprising:

identifying a foveal centralis region of the eye of the user;

determining a second ray originating from the foveal centralis region of the eye and passing through the pupil corresponding to a true line of sight of the user; and

determining an angular offset between the determined pupillary axis for the pupil of the user and the true line of sight of the user.

3. The method of claim 1, wherein the cornea information for the eye of the user is obtained by:

capturing one or more reflections of a corneal surface of the eye of the user, the one or more reflections captured while the corneal surface is reflecting light originating from at least two illumination sources positioned at known locations; and

updating a model of the eye including the radius of the corneal sphere of the eye and the location of the center of the corneal sphere of the eye, the model determined based on the captured one or more reflections of the corneal surface and the known locations of the at least two illumination sources.

4. The method of claim 1, wherein determining the shape of the pupil based on the one or more captured pupil reflections includes:

determining, from the one or more captured pupil reflections, a border between the pupil and an iris of the eye based on a change in reflected intensity of one or more wavelengths of light; and

identifying the shape of the pupil from the determined border.

5. The method of claim 1, further comprising:

modifying the pupil shape for the user to correct for refractive distortion caused by the corneal sphere of the user.

6. The method of claim 1, wherein the optical action is selected from the group comprising:

determining a gaze direction of the user, determining a vergence angle of the eye of the user, determining the vergence depth of the user, determining an accommodation depth for the user, performing user identification, determining a torsional state of the eye of the user, or some combination thereof.

7. The method of claim 1, wherein the content displayed by the HMD being viewed by the user includes a known calibration target displayed on an electronic display of the HMD.

8. A head mounted display (HMD) comprising:

an eye tracking system including;

two or more illumination sources configured to illuminate an eye of a user,

a detector configured to capture light reflected from at least one of a cornea or a pupil of the eye of the user as an image, and

a control module configured to;

obtain cornea information for an eye of a user, the cornea information including a radius of a corneal sphere of the eye and location of a center of the corneal sphere,

illuminate the eye of the user with light from the two or more illumination sources,

capture, at the detector, one or more reflections of the pupil of the eye of the user while the user is viewing content displayed by the HMD,

determine a shape of the pupil based on the one or more captured pupil reflections;

identify a three-dimensional plane parallel to the pupil, the three-dimensional plane identified based on the obtained cornea information and the identified pupil shape;

identify a ray originating from the three-dimensional plane and perpendicular to a surface of the corneal sphere;

determine a pupillary axis for the pupil of the user based on the identified ray; and

perform an optical action based in part on the determined pupillary axis.

9. The HMD of claim 8, wherein the cornea information for the eye of the user is obtained by:

capturing one or more reflections of a corneal surface of the eye of the user, the one or more reflections captured while the corneal surface is reflecting light originating from at least two illumination sources positioned at known locations; and

updating a model of the eye including the radius of the corneal sphere of the eye and the location of the center of the corneal sphere of the eye, the model determined based on the captured one or more reflections of the corneal surface and the known locations of the at least two illumination sources.

10. The HMD of claim 8, wherein the two or more illumination sources are part of the electronic display.

11. The HMD of claim 8, wherein determining the shape of the pupil based on the one or more captured pupil reflections includes:

determining, from the one or more captured pupil reflections, a border between the pupil and an iris of the eye based on a change in reflected intensity of one or more wavelengths of light; and

identifying the shape of the pupil from the determined border.

12. The HMD of claim 8, wherein performing the one or more optical actions comprises performing an action selected from a group consisting of:

determining a user'"'"'s vergence angle, determining a user'"'"'s accommodation depth, identifying the user, determining an eye'"'"'s torsional state, or some combination thereof.

13. The HMD of claim 8, wherein the control module is further configured to:

identify a foveal centralis region of the eye of the user;

determine a second ray originating from the foveal centralis region of the eye and passing through the pupil corresponding to a true line of sight of the user; and

determine an angular offset between the determined pupillary axis for the pupil of the user and the true line of sight of the user.

14. A head mounted display (HMD) comprising:

an electronic display configured to display images to the user;

an optics block configured to magnify light received by the optics block from the electronic display; and

an eye tracking system including;

two or more illumination sources configured to illuminate an eye of a user,

a detector configured to capture light reflected from at least one of a cornea or a pupil of the eye of the user as an image, and

a control module configured to;

obtain cornea information for an eye of a user, the cornea information including a radius of a corneal sphere of the eye and location of a center of the corneal sphere,

illuminate the eye of the user with light from the two or more illumination sources,

capture, at the detector, one or more reflections of the pupil of the eye of the user while the user is viewing content displayed by the HMD,

determine a shape of the pupil based on the one or more captured pupil reflections;

identify a three-dimensional plane parallel to the pupil, the three-dimensional plane identified based on the obtained cornea information and the identified pupil shape;

identify a ray originating from the three-dimensional plane and perpendicular to a surface of the corneal sphere;

determine a pupillary axis for the pupil of the user based on the identified ray;

identify a foveal centralis region of the eye of the user;

determine a second ray originating from the foveal centralis region of the eye and passing through the pupil corresponding to a true line of sight of the user;

determine an angular offset between the determined pupillary axis for the pupil of the user and the true line of sight of the user; and

perform an optical action based in part on the angular offset between the determined pupillary axis for the pupil of the user and the true line of sight of the user.

15. The HMD of claim 14, wherein the two or more illumination sources are positioned relative to one another such that the detector is able to capture one or more images of the two or more illumination sources based on the light reflected from the cornea or the pupil.

16. The HMD of claim 14, wherein performing the one or more optical actions comprises performing an action selected from a group consisting of:

determining a user'"'"'s vergence angle, determining a user'"'"'s accommodation depth, identifying the user, determining an eye'"'"'s torsional state, or some combination thereof.

17. The HMD of claim 14, wherein the two or more illumination sources are part of the electronic display.

18. The HMD of claim 14, wherein the cornea information for the eye of the user is obtained by:

capturing one or more reflections of a corneal surface of the eye of the user, the one or more reflections captured while the corneal surface is reflecting light originating from at least two illumination sources positioned at known locations; and

updating a model of the eye including the radius of the corneal sphere of the eye and the location of the center of the corneal sphere of the eye, the model determined based on the captured one or more reflections of the corneal surface and the known locations of the at least two illumination sources.

19. The HMD of claim 14, wherein determining the shape of the pupil based on the one or more captured pupil reflections includes:

determining, from the one or more captured pupil reflections, a border between the pupil and an iris of the eye based on a change in reflected intensity of one or more wavelengths of light; and

identifying the shape of the pupil from the determined border.

20. The HMD of claim 14, wherein the control module is further configured to:

modify the pupil shape for the user to correct for refractive distortion caused by the corneal sphere of the user.