Eye tracking system with single point calibration

• US 10,614,577 B1
• Filed: 10/16/2018
• Issued: 04/07/2020
• Est. Priority Date: 03/11/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Cavin, Robert Dale, Hunt, Warren Andrew
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US16/162,183
• Time in Patent Office: 539 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1637 Details related to the display arrangement, including those related to the mounting of the display in the housing
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/514 from specularities
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/421 by analysing segments intersecting the pattern
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

A head mounted display (HMD) comprises an eye tracking system configured to perform a calibration process using an eye tracking system of the HMD that includes determining a pupillary axis and/or determining an angular offset between the pupillary axis and the eye'"'"'s true line of sight. The eye tracking system obtains an eye model captures images of the user'"'"'s pupil while the user is looking at a target or other content displayed on the HMD. In some embodiments, the calibration process is based on a single image of the user'"'"'s eye and is performed only once. For example, the process can be performed the first time the user uses the HMD, which stores the calibration data for the user in a memory for future use.

Claims

1. A headset comprising:

an eye tracking system including two or more illumination sources configured to illuminate an eye of a user, the eye tracking system configured to;

obtain a model of the eye of the user including a radius of a corneal sphere of the eye and location of a center of the corneal sphere;

capture one or more images of reflections of the eye of the user reflected from the eye of the user;

determine, from the one or more images of the reflections, a shape of a pupil of the eye of the user;

determine a plane corresponding to a location of the pupil, the plane being identified using the determined shape of the pupil, the radius of the corneal sphere, and the location of the center of the corneal sphere from the model of the eye;

determine, from the plane, a pupillary axis for the eye based on the identified plane parallel to the pupil; and

perform an optical action based on the determined pupillary axis for the eye.

2. The headset of claim 1, wherein the model of the eye is generated by:

capturing an image of one or more reflections of the eye reflected from a cornea of the eye of the user;

determining a radius of a corneal sphere of the cornea based on the captured image of the reflections; and

determining a center of the corneal sphere based on the determined radius of the corneal sphere.

3. The headset of claim 1, wherein the optical action is selected from the group comprising:

determining a gaze direction of the user, determining a vergence angle of the eye of the user, determining the vergence depth of the user, determining an accommodation depth for the user, performing user identification, determining a torsional state of the eye of the user, or some combination thereof.

4. The headset of claim 1, wherein determining the shape of the pupil based on the one or more captured pupil reflections includes:

determining, from the one or more reflections, a border between the pupil and an iris of the eye based on a change in reflected intensity of one or more wavelengths of light; and

identifying the shape of the pupil from the determined border.

5. The headset of claim 1, wherein the user is viewing content displayed by the headset while the one or more images of reflections of the eye of the user reflected from the cornea of the eye of the user are reflected, and wherein the content includes a known calibration target displayed on an electronic display of the headset.

6. The headset of claim 1, wherein the pupillary axis derived by:

identifying a ray originating from the plane that is perpendicular to a surface of the corneal sphere.

7. The headset of claim 6, further comprising:

identifying a foveal centralis region of the eye of the user;

determining a second ray originating from the foveal centralis region of the eye and passing through the pupil corresponding to a true line of sight of the user; and

determining an angular offset between the determined pupillary axis for the pupil of the user and the true line of sight of the user.

8. A method comprising:

capturing, by an eye tracking system of a head mounted display (HMD), one or more images of reflections of an eye of a user, the eye tracking system including two or more illumination sources configured to illuminate the eye of the user;

identifying, from the one or more images of the reflections, a shape of a pupil of the eye of the user;

identifying a plane parallel to the pupil based on the identified shape of the pupil

determining a pupillary axis for the eye based on the identified plane parallel to the pupil; and

performing an optical action based on the determined pupillary axis.

9. The method of claim 8, wherein the plane parallel to the pupil is determined based on the shape of the pupil and further based on a model of the eye of the user, wherein the model of the eye is generated by:

capturing an image of one or more reflections of the eye reflected from a cornea of the eye of the user;

determining a radius of a corneal sphere of the cornea based on the captured image of the reflections; and

determining a center of the corneal sphere based on the determined radius of the corneal sphere.

10. The method of claim 8, wherein identifying the shape of the pupil comprises:

determining, from the one or more captured pupil reflections, a border between the pupil and an iris of the eye based on a change in reflected intensity of one or more wavelengths of light; and

identifying the shape of the pupil from the determined border.

11. The method of claim 8, wherein the optical action is selected from the group comprising:

determining a gaze direction of the user, determining a vergence angle of the eye of the user, determining the vergence depth of the user, determining an accommodation depth for the user, performing user identification, determining a torsional state of the eye of the user, or some combination thereof.

12. The method of claim 8, wherein the user is viewing content displayed by the HMD while the one or more images of reflections of the eye of the user reflected from the cornea of the eye of the user are reflected, and wherein the content includes a known calibration target displayed on an electronic display of the HMD.

13. The method of claim 8, wherein the pupillary axis derived by:

identifying a ray originating from the plane that is perpendicular to a surface of a corneal sphere of the eye of the user.

14. The method of claim 13, further comprising:

identifying a foveal centralis region of the eye of the user;

determining a second ray originating from the foveal centralis region of the eye and passing through the pupil corresponding to a true line of sight of the user; and

determining an angular offset between the determined pupillary axis for the pupil of the user and the true line of sight of the user.

15. A headset comprising:

an eye tracking system including two or more illumination sources configured to illuminate an eye of a user, the eye tracking system configured to;

capture one or more images of reflections of the eye of the user reflected from a cornea of the eye of the user;

determine, from the one or more images of the reflections, a shape of a pupil of the eye of the user;

determine a plane corresponding to a location of the pupil, the plane being identified using the determined shape of the pupil and a model of the eye of the user;

determine, from the determined shape of the pupil and the model of the eye of the user, a center of the corneal sphere; and

perform an optical action based in part on the determined center of the corneal sphere.

16. The headset of claim 15, wherein the model of the eye is generated by:

capturing an image of one or more reflections of the eye reflected from a cornea of the eye of the user;

determining a radius of a corneal sphere of the cornea based on the captured image of the reflections; and

determining a center of the corneal sphere based on the determined radius of the corneal sphere.

17. The headset of claim 15, wherein the optical action is selected from the group comprising:

determining a gaze direction of the user, determining a vergence angle of the eye of the user, determining the vergence depth of the user, determining an accommodation depth for the user, performing user identification, determining a torsional state of the eye of the user, or some combination thereof.

18. The headset of claim 15, wherein determining the shape of the pupil based on the one or more captured pupil reflections includes:

determining, from the one or more captured pupil reflections, a border between the pupil and an iris of the eye based on a change in reflected intensity of one or more wavelengths of light; and

identifying the shape of the pupil from the determined border.

19. The headset of claim 15, wherein the user is viewing content displayed by the headset while the one or more images of reflections of the eye of the user reflected from the cornea of the eye of the user are reflected, and wherein the content includes a known calibration target displayed on an electronic display of the headset.

20. The headset of claim 15, wherein the pupillary axis derived by:

identifying a ray originating from the plane that is perpendicular to a surface of the corneal sphere.