Apparatus and method for determining eye gaze from stereo-optic views

• US 8,824,779 B1
• Filed: 08/11/2012
• Issued: 09/02/2014
• Est. Priority Date: 12/20/2011
• Status: Active Grant

• Current Assignee: Cortical DimensionsLLC
• Original Assignee: Christopher Charles Smyth
• Inventors: Smyth, Christopher Charles
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US13/572,650
• Time in Patent Office: 752 Days
• Field of Search: 382/100, 382/110, 382/115, 382/117, 382/154, 356/9, 356/12, 128/922
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

The invention, exemplified as a single lens stereo optics design with a stepped mirror system for tracking the eye, isolates landmark features in the separate images, locates the pupil in the eye, matches landmarks to a template centered on the pupil, mathematically traces refracted rays back from the matched image points through the cornea to the inner structure, and locates these structures from the intersection of the rays for the separate stereo views. Having located in this way structures of the eye in the coordinate system of the optical unit, the invention computes the optical axes and from that the line of sight and the torsion roll in vision. Along with providing a wider field of view, this invention has an additional advantage since the stereo images tend to be offset from each other and for this reason the reconstructed pupil is more accurately aligned and centered.

Claims

1. A system for tracking an eye, comprising:

an optical device with a light emitting source and an optical system, wherein the light source is configured for illuminating the eye and the optical system is configured for providing multiple-images of the eye;

an optoelectronic device for acquiring an image of the eye collected by the optical system;

an image processor for isolating reflections of source illumination in the image and for isolating features of the pupil and extra-pupillary eye structure in the image, wherein isolating features comprises determining the position of such in the image and categorizing as to structure type, where the features are at least those of a pupil;

a pupil-reconstructor for constructing a model of the pupil within the eye, wherein the model of the pupil is constructed from model parameters derived from pupil image features;

a feature-matcher for matching extra-pupillary image features to a common template located by the model of the pupil;

a feature-locater for locating matched features from multiple images in a coordinate system common to the optical device; and

a gaze processor for estimating the visual line-of sight for the eye in the said coordinate system;

wherein the multiple images are stereo-views about an optical axis of the system with centering on the eye-orbit.

2. The system of claim 1 wherein the optical system comprises a single-lens camera with separate light-collecting optics positioned to form stereo-pairs about said optical axis for the camera sensor.

3. The system of claim 1 wherein the optical system comprises a single-lens camera utilizing a concave mirror system for providing stereo views about said optical axis.

4. The system of claim 1 wherein the optical system comprises a single-lens camera with a lenticular array.

5. The system of claim 1, wherein the optical system comprises a single-lens stereo optics system utilizing one or more deformable segmented mirrors indexed between multiple positions for providing multiple views about said optical axis.

6. The mirror system of claim 5 wherein the mirrors are indexed by microelectro-mechanical actuators.

7. The system of claim 1, wherein the pupil-reconstructor is configured to:

construct a model of the cornea solid-body surface from the isolated reflections of source illumination;

determine a perimeter for a pupil image from isolated pupil image features;

locate a perimeter for the pupil within the eye from the perimeter for the pupil image using the model of the cornea; and

construct a model of the pupil within the eye from the pupil perimeter,

wherein said model construction of the pupil is derived from a fit of the pupil perimeter to the pupil modeled as located in a plane,

wherein the perimeter is used to locate and orient the pupil plane in the eye, and center and align the pupil model in the pupil plane of the eye, and

wherein said model parameters are relatable to pupil plane location, pupil plane orientation, pupil center in the plane.

8. The system of claim 7, wherein the pupil-reconstructor is configured to construct a model of the pupil within the eye from pupil perimeters for multiple images, wherein said model construction of the pupil is derived from a fit of the pupil perimeters to the pupil modeled as located in a plane.

9. The system of claim 1, wherein the feature-matcher is configured:

to locate the image of the eye relative to a reference frame aligned with the pupil model, wherein the reference frame comprises a template of extra-pupillary eye features;

determine the distribution of the isolated features in the image about the feature template of the pupil model reference frame; and

match the isolated features to the elements of the template, where for multiple images, conjugate and non-conjugate pairs of mapped features are formed in stereo image pairs.

10. The system of claim 1, wherein the feature-locator is configured to locate features of the eye in the coordinate system of the means for acquiring images that correspond to the matched image features, where the cornea solid body surface is located in the said coordinate system;

cornea surface points are located that correspond to the matched image features; and

eye features are located that correspond to the cornea surface points;

where for multiple images, conjugate paired features are located by the union of the locations of the eye feature for the separate images in stereo image pairs, and for non-conjugate paired features, the features are located for the separate images as single images by the eye feature location relative to a corresponding feature construct for the eye.

11. The system of claim 1, wherein the gaze processor is configured to determine a principal optical axis from the pupil model and orthogonal optical axes from the locations of the eye features relative to the pupil model, and a principal visual axis from the relation between the visual axis and the optical axes, where a bound is placed on the determination in a hierarchical layered manner by knowledge of source reflection points offsets from the pupil image centers, and from pupil image offsets in the sensor view.

12. The system of claim 1, wherein the image processor is configured to isolate the image of a face in an image collected by the optical system from the image surround, isolate the image of at least one eye in the face image, and isolate the eye-orbit in the eye image, for isolating features of the eye in the image, wherein the features are at least those of a pupil.

13. A method for tracking the eye, comprising:

illuminating the eye;

acquiring images, at the least a single image, of the eye;

isolating reflections of the means of illumination in an image;

isolating features of the pupil and extra-pupillary eye structure in the image, wherein isolating features comprises determining the position of such in the image and categorizing as to the nature, where the features are at least those of a pupil;

constructing a model of the pupil within the eye, wherein the model of the pupil is constructed from model parameters derived from pupil image features;

matching extra-pupillary image features to a common template located by the model of the pupil;

locating the matched features in a coordinate system common to the means of acquiring an image; and

determining the visual line-of sight for the eye in the said coordinate system.

14. The method of claim 13, wherein constructing a model of the pupil comprises:

constructing a model of the cornea solid-body surface from isolated reflections of source illumination;

determining a perimeter for a pupil image from isolated pupil-image features;

locating a perimeter for the pupil within the eye from the perimeter for the pupil image using the model of the cornea; and

constructing a model of the pupil within the eye from the pupil perimeter,

wherein said model construction of the pupil is derived from a fit of pupil perimeter to the pupil modeled as located in a plane,

wherein the perimeter is used to locate and orient the pupil plane in the eye, and center and align the pupil model in the pupil plane of the eye, and

wherein said model parameters are relatable to pupil plane location, pupil plane orientation, pupil center in the plane.

15. The method of claim 14, wherein constructing a model of the pupil comprises constructing a model of the pupil within the eye from pupil perimeters for multiple images, wherein said model construction of the pupil is derived from a fit of the pupil perimeters to the pupil modeled as located in a plane.

16. The method of claim 13, wherein matching features in images comprises:

locating the image relative to a reference frame aligned with the pupil model, wherein the reference frame comprises a template of extra-pupillary structure features;

determining the distribution of the isolated features in the image about the feature template of the pupil model reference frame; and

matching the features to the elements of template;

where for multiple images, conjugate and non-conjugate pairs of mapped features are formed in stereo image pairs.

17. The method of claim 16, where multiple images are acquired consecutively by an indexed means and motion stereo-optics is used for the determination of image element depth as an additional parameter for classification and template matching;

wherein the determination comprises;

tracking isolated feature elements across consecutive images as a function of the acquisition index, computing the apparent optical flow induced by the location displacement across consecutive images for tracked feature elements, and computing the apparent image depth of the features from the apparent optical flow using the varying viewpoint of the acquisition means as a basis;

where the computed image depth is used an additional parameter for matching the elements to the feature template.

18. The method of claim 13, wherein locating the matched features in the coordinate system of the means for acquiring images comprises:

locating the cornea solid body surface in the said coordinate system;

locating cornea surface points that correspond to the matched image features; and

locating eye features that correspond to the cornea surface points;

where for a single image, the features are located by the eye feature location relative to a corresponding feature construct for the eye, and where for multiple images, conjugate paired features are located by the union of the locations of the eye feature for the separate images in stereo image pairs, and for non-conjugate paired features, the features are located for the separate images as single images by the eye feature location relative to a corresponding feature construct for the eye.

19. The method of claim 13, wherein determining the visual line-of sight comprises:

determining a principal optical axis from the model of the pupil;

determining orthogonal optical axes from the locations of the eye features relative to the pupil model; and

determining a principal visual axis from a knowledge of the relation between the visual axis and the optical axes;

where a bound is placed on the determination in a hierarchical layered manner by knowledge of source reflection points offsets from the pupil image centers, and from pupil image offsets in the sensor view.

20. A method for tracking the eye with a hand-holdable device, comprising:

illuminating the face of a user;

acquiring multiple images of the face;

isolating an image of the face in an acquired image;

isolating at least one eye in the face image;

isolating the eye-orbit of the eye in the image;

isolating an eye image centered on the eye-orbit;

isolating reflections of the means of illumination in the eye image;

isolating features of the pupil and extra-pupillary eye structure in the image, wherein isolating features comprises determining the position of such in the image and categorizing as to the nature, where the features are at least those of a pupil;

constructing a model of the pupil within the eye, wherein the model of the pupil is constructed from model parameters derived from pupil image features;

matching extra-pupillary image features to a common template located by the model of the pupil;

locating the matched features in a coordinate system common to the means of acquiring images; and

determining the visual line-of sight for the eye in the said coordinate system;

wherein the multiple images are stereo-views about an optical axis of light collecting optics incorporated with the hand-holdable device.