Systems and methods for high-resolution gaze tracking

• US 8,929,589 B2
• Filed: 11/07/2011
• Issued: 01/06/2015
• Est. Priority Date: 11/07/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Eyefluence, Inc. (Alphabet Inc.)
• Inventors: Publicover, Nelson G., Torch, William C., Spitler, Christopher N.
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US13/290,948
• Publication Number:
  US 20130114850A1
• Time in Patent Office: 1,156 Days
• Field of Search: 382/100, 382/103, 382/117, 348/78, 345/8, 345/156, 345/157, 345/158, 351/209, 351/210
• US Class Current: 382/103
• CPC Class Codes: A61B 3/0025 characterised by electronic signal processing, e.g. eye models
• CPC Class Codes: A61B 3/024 for determining the visual field, e.g. perimeter types
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A system is mounted within eyewear or headwear to unobtrusively produce and track reference locations on the surface of one or both eyes of an observer. The system utilizes multiple illumination sources and/or multiple cameras to generate and observe glints from multiple directions. The use of multiple illumination sources and cameras can compensate for the complex, three-dimensional geometry of the head and anatomical variations of the head and eye region that occurs among individuals. The system continuously tracks the initial placement and any slippage of eyewear or headwear. In addition, the use of multiple illumination sources and cameras can maintain high-precision, dynamic eye tracking as an eye moves through its full physiological range. Furthermore, illumination sources placed in the normal line-of-sight of the device wearer increase the accuracy of gaze tracking by producing reference vectors that are close to the visual axis of the device wearer.

Claims

1. A system for determining gaze-tracking locations, comprising:

a device configured to be worn on a wearer'"'"'s head;

one or more illumination sources mounted on the device within a field of view of an eye of a wearer wearing the device and oriented to illuminate the eye, the one or more illumination sources configured to minimize interference with the wearer'"'"'s vision within the field of view;

an eye-tracking camera mounted on the device and positioned for viewing the eye; and

a processor coupled to the eye-tracking camera for analyzing images of the eye acquired by the eye-tracking camera to identify one or more glints reflected off the eye from the one or more illumination sources and identify the location of features of the eye relative to the one or more glints to determine locations being viewed by the wearer;

further comprising one or more additional eye-tracking cameras mounted on the device and positioned for viewing the eye, the processor coupled to the one or more additional eye-tracking cameras to track locations of the eye from different viewing angles, wherein the eye-tracking cameras are sufficiently spaced apart from one another such that the features of the eye appear in eye-tracking camera images from at least one of the eye-tracking cameras when the eye moves through its complete normal range of motion.

2. The system of claim 1, wherein the one or more illumination sources comprise one of a LED and OLED emitter.

3. The system of claim 1, wherein the device comprises a lens mounted thereto within the line of sight of the wearer, and wherein the one or more illumination sources comprise a plurality of light sources provided on the lens to illuminate the eye.

4. The system of claim 1, wherein the illumination source comprises a plurality of reflective surfaces embedded in a lens mounted to the device in front of the wearer'"'"'s eye, and one or more light sources configured to direct light into the lens to be reflected from the plurality of reflective surfaces towards the eye.

5. The system of claim 1, further comprising a controller coupled to one or more light sources of the illumination source to modulate the brightness levels of the one or more light sources to provide desired brightness levels within a field-of-view of the eye-tracking camera.

6. The system of claim 5, wherein the controller is configured to illuminate a single light source to produce a single glint, and wherein the processor is configured for isolating the single glint in the gaze-tracking camera images.

7. The system of claim 1, wherein the eye-tracking cameras comprise one of a CCD and CMOS detector.

8. The system of claim 1, wherein the eye-tracking camera comprises an active area for generating eye-tracking camera images, and wherein the active area surface is curved.

9. The system of claim 1, further comprising a scene camera mounted on the device and positioned for viewing reference locations in the environment of the wearer, the processor coupled to the scene camera for acquiring scene images of the field of view and correlating the location of the features of the eye relative to the one or more glints to determine locations being viewed by the wearer within the scene images.

10. The system of claim 1, wherein the features of the eye comprise the eye'"'"'s pupil.

11. A system for determining gaze-tracking locations, comprising:

a device configured to be worn on a wearer'"'"'s head;

one or more illumination sources mounted on the device oriented to illuminate an eye of the wearer;

a plurality of eye-tracking cameras mounted on the device and positioned for viewing the eye; and

a processor coupled to the eye-tracking cameras for analyzing images of the eye acquired by the eye-tracking cameras to identify one or more glints reflected off the eye from the one or more illumination sources and identify the location of features of the eye relative to the one or more glints to determine locations being viewed by the wearer,

wherein the eye-tracking cameras are sufficiently spaced apart from one another such that the eye'"'"'s pupil appears in eye-tracking camera images from at least one of the eye-tracking cameras when the eye moves through its complete normal range of motion.

12. The system of claim 11, further comprising a scene camera mounted on the device and positioned for viewing reference locations in the environment of the wearer, the processor coupled to the scene camera for acquiring scene images of the field of view and correlating the location of features of the eye relative to the one or more glints to determine locations being viewed by the wearer within the scene images.

13. A method for gaze tracking, comprising:

placing a device on a wearer'"'"'s head, the device comprising one or more illumination sources and a plurality of eye-tracking cameras oriented towards an eye of the wearer, the eye-tracking cameras positioned at different locations on the device to provide overlapping fields-of-view of the eye; and

analyzing images of the eye acquired by the eye-tracking cameras to identify one or more glints reflected off the eye from the one or more illumination sources and identify the location of features of the eye relative to the one or more glints to determine locations being viewed by the wearer,

wherein the location of the features of the eye is monitored from the images of the eye-tracking cameras to identify an eye-tracking camera in which the features of the eye are closest to the center of the field-of-view relative to the other eye-tracking cameras, and wherein analyzing images of the eye comprises analyzing images from the identified eye-tracking camera to determine locations being viewed by the wearer.

14. The method of claim 13, further comprising periodically identifying an eye-tracking camera in which the features of the eye are closest to the center of the field-of-view relative to the other eye-tracking cameras, and comprises analyzing images from the identified eye-tracking camera to determine locations being viewed by the wearer.

15. The system of claim 13, wherein the features of the eye comprise the eye'"'"'s pupil.

16. A method for gaze tracking, comprising:

placing a device on a wearer'"'"'s head, the device comprising one or more illumination sources and a plurality of eye-tracking cameras oriented towards an eye of the wearer, the eye-tracking cameras positioned at different locations on the device to provide overlapping fields-of-view of the eye; and

analyzing images of the eye acquired by the eye-tracking cameras to identify one or more glints reflected off the eye from the one or more illumination sources and identify the location of features of the eye relative to the one or more glints to determine locations being viewed by the wearer,

wherein the location of the features of the eye is monitored from the images of the eye-tracking cameras, and wherein analyzing images of the eye comprises averaging the location of the features of the eye in the images to determine locations being viewed by the wearer.

17. The method of claim 16, further comprising applying corrective spatial mapping functions to the images to accommodate for at least one of spherical aberration and curvature of the eye.

18. The method of claim 16, wherein the location of the features of the eye from the images of the eye-tracking cameras is weighted prior to averaging to accommodate for at least one of spherical aberration and curvature of the eye.