Method and apparatus for calibration-free eye tracking using multiple glints or surface reflections

• US 20050175218A1
• Filed: 11/15/2004
• Published: 08/11/2005
• Est. Priority Date: 11/14/2003
• Status: Active Grant

• Current Assignee: Changuk Sohn, Daniel Cheng, Jeffrey S. Shell, Roel Vertegaal
• Original Assignee: Queen's University At Kingston
• Inventors: Macfarlane, Victor, Cheng, Daniel, Vertegaal, Roel, Sohn, Changuk
• Granted Patent: US 7,809,160 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system and method for eye gaze tracking in human or animal subjects without calibration of cameras, specific measurements of eye geometries or the tracking of a cursor image on a screen by the subject through a known trajectory. The preferred embodiment includes one uncalibrated camera for acquiring video images of the subject'"'"'s eye(s) and optionally having an on-axis illuminator, and a surface, object, or visual scene with embedded off-axis illuminator markers. The off-axis markers are reflected on the corneal surface of the subject'"'"'s eyes as glints. The glints indicate the distance between the point of gaze in the surface, object, or visual scene and the corresponding marker on the surface, object, or visual scene. The marker that causes a glint to appear in the center of the subject'"'"'s pupil is determined to be located on the line of regard of the subject'"'"'s eye, and to intersect with the point of gaze. Point of gaze on the surface, object, or visual scene is calculated as follows. First, by determining which marker glints, as provided by the corneal reflections of the markers, are closest to the center of the pupil in either or both of the subject'"'"'s eyes. This subset of glints forms a region of interest (ROI). Second, by determining the gaze vector (relative angular or cartesian distance to the pupil center) for each of the glints in the ROI. Third, by relating each glint in the ROI to the location or identification (ID) of a corresponding marker on the surface, object, or visual scene observed by the eyes. Fourth, by interpolating the known locations of each these markers on the surface, object, or visual scene, according to the relative angular distance of their corresponding glints to the pupil center.

Claims

1. A method for eye gaze tracking, comprising:

providing an imaging device for acquiring images of at least one of a subject'"'"'s eyes;

providing one or more markers associated with a surface, object, or visual scene for producing corresponding glints or reflections in the subject'"'"'s eyes;

analyzing the images to find said glints and the center of the pupil; and

(i) identifying at least one marker corresponding to at least one glint that is within a threshold distance of the pupil center;

or

(ii) identifying at least two markers corresponding to at least two glints, and calculating a coordinate within the surface, object, or visual scene by interpolating between the location of the at least two markers on the surface, object, or visual scene according to the relative distance to the center of the pupil of each corresponding glint;

wherein the identified marker or interpolated coordinate is indicative of the subject'"'"'s point of gaze at the surface, object, or visual scene.

2. The method of claim 1, further comprising providing an illuminator for producing a glint in the cornea of the subject'"'"'s eyes, the illuminator being substantially aligned on an optical axis of the imaging device.

3. The method of claim 2, further comprising:

aligning the one or more markers off the optical axis of the imaging device; and

acquiring images of the subject'"'"'s cornea, the images containing pupils and glints corresponding to at least one on-axis illuminator and at least one off-axis marker.

4. The method of claim 3, wherein the at least one off-axis glint consists of a reflection of at least a portion of the surface, object, or visual scene being viewed by the subject.

5. The method of claim 3, wherein;

acquiring images comprises acquiring alternate on-axis and off-axis images of at least one eye; and

analyzing comprises subjecting alternate on-axis and off-axis images to a rolling subtraction algorithm.

6. The method of claim 5, wherein, for an image sequence A, B, C, D, E, . . . , generated by successive image frames, the rolling subtraction algorithm comprises subtracting image frames as follows:

A-B, C-B, C-D, E-D, . . . .

7. The method of claim 1, wherein identifying comprises comparing a position or pattern of one or more markers on the surface, object, or visual scene with a position or pattern of one or more corresponding glints, so that at least one unique marker on the surface, object, or visual scene is identified.

8. The method of claim 1, further comprising uniquely coding each marker in the visual scene.

9. The method of claim 1, further comprising arranging markers into groups, and uniquely coding each group of markers.

10. The method of claim 8, wherein identifying comprises detecting a code of a marker or group of markers, so that at least one unique marker on the surface, object, or visual scene is identified.

11. The method of claim 8, wherein uniquely coding markers comprises using specific wavelengths for individual markers or groups of markers.

12. The method of claim 8, wherein uniquely coding markers comprises uniquely modulating light produced by individual markers or groups of markers.

13. The method of claim 1, wherein identifying comprises determining a two-dimensional distance metric for the pupil center relative to a coordinate system provided by a position or pattern of the one or more markers.

14. The method of claim 1, wherein identifying comprises:

determining, for three markers, three glints closest to the pupil center in the images; and

triangulating between the location of the markers within the visual scene according to relative contributions of gaze vectors of each of said three glints.

15. The method of claim 1, wherein identifying comprises:

determining a region of interest (ROI) containing one or more glints closest to the center of the pupil;

determining a relative distance to the pupil center for each glint in the ROI;

relating each glint in the ROI to the location of a corresponding marker on the surface, object, or visual scene; and

interpolating known locations of each said corresponding marker on the surface, object, or visual scene according to the relative distance of its glint to the pupil center.

21. The method of claim 1, wherein the subject is a shopper and the surface, object, or visual scene comprises at least one item on display.

22. The method of claim 21, further comprising:

determining whether location of the point of gaze is on the item; and

disclosing information about the item to the subject when the location of the point of gaze is or has been on the item.

23. The method of claim 22, wherein determining comprises determining duration of point of gaze on an item;

and disclosing depends on length of such duration.

24. The method of claim 22, further comprising disclosing information about location and/or duration of point of gaze on an item to a third party.

25. The method of claim 24, wherein said information is used to determine a cost of displaying said item.

26. The method of claim 1, wherein the surface, object, or visual scene comprises an electronic device, the method further comprising:

determining location and/or duration of point of gaze on the electronic device; and

initiating speech dialogue with the electronic device in accordance with the location and/or duration of point of gaze.

27. The method of claim 1, wherein the surface, object, or visual scene comprises an electronic device, the method further comprising:

determining location and/or duration of point of gaze on the electronic device;

wherein the electronic device progressively discloses information in accordance with the location and/or duration of point of gaze.

28. The method of claim 1, wherein the surface, object, or visual scene comprises a video game or a robot, the method further comprising:

determining location and/or duration of point of gaze on an item of the video game or on the robot; and

modulating an action of the game or robot in accordance with the location and/or duration of point of gaze.

29. The method of claim 1, wherein the surface, object, or visual scene comprises a device or appliance, the method further comprising:

determining location and/or duration of point of gaze on the device or appliance; and

routing information from a computer, or input device such as keyboard, mouse or remote control to the device or appliance in accordance with the location and/or duration of point of gaze.

30. The method of claim 1, wherein the surface, object, or visual scene comprises a graphical user interface, the method further comprising:

determining duration of point of gaze on a location on the graphical user interface; and

controlling placement or arrangement of information on the graphical user interface in accordance with the duration of point of gaze on the location.

31. The method of claim 1, wherein the surface, object, or visual scene comprises a graphical user interface, the method further comprising:

detecting point of gaze of the subject and one or more additional subjects on the graphical user interface; and

modulating appearance of information on the graphical user interface when point of gaze of at least a second subject is detected.

32. The method of claim 31, wherein respective points of gaze of the subject and of the one or more additional subjects overlap.

33. The method of claim 31, wherein modulating appearance comprises positioning a lens or filter on the display according to the point(s) of gaze of the subject and/or the one or more additional subjects.

34. The method of claim 31, wherein modulating appearance comprises notifying the subject visually and/or aurally of detection of eyes or point of gaze of the one or more additional subjects.

35. The method of claim 1, wherein the surface, object, or visual scene comprises a noise-cancelling device, the method further comprising:

determining point of gaze on the noise-cancelling device; and

modulating noise cancelling by the device in accordance with the point of gaze.

36. The method of claim 1, wherein the surface, object, or visual scene comprises a communications device, the method further comprising:

determining location and/or duration of point of gaze on the communications device; and

modulating operation of the communications device in accordance with the location and/or duration of point of gaze.

37. The method of claim 1, wherein the surface, object, or visual scene comprises a device selected from a musical instrument, a loudspeaker, and a hearing aid, the method further comprising:

determining location and/or duration of point of gaze on the device; and

modulating volume of the device in accordance with the location and/or duration of point of gaze.

16. Apparatus for tracking eye gaze of a subject, comprising an imaging device for acquiring images of at least one of a subject'"'"'s eyes;

one or more markers associated with a surface, object, or visual scene for producing corresponding glints in the subject'"'"'s eyes; and

an analyzer of the images that finds said glints and the center of the pupil, and

(i) identifies at least one marker corresponding to at least one glint that is within a threshold distance of the pupil center;

or

(ii) identifies at least two markers corresponding to at least two glints, and calculates a coordinate within the surface, object, or visual scene by interpolating between the location of the at least two markers on the surface, object, or visual scene according to the relative distance to the center of the pupil of each corresponding glint;

wherein the identified marker or interpolated coordinate is indicative of the subject'"'"'s point of gaze at the surface, object, or visual scene.

17. The apparatus of claim 16, further comprising an illuminator for producing a glint in the subject'"'"'s eyes, the illuminator being substantially aligned on an optical axis of the imaging device.

18. The apparatus of claim 16, wherein the one or more markers are aligned off the optical axis of the imaging device.

19. The apparatus of claim 16, wherein the imaging device is adapted to be worn by the user.

20. The apparatus of claim 19, further comprising a display unit to be worn by the user.

38. A method for tracking eye gaze at a moving object, comprising:

acquiring images of at least one of a subject'"'"'s eyes;

detecting in the images movement of at least one glint in the subject'"'"'s eye;

correlating movement of the pupil of the eye with movement of the at least one glint; and

identifying the object by

(i) detecting a glint associated with the object that appears within a threshold distance from the pupil;

or

(ii) detecting a glint associated with the object that is moving at the same velocity as the pupil;

or

(iii) detecting a glint that is moving at the same velocity as the pupil and at the same velocity as the object.

39. The method of claim 38, further comprising providing one or more markers associated with the object.

40. The method of claim 39, further comprising modulating the one or more markers, wherein identifying further comprises demodulating a glint associated with the one or more markers.

41. (canceled)