Method and apparatus for calibration-free eye tracking using multiple glints or surface reflections

• US 7,809,160 B2
• Filed: 11/15/2004
• Issued: 10/05/2010
• Est. Priority Date: 11/14/2003
• Status: Active Grant

• Current Assignee: Changuk Sohn, Daniel Cheng, Jeffrey S. Shell, Roel Vertegaal
• Original Assignee: Queen's University At Kingston
• Inventors: Cheng, Daniel, Vertegaal, Roel, Sohn, Changuk, Shell, Jeffrey S., Macfarlane, Victor
• Primary Examiner(s): Mariam; Daniel G
• Assistant Examiner(s): Woldemariam; Aklilu k
• Application Number:
  US10/987,299
• Publication Number:
  US 20050175218A1
• Time in Patent Office: 2,150 Days
• Field of Search: 382/103, 382/107, 351/209, 351/210, 345/8, 345/156, 345/157
• US Class Current: 382/103
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Apparatus and method for eye gaze tracking in human or animal subjects without calibration of cameras, specific measurements of eye geometries or the tracking of a cursor image on a screen by the subject through a known trajectory. One embodiment includes one uncalibrated camera for acquiring video images of the subject'"'"'s eye(s) and optionally having an on-axis illuminator, and a surface, object, or visual scene with embedded off-axis illuminator markers. The off-axis markers are reflected on the corneal surface of the subject'"'"'s eyes as glints. The glints indicate the distance between the point of gaze in the surface, object, or visual scene and the corresponding marker on the surface, object, or visual scene.

Claims

1. A method for determining a subject'"'"'s point of gaze, comprising:

providing an imaging device for acquiring images of at least one of a subject'"'"'s eyes;

providing one or more markers associated with a surface, object, or objects in a visual scene for producing one or more corresponding glints or reflections in the subject'"'"'s eyes;

analyzing the images to find glints that are within a threshold distance of the pupil center;

identifying, from the glints that are within a threshold distance of the pupil center, a glint that is closest to the pupil center; and

identifying the marker corresponding to the glint that is closest to the pupil center;

wherein the one or more markers are off an optical axis of the imaging device; and

wherein the identified marker is indicative of the subject'"'"'s point of gaze at the surface, object, or objects in the visual scene.

2. The method of claim 1, further comprising providing an illuminator for producing a glint in the subject'"'"'s eyes, the illuminator being substantially aligned on an optical axis of the imaging device.

3. The method of claim 2, further comprising:

acquiring images of the subject'"'"'s eyes, the images containing pupils and glints corresponding to at least one on-axis illuminator and at least one off-axis marker.

4. The method of claim 3, wherein the at least one off-axis glint consists of a reflection of at least a portion of the surface, object, or objects in the visual scene being viewed by the subject.

5. The method of claim 3, wherein:

acquiring images comprises acquiring alternate on-axis and off-axis images of at least one eye; and

analyzing comprises subjecting alternate on-axis and off-axis images to a rolling subtraction algorithm.

6. The method of claim 5, wherein, for an image sequence A, B, C, D, E, . . . , generated by successive image frames, the rolling subtraction algorithm comprises subtracting image frames as follows:

A-B, C-B, C-D, E-D, . . . .

7. The method of claim 1, wherein identifying a marker comprises comparing a position or pattern of one or more markers on the surface, object, or objects in the visual scene with a position or pattern of one or more corresponding glints, so that the marker is identified.

8. The method of claim 1, further comprising uniquely coding each marker in the visual scene.

9. The method of claim 1, further comprising arranging markers into groups, and uniquely coding each group of markers.

10. The method of claim 8, wherein identifying a marker comprises detecting a code of a marker or group of markers, so that the marker is identified.

11. The method of claim 8, wherein uniquely coding markers comprises using specific wavelengths for individual markers or groups of markers.

12. The method of claim 8, wherein uniquely coding markers comprises uniquely modulating light produced by individual markers or groups of markers.

17. The method of claim 1, wherein the subject is a shopper and the surface, object, or objects in the visual scene comprises at least one item on display.

18. The method of claim 17, further comprising:

determining whether location of the point of gaze is on the item; and

disclosing information about the item to the subject when the location of the point of gaze is or has been on the item.

19. The method of claim 18, wherein determining comprises determining duration of point of gaze on the item;

and disclosing depends on length of such duration.

20. The method of claim 18, further comprising disclosing information about location, duration, or location and duration of point of gaze on the item to a third party.

21. The method of claim 20, wherein the information is used to determine a cost of displaying the item.

22. The method of claim 1, wherein the visual scene comprises an electronic device, the method further comprising:

determining location, determining duration, or determining location and duration of point of gaze on the electronic device; and

initiating speech dialogue with the electronic device in accordance with the location, duration, or location and duration of point of gaze.

23. The method of claim 1, wherein the one visual scene comprises an electronic device, the method further comprising:

determining location, determining duration, or determining location and duration of point of gaze on the electronic device;

wherein the electronic device progressively discloses information in accordance with the location, duration, or location and duration of point of gaze.

24. The method of claim 1, wherein the visual scene comprises a video game or a robot, the method further comprising:

determining location, determining duration, or determining location and duration of point of gaze on the video game or on the robot; and

modulating an action of the game or robot in accordance with the location, duration, or location and duration of point of gaze.

25. The method of claim 1, wherein the visual scene comprises a device or appliance, the method further comprising:

determining location, determining duration, or determining location and duration of point of gaze on the device or appliance; and

routing information from a computer, or input device such as keyboard, mouse or remote control to the device or appliance in accordance with the location, duration, or location and duration of point of gaze.

26. The method of claim 1, wherein the visual scene comprises a graphical user interlace, the method further comprising:

determining duration of point of gaze on a location on the graphical user interface; and

controlling placement or arrangement of information on the graphical user interface in accordance with the duration of point of gaze on the location.

27. The method of claim 1, wherein the visual scene comprises a graphical user interface, the method further comprising:

detecting point of gaze of the subject and one or more additional subjects on the graphical user interface; and

modulating appearance of information on the graphical user interface when point of gaze of at least a second subject is detected.

28. The method of claim 27, wherein respective points of gaze of the subject and of the one or more additional subjects overlap.

29. The method of claim 27, wherein modulating appearance comprises positioning a lens or filter on the display according to the point(s) of gaze of the subject and/or the one or more additional subjects.

30. The method of claim 27, wherein modulating appearance comprises notifying the subject visually and/or aurally of detection of eyes or point of gaze of the one or more additional subjects.

31. The method of claim 1, wherein the visual scene comprises a noise-cancelling device, the method further comprising:

determining point of gaze on the noise-cancelling device; and

modulating noise cancelling by the device in accordance with the point of gaze.

32. The method of claim 1, wherein the visual scene comprises a communications device, the method further comprising:

determining location, determining duration, or determining location and duration of point of gaze on the communications device; and

modulating operation of the communications device in accordance with the location, duration, or location and duration of point of gaze.

33. The method of claim 1, wherein the visual scene comprises a device selected from a musical instrument, a loudspeaker, and a hearing aid, the method further comprising:

determining location, determining duration, or determining location and duration of point of gaze on the device; and

modulating volume of the device in accordance with the location, duration, or location and duration of point of gaze.

34. The method of claim 1, including providing two or more markers associated with the surface, object, or objects in the visual scene for producing two or more corresponding glints or reflections in the subject'"'"'s eyes.

13. Apparatus for determining a subject'"'"'s point of gaze, comprising:

an imaging device for acquiring images of at least one of a subject'"'"'s eyes;

one or more markers associated with a surface, object, or objects in a visual scene for producing one or more corresponding glints in the subject'"'"'s eyes; and

a means for analyzing the images to find glints that are within a threshold distance of the pupil center;

for identifying, from the glints that are within a threshold distance of the pupil center, the glint that is closest to the pupil center; and

for identifying the marker corresponding to the glint that is closest to the pupil center;

wherein the one or more markers are off an optical axis of the imaging device; and

wherein the identified marker is indicative of the subject'"'"'s point of gaze at the surface, object, or objects in the visual scene.

14. The apparatus of claim 13, further comprising an illuminator for producing a glint in the subject'"'"'s eyes, the illuminator being substantially aligned on an optical axis of the imaging device.

15. The apparatus of claim 13, wherein the imaging device is adapted to be worn by the user.

16. The apparatus of claim 15, further comprising a display unit to be worn by the user.

35. The apparatus of claim 13, including two or more markers associated with the surface, object, or objects in the visual scene for producing two or more corresponding glints in the subject'"'"'s eyes.