Method and apparatus for calibration-free eye tracking

• US 7,963,652 B2
• Filed: 10/21/2005
• Issued: 06/21/2011
• Est. Priority Date: 11/14/2003
• Status: Active Grant

• Current Assignee: Changuk Sohn, Daniel Cheng, Humanmedialab Incorporated, Jeffrey S. Shell
• Original Assignee: Queen's University At Kingston
• Inventors: Cheng, Daniel, Sohn, Changuk, MacFarlane, Victor, Vertegaal, Roel, Shell, Jeffrey S.
• Primary Examiner(s): PINKNEY, DAWAYNE
• Application Number:
  US11/255,174
• Publication Number:
  US 20060110008A1
• Time in Patent Office: 2,069 Days
• Field of Search: 351/205, 351208-210
• US Class Current: 351/205
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A method and apparatus for eye gaze tracking in human or animal subjects without calibration of cameras, specific measurements of eye geometries, or tracking of a cursor image on a screen by the subject through a known trajectory. One embodiment provides a method for tracking a user'"'"'s eye gaze at a surface, object, or visual scene, comprising: providing an imaging device for acquiring images of at least one of the user'"'"'s eves: modeling, measuring, estimating, and/or calibrating for the user'"'"'s head position: providing one or more markers associated with the surface, object, or visual scene for producing corresponding glints or reflections in the user'"'"'s eyes; analyzing the images to find said glints or reflections and/or the pupil: and determining eye gaze of the user upon a said one or more marker as indicative of the user'"'"'s eye gaze at the surface, object, or visual scene.

Claims

1. A method for tracking a user'"'"'s eye gaze at a surface, object, or visual scene, comprising:

providing an imaging device for acquiring images of at least one of the user'"'"'s eyes;

modeling, measuring, estimating, and/or calibrating for the user'"'"'s head position;

providing one or more markers associated with the surface, object, or visual scene for producing corresponding glints or reflections in the user'"'"'s eyes;

analyzing the images to find said glints or reflections and/or the pupil; and

determining eye gaze of the user upon a said one or more marker without calibration relative to a said one or more marker, as indicative of the user'"'"'s eye gaze at the surface, object, or visual scene.

2. The method of claim 1, wherein modeling, measuring, estimating, and/or calibrating for the user'"'"'s head position is effected by adapting the imaging device to be worn by the user.

3. The method of claim 1, wherein analyzing comprises:

analyzing the images to find one or more glints that are within a threshold distance of the pupil center;

identifying, from the one or more glints that are within a threshold distance of the pupil center, one or more glints that are closest to the pupil center; and

identifying the one or more markers corresponding to the one or more glints that are closest to the pupil center;

wherein the one or more identified markers are indicative of the user'"'"'s point of gaze at the surface, object, or visual scene.

4. The method of claim 1, wherein modeling, measuring, estimating, and/or calibrating for the user'"'"'s head position is effected by face tracking.

5. The method of claim 4, wherein analyzing comprises:

identifying at least one pupil inside the user'"'"'s face;

analyzing the images to find one or more glints that are within a threshold distance of the pupil center;

identifying, from the one or more glints that are within a threshold distance of the pupil center, one or more glints that are closest to the pupil center; and

identifying the one or more markers corresponding to the one or more glints that are closest to the pupil center;

wherein the one or more identified markers are indicative of the user'"'"'s point of gaze at the surface, object, or visual scene.

6. The method of claim 5, wherein identifying at least one pupil comprises determining at least one region of interest of the image and locating a pupil in the image.

7. The method of claim 6, further comprising:

determining a region of interest (ROI) containing one or more glints closest to the center of the pupil;

determining a relative distance to the pupil center for each glint in the ROI;

relating each glint in the ROI to the location of a corresponding marker on the surface, object, or visual scene; and

interpolating known locations of each said corresponding marker on the surface, object, or visual scene according to the relative distance of its glint to the pupil center.

8. The method of claim 2, wherein identifying comprises:

determining a region of interest (ROI) containing one or more glints closest to the center of the pupil;

determining a relative distance to the pupil center for each glint in the ROI;

relating each glint in the ROI to the location of a corresponding marker on the surface, object, or visual scene; and

interpolating known locations of each said corresponding marker on the surface, object, or visual scene according to the relative distance of its glint to the pupil center.

9. The method of claim 1, further comprising providing an illuminator substantially aligned on an optical axis of the imaging device.

10. The method of claim 1, wherein the one or more markers are aligned off an optical axis of the imaging device.

11. The method of claim 1, further comprising:

acquiring images of the user'"'"'s cornea, the images containing pupils and glints corresponding to at least one on-axis illuminator and at least one off-axis marker.

12. The method of claim 3, wherein at least one marker consists of a reflection of at least a portion of the surface, object, or visual scene being viewed by the user.

13. The method of claim 1, wherein one or more marker emits infra-red (IR) light.

14. The method of claim 10, wherein analyzing comprises subjecting alternate on-axis and off-axis images to a rolling subtraction algorithm.

15. The method of claim 3, wherein, for an image sequence A, B, C, D, E, . . . , generated by successive image frames, the rolling subtraction algorithm comprises subtracting image frames as follows:

A-B, C-B, C-D, E-D, . . . .

16. The method of claim 3, wherein identifying comprises comparing a position or pattern of one or more markers on the surface, object, or visual scene with a position or pattern of one or more corresponding glints, so that at least one unique marker on the surface, object, or visual scene is identified.

17. The method of claim 3, further comprising uniquely coding each marker in the visual scene, or arranging markers into groups, and uniquely coding each group of markers.

18. The method of claim 17, wherein identifying comprises detecting a code of a marker or group of markers, so that at least one unique marker on the surface, object, or visual scene is identified.

19. The method of claim 17, wherein uniquely coding comprises using specific wavelengths for individual markers or groups of markers.

20. The method of claim 17, wherein uniquely coding comprises uniquely modulating light produced by individual markers or groups of markers.

21. The method of claim 20, wherein modulating light comprises pulsing light.

22. The method of claim 20, wherein modulating light comprises pulsing light according to a binary code.

23. The method of claim 20, wherein one or more markers transmit data to the user.

24. The method of claim 23, wherein the data comprises a uniform resource locator (URL).

25. The method of claim 1, further comprising disposing a marker or a group of markers on the user.

26. The method of claim 25, further comprising uniquely coding the marker or group of markers disposed on the user such that said marker provides information about the user.

27. The method of claim 1, wherein one or more markers are passively reflective and printed on or affixed to the surface, object, or visual scene.

28. The method of claim 17, wherein uniquely coding markers comprises disposing one or more markers in a unique spatial arrangement or pattern that identifies the surface, object, or visual scene associated with the one or more markers.

29. The method of claim 1, wherein the surface, object, or visual scene comprises at least one advertisement, product, or product display shelf.

30. The method of claim 1, further comprising:

determining whether the location of the point of gaze is on the surface, object, or visual scene; and

disclosing information about the surface, object, or visual scene to the user when the location of the gaze is or has been on the surface, object, or visual scene; and

/or

establishing communication between the user and the surface, object, or visual scene.

31. The method of claim 30, further comprising disclosing information about location and/or duration of point of gaze on a surface, object, or visual scene to a third party.

32. The method of claim 30, wherein said information is used to determine a cost of displaying said surface, object, or visual scene.

33. The method of claim 29, wherein said information is used for at least one of:

assessing interest in one or more advertisements; and

managing or modulating advertisements.

34. The method of claim 1, wherein the surface, object, or visual scene is selected from an appliance, an electronic device, an electronic display, a graphical user interface, a robot, a video game, a noise cancelling device, a communications device, a musical instrument, a loudspeaker, and a hearing aid, the method further comprising:

determining location and/or duration of point of gaze thereon; and

modulating operation of the surface, object, or visual scene in accordance with the location and/or duration of point of gaze.

35. Apparatus for tracking a user'"'"'s eye gaze at a surface, object, or visual scene, comprising:

an imaging device for acquiring images of at least one of the user'"'"'s eyes, the imaging device modeling, measuring, estimating, and/or calibrating for the user'"'"'s head position;

one or more markers associated with the surface, object, or visual scene for producing corresponding glints or reflections in the user'"'"'s eyes; and

means for analyzing the images to find said glints or reflections and/or the pupil and determining the user'"'"'s eye gaze upon a said one or more marker without calibration relative to a said one or more marker, as indicative of the user'"'"'s eye gaze at the surface, object, or visual scene.

36. The apparatus of claim 35, wherein modelling modeling, measuring, estimating, and/or calibrating for the user'"'"'s head position is effected by the imaging device being adapted to be worn by the user.

37. The apparatus of claim 35, further comprising an illuminator substantially aligned on an optical axis of the imaging device.

38. The apparatus of claim 35, wherein the one or more markers are aligned off the optical axis of the imaging device.

39. The apparatus of claim 35, wherein one or more markers or one or more groups of markers are uniquely coded.

40. The apparatus of claim 35, wherein one or more markers emits infra-red (IR) light.

41. The apparatus of claim 35, wherein one or more markers or one or more groups of markers comprises at least a portion of the object, surface, or visual scene.

42. The apparatus of claim 39, wherein uniquely coded markers or uniquely coded groups of markers emit specific wavelengths of light.

43. The apparatus of claim 39, wherein uniquely coded markers or uniquely coded groups of markers emit modulated light.

44. The apparatus of claim 43, wherein modulated light comprises pulsed light.

45. The apparatus of claim 44, wherein the modulated light is pulsed according to a binary code.

46. The apparatus of claim 43, wherein one or more markers or one or more groups of markers transmit data to the user.

47. The apparatus of claim 46, wherein the data comprise a uniform resource locator (URL).

48. The apparatus of claim 39, wherein uniquely coded markers comprise one or more markers in a unique spatial arrangement or pattern that identifies the surface, object, or visual scene associated with the one or more markers.

49. The apparatus of claim 35, further comprising a marker or group of markers disposed on the user.

50. The apparatus of claim 49, wherein the marker or group of markers disposed on the user is uniquely coded such that said marker or group of markers provides information about the user.

51. The apparatus of claim 35, further comprising a display unit adapted to be worn by the user.

52. A method for exchanging information and/or communicating among two or more users, comprising:

providing each user with the apparatus of claim 35;

further providing each user with means for receiving information and/or communicating;

wherein, upon eye gaze by a first user upon a second user, information about the second user is received by the first user.

53. The method of claim 52, wherein, upon eye gaze by a first user upon a second user, information is exchanged and/or communication is established between the first and second users.

54. The method of claim 53, wherein exchanging information and/or communicating is through one or more of instant messaging, video, hearing aid, headphones, radio device, speech recognition, social network, and keyboard.

55. The method of claim 54, where communicating includes a microphone associated with the second user or a surface, object, or visual scene, wherein gaze at a marker worn by the second user switches the hearing aid of the first user to the microphone associated with the second user, and/or modulates sound obtained from the microphone of the second user.

56. The method of claim 55, wherein microphones of additional users are excluded or attenuated.