SYSTEMS AND METHODS FOR IDENTIFYING GAZE TRACKING SCENE REFERENCE LOCATIONS

• US 20120294478A1
• Filed: 05/20/2011
• Published: 11/22/2012
• Est. Priority Date: 05/20/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Eye-Com Corporation
• Inventors: Publicover, Nelson G., Torch, William C., Amayeh, Gholamreza, Leblanc, David
• Granted Patent: US 8,885,877 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A system is provided for identifying reference locations within the environment of a device wearer. The system includes a scene camera mounted on eyewear or headwear coupled to a processing unit. The system may recognize objects with known geometries that occur naturally within the wearer'"'"'s environment or objects that have been intentionally placed at known locations within the wearer'"'"'s environment. One or more light sources may be mounted on the headwear that illuminate reflective surfaces at selected times and wavelengths to help identify scene reference locations and glints projected from known locations onto the surface of the eye. The processing unit may control light sources to adjust illumination levels in order to help identify reference locations within the environment and corresponding glints on the surface of the eye. Objects may be identified substantially continuously within video images from scene cameras to provide a continuous data stream of reference locations.

Claims

1. A system for determining reference locations, comprising:

a device configured to be worn on a wearer'"'"'s head;

a scene camera mounted on the device oriented away from the wearer for capturing scene camera images of the environment of the wearer;

an eye-tracking camera mounted on the device oriented towards an eye of the wearer for capturing eye-tracking locations of at least one of the wearer'"'"'s eyes; and

one or more processors coupled to the scene camera for determining scene reference locations within the scene camera images and coupled to the eye-tracking camera, the one or more processors using the scene reference locations and the eye-tracking locations to determine locations being viewed by the wearer.

4. The system of claim 1, wherein the eye-tracking camera comprises a first eye-tracking camera for capturing eye-tracking locations of a first eye of the wearer, and a second eye-tracking camera for capturing eye-tracking locations of a second eye of the wearer.

5. The system of claim 1, wherein object recognition algorithms are used by the one or more processors to identify scene reference locations within the field-of-view.

6. The system of claim 5, wherein object recognition is based at least in part on a shape of an object, a color of an object, and at least one edge of an object.

7. The system of claim 1, further comprising a database of templates mapping known objects with scene reference locations associated with respective known objects, the one or more processors coupled to the database for identifying an object associated with the scene reference locations acquired from the scene images.

8. The system of claim 7, wherein the object comprises a display of an electronic device, the scene reference locations identifying reference objects fixed relative to the display.

9. The system of claim 5, wherein the one or more processors use object recognition to identify a location of at least one of the corners of a display device.

10. The system of claim 1, further comprising additional reference objects affixed to an object within a field of view of the scene camera, the object located within the scene camera images such that these additional reference objects are recognized by the one or more processors.

16. The system of claim 1, further comprising one or more light sources on the device and oriented away from the wearer to illuminate the environment of the wearer.

17. The system of claim 16, wherein the one or more light sources comprise one or more light-emitting diodes (LEDs).

18. The system of claim 16, wherein the light sources are configured to illuminate reflective reference objects within the environment of the wearer such that the additional reflective reference objects are recognizable within the scene camera images.

19. The system of claim 16, further comprising a controller coupled to the one or more light sources to modulate the one or more light sources to desired brightness levels.

2-3. -3. (canceled)

11-15. -15. (canceled)

20-21. -21. (canceled)

22. A system for gaze tracking, comprising:

an electronic object comprising a display and a plurality of reflective reference objects located around the display; and

a device configured to be worn on a wearer'"'"'s head, the device comprising;

a) a scene camera mounted on the device oriented away from the wearer for capturing scene camera images of the environment of the wearer;

b) an eye-tracking camera mounted on the device oriented towards an eye of the wearer for capturing eye-tracking locations of at least one of the wearer'"'"'s eyes; and

c) one or more processors coupled to the scene camera for identifying the reference objects within the scene camera images and coupled to the eye-tracking camera, the one or more processors using the locations of the reference objects in the scene camera images and the eye-tracking locations to determine locations on the display being viewed by the wearer.

23. The system of claim 22, wherein the device further comprises one or more light sources on the device oriented away from the wearer for illuminating the reference objects to enhance identification of the reference objects in the scene camera images.

24. The system of claim 23, wherein the one or more processors comprise a controller coupled to the one or more light sources for modulating the one or more light sources to desired brightness levels.

25. The system of claim 23, wherein the one or more processors comprise a controller coupled to the one or more light sources for operating the one or more light sources intermittently such that the reference objects are alternately illuminated and unilluminated in the scene camera images, the one or more processors configured for subtracted illuminated and unilluminated images from the scene camera images to enhance identification of the reference objects.

26. The system of claim 22, further comprising a database of templates mapping known objects with reference objects associated with respective known objects, the one or more processors coupled to the database for identifying the electronic object based on the reference objects identified in the scene camera images.

27-28. -28. (canceled)

29. A system for gaze tracking, comprising:

an electronic object comprising a display, the electronic object configured for including a plurality of virtual reference objects in images presented on the display; and

a device configured to be worn on a wearer'"'"'s head, the device comprising;

a) a scene camera mounted on the device oriented away from the wearer for capturing scene camera images of the environment of the wearer;

b) an eye-tracking camera mounted on the device oriented towards an eye of the wearer for capturing eye-tracking locations of at least one of the wearer'"'"'s eyes; and

c) one or more processors coupled to the scene camera for identifying the virtual reference objects within the scene camera images and coupled to the eye-tracking camera, the one or more processors using the locations of the virtual reference objects in the scene camera images and the eye-tracking locations to determine locations on the display being viewed by the wearer.

30. The system of claim 29, further comprising a database of templates mapping known objects with reference objects associated with respective known objects, the one or more processors coupled to the database for identifying the electronic object based on the virtual reference objects identified in the scene camera images.

31-49. -49. (canceled)

50. A method for determining reference locations, comprising:

placing a device on a wearer'"'"'s head, the device comprising a scene camera oriented away from the wearer and an eye-tracking camera oriented towards at least one of the wearer'"'"'s eyes;

acquiring scene images of the environment around the wearer with the scene camera;

identifying scene reference locations within the scene camera images to identify an object associated with the scene reference locations within the environment; and

analyzing the scene camera images and the eye-tracking images to determine locations being viewed by the wearer relative to the object.