Systems and methods for identifying gaze tracking scene reference locations

• US 8,885,877 B2
• Filed: 05/20/2011
• Issued: 11/11/2014
• Est. Priority Date: 05/20/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Eyefluence, Inc. (Alphabet Inc.)
• Inventors: Publicover, Nelson G., Torch, William C., Amayeh, Gholamreza, Leblanc, David
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US13/113,003
• Publication Number:
  US 20120294478A1
• Time in Patent Office: 1,271 Days
• Field of Search: 382/103, 382/106, 348/169, 348/333.03, 715/863, 396/51
• US Class Current: 382/103
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A system is provided for identifying reference locations within the environment of a device wearer. The system includes a scene camera mounted on eyewear or headwear coupled to a processing unit. The system may recognize objects with known geometries that occur naturally within the wearer'"'"'s environment or objects that have been intentionally placed at known locations within the wearer'"'"'s environment. One or more light sources may be mounted on the headwear that illuminate reflective surfaces at selected times and wavelengths to help identify scene reference locations and glints projected from known locations onto the surface of the eye. The processing unit may control light sources to adjust illumination levels in order to help identify reference locations within the environment and corresponding glints on the surface of the eye. Objects may be identified substantially continuously within video images from scene cameras to provide a continuous data stream of reference locations.

Claims

1. A system for determining reference locations, comprising:

a device configured to be worn on a wearer'"'"'s head;

a scene camera mounted on the device oriented away from the wearer for capturing scene camera images of the environment of the wearer;

an eye-tracking camera mounted on the device oriented towards an eye of the wearer for capturing eye-tracking locations of at least one of the wearer'"'"'s eyes;

one or more processors coupled to the scene camera for determining scene reference locations within the scene camera images and coupled to the eye-tracking camera, the one or more processors using the scene reference locations and the eye-tracking locations to determine locations being viewed by the wearer; and

a database of templates mapping known objects with scene reference locations associated with respective known objects, the one or more processors coupled to the database for identifying a display of an electronic device associated with the scene reference locations acquired from the scene images, the scene reference locations identifying reference objects fixed relative to the display.

2. The system of claim 1, wherein the eye-tracking camera comprises a first eye-tracking camera for capturing eye-tracking locations of a first eye of the wearer, and a second eye-tracking camera for capturing eye-tracking locations of a second eye of the wearer.

3. The system of claim 1, wherein object recognition algorithms are used by the one or more processors to the scene reference locations.

4. The system of claim 3, wherein object recognition is based at least in part on a shape of an object and a color of an object.

5. The system of claim 3, wherein the one or more processors configured to identify the reference fixed objects at the corners of the display.

6. The system of claim 1, further comprising one or more light sources on the device and oriented away from the wearer to illuminate the environment of the wearer.

7. The system of claim 6, wherein the one or more light sources comprise one or more light-emitting diodes (LEDs).

8. The system of claim 6, wherein the light sources are configured to illuminate reflective reference objects within the environment of the wearer such that the additional reflective reference objects are recognizable within the scene camera images.

9. The system of claim 6, further comprising a controller coupled to the one or more light sources to modulate the one or more light sources to desired brightness levels.

10. A system for gaze tracking, comprising:

an electronic object comprising a display and a plurality of reflective reference objects located around the display; and

a device configured to be worn on a wearer'"'"'s head, the device comprising;

a) a scene camera mounted on the device oriented away from the wearer for capturing scene camera images of the environment of the wearer;

b) an eye-tracking camera mounted on the device oriented towards an eye of the wearer for capturing eye-tracking locations of at least one of the wearer'"'"'s eyes; and

c) one or more processors coupled to the scene camera for identifying the reference objects within the scene camera images and coupled to the eye-tracking camera, the one or more processors using the locations of the reference objects in the scene camera images and the eye-tracking locations to determine locations on the display being viewed by the wearer; and

a database of templates mapping known objects with reference objects associated with respective known objects, the one or more processors coupled to the database for identifying the electronic object based on the reference objects identified in the scene camera images.

11. The system of claim 10, wherein the device further comprises one or more light sources on the device oriented away from the wearer for illuminating the reference objects to enhance identification of the reference objects in the scene camera images.

12. The system of claim 11, wherein the one or more processors comprise a controller coupled to the one or more light sources for modulating the one or more light sources to desired brightness levels.

13. A system for gaze tracking, comprising:

an electronic object comprising a display and a plurality of reflective reference objects located around the display; and

a device configured to be worn on a wearer'"'"'s head, the device comprising;

a) a scene camera mounted on the device oriented away from the wearer for capturing scene camera images of the environment of the wearer;

b) an eye-tracking camera mounted on the device oriented towards an eye of the wearer for capturing eye-tracking locations of at least one of the wearer'"'"'s eyes;

c) one or more processors coupled to the scene camera for identifying the reference objects within the scene camera images and coupled to the eye-tracking camera, the one or more processors using the locations of the reference objects in the scene camera images and the eye-tracking locations to determine locations on the display being viewed by the wearer; and

d) one or more light sources on the device oriented away from the wearer for illuminating the reference objects to enhance identification of the reference objects in the scene camera images,

wherein the one or more processors comprise a controller coupled to the one or more light sources for operating the one or more light sources intermittently such that the reference objects are alternately illuminated and unilluminated in the scene camera images, the one or more processors configured for subtracting illuminated images including the reference objects from corresponding unilluminated images including the reference objects acquired by the scene camera images to enhance identification of the reference objects.

14. A system for gaze tracking, comprising:

an electronic object comprising a display, the electronic object configured for including a plurality of virtual reference objects in images presented on the display; and

a device configured to be worn on a wearer'"'"'s head, the device comprising;

a) a scene camera mounted on the device oriented away from the wearer for capturing scene camera images of the environment of the wearer;

b) an eye-tracking camera mounted on the device oriented towards an eye of the wearer for capturing eye-tracking locations of at least one of the wearer'"'"'s eyes; and

c) one or more processors coupled to the scene camera for identifying the virtual reference objects within the scene camera images and coupled to the eye-tracking camera, the one or more processors using the locations of the virtual reference objects in the scene camera images and the eye-tracking locations to determine locations on the display being viewed by the wearer; and

a database of templates mapping known objects with reference objects associated with respective known objects, the one or more processors coupled to the database for identifying the electronic object based on the virtual reference objects identified in the scene camera images.