EYE TRACKING VIA DEPTH CAMERA

• US 20140375541A1
• Filed: 06/25/2013
• Published: 12/25/2014
• Est. Priority Date: 06/25/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Nister, David, Eden, Ibrahim
• Application Number:
  US13/926,223
• Publication Number:
  US 20140375541A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

Embodiments are disclosed that relate to tracking a user'"'"'s eye based on time-of-flight depth image data of the user'"'"'s eye are disclosed. For example, one disclosed embodiment provides an eye tracking system comprising a light source, a sensing subsystem configured to obtain a two-dimensional image of a user'"'"'s eye and depth data of the user'"'"'s eye using a depth sensor having an unconstrained baseline distance, and a logic subsystem configured to control the light source to emit light, control the sensing subsystem to acquire a two-dimensional image of the user'"'"'s eye while illuminating the light source, control the sensing subsystem to acquire depth data of the user'"'"'s eye, determine a gaze direction of the user'"'"'s eye from the two-dimensional image, determine a location on a display at which the gaze direction intersects the display based on the gaze direction and the depth data, and output the location.

Claims

1. An eye tracking system, comprising:

a light source;

an image sensing subsystem configured to obtain a two-dimensional image of a user'"'"'s eye and time-of-flight depth image data of a region that contains the user'"'"'s eye;

a logic subsystem configured to

control the light source to emit light;

control the image sensing subsystem to acquire a two-dimensional image of the user'"'"'s eye while emitting light via the light source;

control the image sensing subsystem to acquire a time-of-flight depth image of the user'"'"'s eye;

determine a gaze direction of the user'"'"'s eye from the two-dimensional image;

determine a location at which the gaze direction intersects the display based on the gaze location; and

output the location.

2. The system of claim 1, wherein the image sensing subsystem comprises a time-of-flight depth camera and a two-dimensional image sensor.

3. The system of claim 1, wherein the image sensing subsystem comprises a time-of-flight depth camera, and wherein the instructions are executable to detect a location of a pupil of the user'"'"'s eye from image data acquired by the time-of-flight depth camera to determine the gaze direction of the user'"'"'s eye.

4. The system of claim 1, wherein the system further comprises the display.

5. The system of claim 1, wherein the image sensing subsystem comprises a time-of-flight depth camera and the light source comprises a light source of the time-of-flight depth camera.

6. The system of claim 1, wherein the instructions are executable to detect a distance from the user'"'"'s eye to the display along the gaze direction from the time-of-flight depth image to determine the location on the display at which the gaze direction intersects the display.

7. The system of claim 1, wherein the two-dimensional image is a first two-dimensional image, and wherein the instructions are further executable to:

control the image sensing subsystem to acquire a second two-dimensional image, the second two-dimensional image having a wider field of view than the first two-dimensional image, and

determine via the second two-dimensional image a location of the user'"'"'s eye before determining the gaze direction of the user'"'"'s eye from the first two-dimensional image.

8. The system of claim 7, wherein the image sensing subsystem comprises a time-of-flight depth camera, a higher resolution two-dimensional image sensor, and a lower resolution two-dimensional image sensor, and wherein the second two-dimensional image is acquired via the lower resolution two-dimensional image sensor and the first two-dimensional image is acquired via the higher resolution two-dimensional image sensor.

9. An eye tracking module, comprising:

a time-of-flight camera;

a light source;

a logic subsystem; and

a storage subsystem comprising instructions stored thereon that are executable by the logic subsystem to;

illuminate the light source;

acquire image data including an image of a user'"'"'s eye while illuminating the light source and a time-of-flight depth image of the user'"'"'s eye;

detect a location of a pupil of the user'"'"'s eye and a location of a reflection in the user'"'"'s eye from the image data;

determine a gaze direction of the user'"'"'s eye from the location of the pupil and the location of the reflection; and

output a location on a display at which the gaze direction intersects the display based on the gaze direction and the time-of-flight depth image.

10. The module of claim 9, wherein the location of the pupil is detected via image data acquired by the time-of-flight image sensor.

11. The module of claim 9, further comprising a two-dimensional image sensor, and wherein the location of the pupil is detected via image data acquired via the two-dimensional image sensor.

12. The module of claim 9, wherein the module is coupled to a display device.

13. The module of claim 9, wherein the instructions are further executable to acquire an image of the user and determine via the image of the user a location of a region of a user containing the user'"'"'s eye before determining the gaze direction of the user'"'"'s eye.

14. The module of claim 9, wherein the body comprises a body of a mobile computing device.

15. The module of claim 9, wherein the body comprises a body of a wearable computing device.

16. On a mobile computing device, a method for tracking an eye of a user relative to a user interface displayed on a display, the method comprising illuminating a light source;

acquiring image data including an image of the eye while illuminating the light source;

acquiring depth data of the eye via a depth sensor having an unconstrained baseline distance;

detecting a location of a pupil of the eye and a location of a reflection of light from the light source on the eye from the image data;

determining a gaze direction of the eye from the location of the pupil and the location of the reflection;

detecting a distance from the eye to the display along the gaze direction from the depth data; and

outputting a location at which the gaze direction intersects the display.

17. The method of claim 16, wherein the depth sensor comprises a time-of-flight depth camera, and wherein the location of the pupil and the location of the reflection are detected via image data from the time-of-flight depth camera.

18. The method of claim 16, wherein the light source comprises a light source in a time-of-flight depth camera.

19. The method of claim 16, further comprising determining via the image data a location of the eye before determining the gaze direction of the eye.

20. The method of claim 16, wherein the image data is acquired from a time-of-flight depth camera.