Eye-tracking enabled wearable devices

• US 9,977,960 B2
• Filed: 09/26/2016
• Issued: 05/22/2018
• Est. Priority Date: 09/24/2015
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Gustafsson, Simon, Johansson Tornus, Daniel, Andersson, Jonas, Kron, Johannes, Skog, Mrten, Biedert, Ralf
• Primary Examiner(s): Moon, Seokyun
• Application Number:
  US15/276,448
• Publication Number:
  US 20170091549A1
• Time in Patent Office: 603 Days
• Field of Search: None
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G09G 2340/0464 Positioning
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/34 for rolling or scrolling

Abstract

A system for determining a gaze direction of a user is disclosed. The system may include a first illuminator, a first profile sensor, and at least one processor. The first illuminator may be configured to illuminate an eye of a user. The first profile sensor may be configured to detect light reflected by the eye of the user. The processor(s) may be configured to determine a gaze direction of the user based at least in part on light detected by the first profile sensor.

Claims

1. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user;

a first profile sensor configured to detect light reflected by the eye of the user;

an area image sensor configured to detect light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns;

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the first profile sensor and based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the first profile sensor comprises;

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the first profile sensor;

identifying a subset of a total number of pixels of the first profile sensor which include and surround the one or more pixels of the first profile sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the first profile sensor during a subsequent determination of gaze direction; and

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises;

determining magnitudes of light exposure at a subset of the total number of rows of the sensitive area during one or more illumination periods;

determining magnitudes of light exposure at a subset of the total number of columns of the sensitive area during one or more illumination periods;

identifying a subset of the total number of pixels of the sensitive area based on the magnitudes of light exposure at the subset of the total number of rows and the magnitudes of light exposure at the subset of the total number of columns; and

analyzing only the subset of the total number of pixels of the sensitive area during a subsequent illumination period.

2. The system for determining a gaze direction of a user of claim 1, wherein the system further comprises:

a second profile sensor configured to detect light reflected by the eye of the user, wherein determining the gaze direction of the user is further based at least in part on light detected by the second profile sensor.

3. The system for determining a gaze direction of a user of claim 2, wherein the at least one processor is further configured to:

determine a location of a cornea center of the eye based at least in part on light detected by the first profile sensor and light detected by the second profile sensor.

4. The system for determining a gaze direction of a user of claim 1, wherein the first profile sensor is configured to:

detect light in two dimensions.

5. The system for determining a gaze direction of a user of claim 2, wherein:

the first profile sensor is configured to detect light in one dimension; and

the second profile sensor is configured to detect light in one dimension.

6. The system for determining a gaze direction of a user of claim 5, wherein:

the first profile sensor is orientated orthogonally to the second profile sensor.

7. The system for determining a gaze direction of a user of claim 5, wherein the system further comprises:

a first cylindrical lens between the first profile sensor and the eye; and

a second cylindrical lens between the second profile sensor and the eye.

8. The system for determining a gaze direction of a user of claim 1, wherein the system further comprises:

a second illuminator configured to illuminate the eye of the user.

9. The system for determining a gaze direction of a user of claim 8, wherein the at least one processor is further configured to:

modulate activation of the first illuminator and the second illuminator such that only the first illuminator or the second illuminator is activated at a particular point in time.

10. The system for determining a gaze direction of a user of claim 1, wherein:

the area image sensor operates at between about 0.5 and about 10.0 hertz.

11. A method for determining a gaze direction of a user, wherein the method comprises:

illuminating an eye of a user;

detecting, with an area image sensor, light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns;

determining, with at least one processor, a gaze direction of the user based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises;

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the area image sensor;

determining magnitudes of light exposure at a subset of the total number of rows during one or more illumination periods;

determining magnitudes of light exposure at a subset of the total number of columns during one or more illumination periods;

based on the magnitudes of light exposure at the subset of the total number of rows and the magnitudes of light exposure at the subset of the total number of columns, identifying a subset of a total number of pixels of the area image sensor which include and surround the one or more pixels of the area image sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the area image sensor during a subsequent determination of gaze direction during a subsequent illumination period.

12. A non-transitory machine readable medium having instructions stored thereon for determining a gaze direction of a user, wherein the instructions are executable by one or more processors to at least:

cause an illuminator to illuminate an eye of a user;

detect, with an area image sensor, light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns;

determine a gaze direction of the user based at least in part on light detected by the an area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the an area image sensor comprises;

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the an area image sensor;

determining magnitudes of light exposure at a subset of the total number of rows during one or more illumination periods;

determining magnitudes of light exposure at a subset of the total number of columns during one or more illumination periods;

based on the magnitudes of light exposure at the subset of the total number of rows and the magnitudes of light exposure at the subset of the total number of columns, identifying a subset of a total number of pixels of the an area image sensor which include and surround the one or more pixels of the profile sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the an area image sensor during a subsequent determination of gaze direction during a subsequent illumination period.

13. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user;

an area image sensor configured to detect light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns; and

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises;

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the area image sensor;

determining magnitudes of light exposure at a subset of the total number of rows during one or more illumination periods;

determining magnitudes of light exposure at a subset of the total number of columns during one or more illumination periods;

based on the magnitudes of light exposure at the subset of the total number of rows and the magnitudes of light exposure at the subset of the total number of columns, identifying a subset of a total number of pixels of the area image sensor which include and surround the one or more pixels of the area image sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the area image sensor during a subsequent determination of gaze direction during a subsequent illumination period.

14. The system of claim 13, wherein the system further comprises:

a Fresnel lens in front of a display device;

a hot mirror on at least one side of the Fresnel lens or the display device;

and wherein;

the at least one processor is further configured to adjust the trajectory of virtual missiles which are caused to be virtually propelled by the user in virtual space; and

the first illuminator and image sensor are removable from the system so that they may be used in other devices.

15. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user;

a first profile sensor configured to detect light reflected by the eye of the user;

an area image sensor configured to detect light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns;

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the first profile sensor and based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the first profile sensor comprises;

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the first profile sensor;

identifying a subset of a total number of pixels of the first profile sensor which include and surround the one or more pixels of the first profile sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the first profile sensor during a subsequent determination of gaze direction; and

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises;

determining magnitudes of light exposure at a subset of pixels in each of the total number of rows of the sensitive area during one or more illumination periods;

determining magnitudes of light exposure at a subset of pixels in each of the total number of columns of the sensitive area during one or more illumination periods;

identifying a subset of the total number of pixels of the sensitive area based on the magnitudes of light exposure at the subset of pixels in each of the total number of rows and the magnitudes of light exposure at the subset of pixels in each of the total number of columns; and

analyzing only the subset of the total number of pixels of the of the sensitive area during a subsequent illumination period.

16. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user, the first illuminator comprising a near-infrared illuminator;

a first profile sensor configured to detect light reflected by the eye of the user;

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the first profile sensor;

wherein at least the first illuminator and the first profile sensor are disposed within a wearable device comprising a display device and a cold mirror on a back side of the display device, wherein the cold mirror is configured to reflect visible light toward a front side of the display device, and allow near-infrared to pass through the cold mirror, and wherein the first profile sensor is disposed on the opposite side of the cold mirror from the display device; and

wherein determining the gaze direction of the user based at least in part on light detected by the first profile sensor comprises;

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the first profile sensor;

identifying a subset of a total number of pixels of the first profile sensor which include and surround the one or more pixels of the first profile sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the first profile sensor during a subsequent determination of gaze direction.

17. The system for determining a gaze direction of a user of claim 16, wherein:

the wearable device further comprises a lens in front of the display device; and

the at least one processor is further configured to compensate for the presence of the lens before or during determining the gaze direction of the user.

18. A system for determining a gaze direction of a user, wherein the system comprises:

a first illuminator configured to illuminate an eye of a user;

an area image sensor configured to detect light reflected by the eye of the user, wherein the area image sensor has a sensitive area having a total number of pixels characterized by a total number of rows and a total number of columns; and

at least one processor configured to determine a gaze direction of the user based at least in part on light detected by the area image sensor;

wherein determining the gaze direction of the user based at least in part on light detected by the area image sensor comprises;

determining a location of a glint on the eye caused by light reflected by the eye of the user, based at least in part on a higher magnitude light level detected at one or more pixels of the area image sensor;

determining magnitudes of light exposure at a subset of pixels in each of the total number of rows during one or more illumination periods;

determining magnitudes of light exposure at a subset of pixels in each of the total number of columns during one or more illumination periods;

based on the magnitudes of light exposure at the subset of pixels in each of the total number of rows and the magnitudes of light exposure at the subset of pixels in each of the total number of columns, identifying a subset of a total number of pixels of the area image sensor which include and surround the one or more pixels of the area image sensor which detected the higher magnitude light level; and

analyzing only the subset of the total number of pixels of the area image sensor during a subsequent determination of gaze direction during a subsequent illumination period.

19. The system of claim 18, wherein the system further comprises:

a Fresnel lens in front of a display device;

a hot mirror on at least one side of the Fresnel lens or the display device;

and wherein;

the at least one processor is further configured to adjust the trajectory of virtual missiles which are caused to be virtually propelled by the user in virtual space; and

the first illuminator and image sensor are removable from the system so that they may be used in other devices.