Systems, devices, and methods for laser eye tracking in wearable heads-up displays

• US 10,409,057 B2
• Filed: 11/30/2017
• Issued: 09/10/2019
• Est. Priority Date: 11/30/2016
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: North Inc. (Alphabet Inc.)
• Inventors: Aleem, Idris S., Bhargava, Mayank
• Primary Examiner(s): Mercedes, Dismery
• Application Number:
  US15/827,667
• Publication Number:
  US 20180149863A1
• Time in Patent Office: 649 Days
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/015 involving arrangement aiming to get less bulky devices
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 26/0833 the reflecting element being a micromechanical device, e.g. a MEMS mirror, DMD G02B26/0825 takes precedence; micromechanical devices in general B81B
• CPC Class Codes: G02B 26/101 with both horizontal and vertical deflecting means, e.g. raster or XY scanners colour television using laser beams scanning a display screen H04N9/3129
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/0103 comprising holographic elements
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 5/32 Holograms used as optical elements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 9/3102 using two-dimensional electronic spatial light modulators micromechanical modulators as such G02B26/0833; liquid crystal modulators as such G02F1/13
• CPC Class Codes: H04N 9/3129 scanning a light beam on the display screen scanning a light beam on a screen in displays other than projection devices G09G3/02; scanning systems in general G02B26/10; projectors using laser light sources in general H04N9/3161
• CPC Class Codes: H04N 9/3161 using laser light sources using laser beams scanning the display screen H04N9/3129
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

Systems, devices, and methods that use elements of a scanning laser projector (“SLP”) to determine the gaze direction of a user of a wearable heads-up display (“WHUD”) are described. An infrared laser diode is added to an RGB SLP and an infrared photodetector is aligned to detect reflections of the infrared light from the eye. A scan mirror in the SLP sweeps through a range of orientations and the intensities of reflections of the infrared light are monitored by a processor to determine when a spectral reflection or “glint” is produced. The processor determines the orientation of the scan mirror that produced the glint and maps the scan mirror orientation to a region in the field of view of the eye of the user, such as a region in visible display content projected by the WHUD, to determine the gaze direction of the user.

Claims

1. A method of determining a gaze direction of an eye of a user, the method comprising:

generating an infrared laser light by an infrared laser diode;

scanning the infrared laser light over the eye of the user by at least one scan mirror, wherein scanning the infrared laser light over the eye of the user by the at least one scan mirror includes sweeping the at least one scan mirror through a range of orientations and, for a plurality of orientations of the at least one scan mirror, reflecting the infrared laser light to a respective region of the eye of the user;

detecting reflections of the infrared laser light from the eye of the user by at least one infrared photodetector;

determining a respective intensity of a plurality of detected reflections of the infrared laser light by at least one processor communicatively coupled to the at least one infrared photodetector;

identifying, by the at least one processor, at least one detected reflection for which the intensity exceeds a threshold value;

determining, by the at least one processor, the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value; and

determining, by the at least one processor, a region in a field of view of the eye of the user at which a gaze of the eye is directed based on the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value.

2. The method of claim 1 wherein scanning the infrared laser light over the eye of the user by at least one scan mirror includes scanning, by the at least one scan mirror, the infrared laser light over an area of a holographic optical element positioned in the field of view of the eye of the user and redirecting the infrared laser light towards the eye of the user by the holographic optical element.

3. The method of claim 2 wherein redirecting the infrared laser light towards the eye of the user by the holographic optical element includes converging the infrared laser light to an exit pupil at the eye of the user by the holographic optical element.

4. The method of claim 1 wherein scanning the infrared laser light over the eye of the user by at least one scan mirror includes scanning the infrared laser light across a first dimension of the eye of the user by a first scan mirror and scanning the infrared laser light across a second dimension of the eye of the user by a second scan mirror, and wherein:

sweeping the at least one scan mirror through a range of orientations includes sweeping the first scan mirror through a first range of orientations and, for each respective orientation of the first scan mirror, sweeping the second scan mirror through a second range of orientations;

for a plurality of orientations of the at least one scan mirror, reflecting the infrared laser light to a respective region of the eye of the user includes, for a plurality of combinations of a first orientation of the first scan mirror and a second orientation of the second scan mirror, reflecting the infrared laser light to a respective region of the eye of the user;

determining, by the at least one processor, the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value includes determining, by the at least one processor, the combination of the first orientation of the first scan mirror and the second orientation of the second scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value; and

determining, by the at least one processor, a region in a field of view of the eye of the user at which a gaze of the eye is directed based on the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value includes determining, by the at least one processor, the region in the field of view of the eye of the user at which the gaze of the eye is directed based on the combination of the first orientation of the first scan mirror and the second orientation of the second scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value.

5. The method of claim 1 wherein identifying, by the at least one processor, at least one detected reflection for which the intensity exceeds a threshold value includes detecting, by the infrared photodetector, a spectral reflection of the infrared laser light from the eye of the user.

6. The method of claim 1 wherein identifying, by the at least one processor, at least one detected reflection for which the intensity exceeds a threshold value includes sampling, by the processor, a signal from the infrared photodetector and identifying, by the processor, a first sample for which the magnitude exceeds a threshold magnitude.

7. The method of claim 6 wherein identifying, by the at least one processor, at least one detected reflection for which the intensity exceeds a threshold value further includes identifying, by the at least one processor, a second sample for which the magnitude does not exceed the threshold magnitude.

8. The method of claim 1 wherein determining, by the at least one processor, a region in a field of view of the eye of the user at which a gaze of the eye is directed based on the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value includes effecting, by the at least one processor, a mapping between the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value and the field of view of the eye of the user.

9. The method of claim 8 wherein effecting, by the at least one processor, a mapping between the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value and the field of view of the eye of the user includes performing, by the at least one processor, at least one transformation selected from a group consisting of:

a linear transformation between a set of scan mirror orientations and a set of gaze directions of the eye of the user, a geometric transformation between a set of scan mirror orientations and a set of gaze directions of the eye of the user, an affine transformation between a set of the scan mirror orientations and a set of gaze directions of the eye of the user, and a neural network transformation between a set of scan mirror orientations and a set of gaze directions of the eye of the user.

10. The method of claim 1 wherein the infrared laser diode and the at least scan mirror are components of a scanning laser projector, the scanning laser projector further including at least one additional laser diode to generate visible laser light, and wherein the method further comprises:

projecting visible display content in the field of view of the eye of the user by the scanning laser projector, wherein determining, by the at least one processor, a region in a field of view of the eye of the user at which a gaze of the eye is directed based on the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value includes determining, by the at least one processor, a region of the visible display content at which the gaze of the eye is directed based on the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value.

11. The method of claim 10 wherein determining, by the at least one processor, a region of the visible display content at which the gaze of the eye is directed based on the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value includes performing, by the at least one processor, at least one transformation selected from a group consisting of:

a linear transformation between a set of scan mirror orientations and a set of regions of the visible display content, a geometric transformation between a set of scan mirror orientations and a set of regions of the visible display content, an affine transformation between a set of the scan mirror orientations and a set of regions of the visible display content, and a neural network transformation between a set of scan mirror orientations and a set of regions of the visible display content.

12. The method of claim 10 wherein the at least one infrared photodetector is positioned at a first position at a periphery of the field of view of the eye of the user when the eye is gazing straight ahead, and wherein projecting visible display content in the field of view of the eye of the user by the scanning laser projector includes positioning, by the scanning laser projector, the visible display content away-from-center in the field of view of the eye of the user and towards the position of the at least one infrared photodetector at the periphery of the field of view of the eye of the user.