Systems, devices, and methods for laser eye tracking in wearable heads-up displays

• US 10,459,220 B2
• Filed: 11/30/2017
• Issued: 10/29/2019
• Est. Priority Date: 11/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: North Inc.
• Inventors: Aleem, Idris S., Bhargava, Mayank
• Primary Examiner(s): Mercedes, Dismery
• Application Number:
  US15/827,675
• Publication Number:
  US 20180149874A1
• Time in Patent Office: 698 Days
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/015 involving arrangement aiming to get less bulky devices
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 26/0833 the reflecting element being a micromechanical device, e.g. a MEMS mirror, DMD G02B26/0825 takes precedence; micromechanical devices in general B81B
• CPC Class Codes: G02B 26/101 with both horizontal and vertical deflecting means, e.g. raster or XY scanners colour television using laser beams scanning a display screen H04N9/3129
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/0103 comprising holographic elements
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 5/32 Holograms used as optical elements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 9/3102 using two-dimensional electronic spatial light modulators micromechanical modulators as such G02B26/0833; liquid crystal modulators as such G02F1/13
• CPC Class Codes: H04N 9/3129 scanning a light beam on the display screen scanning a light beam on a screen in displays other than projection devices G09G3/02; scanning systems in general G02B26/10; projectors using laser light sources in general H04N9/3161
• CPC Class Codes: H04N 9/3161 using laser light sources using laser beams scanning the display screen H04N9/3129
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

Systems, devices, and methods that use elements of a scanning laser projector (“SLP”) to determine the gaze direction of a user of a wearable heads-up display (“WHUD”) are described. An infrared laser diode is added to an RGB SLP and an infrared photodetector is aligned to detect reflections of the infrared light from the eye. A scan mirror in the SLP sweeps through a range of orientations and the intensities of reflections of the infrared light are monitored by a processor to determine when a spectral reflection or “glint” is produced. The processor determines the orientation of the scan mirror that produced the glint and maps the scan mirror orientation to a region in the field of view of the eye of the user, such as a region in visible display content projected by the WHUD, to determine the gaze direction of the user.

Claims

1. A wearable heads-up display (“

WHUD”

) comprising;

a support frame that in use is worn on a head of a user;

a scanning laser projector carried by the support frame, the scanning laser projector including;

an infrared laser diode;

at least one visible light laser diode; and

at least one scan mirror;

an infrared photodetector carried by the support frame;

a processor carried by the support frame, the processor communicatively coupled to the scanning laser projector and the at least one infrared photodetector; and

a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores data and/or processor-executable instructions that, when executed by the processor, cause the WHUD to;

generate an infrared light by the infrared laser diode;

scan the infrared laser light over the eye of the user by the at least one scan mirror, wherein scanning the infrared laser light over the eye of the user by the at least one scan mirror includes sweeping the at least one scan mirror through a range of orientations and, for a plurality of orientations of the at least one scan mirror, reflecting the infrared laser light to a respective region of the eye of the user;

detect reflections of the infrared laser light from the eye of the user by the at least one infrared photodetector;

determine a respective intensity of a plurality of detected reflections of the infrared light by the processor;

identify, by the processor, at least one detected reflection for which the intensity exceeds a threshold value;

determine, by the processor, the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value; and

determine, by the processor, a region in the field of view of the eye of the user at which a gaze of the eye is directed based on the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value.

2. The WHUD of claim 1, further comprising:

a wavelength-multiplexed holographic optical element (“

HOE”

) carried by the support frame and positioned within the field of view of an eye of the user when the support frame is worn on the head of the user, the wavelength-multiplexed HOE aligned to receive both the infrared light and the visible light from the scanning laser projector and to redirect both the infrared light and the visible light towards the eye of the user when the support frame is worn on the head of the user, wherein the wavelength-multiplexed HOE includes a first hologram that is responsive to the visible light and unresponsive to the infrared light and a second hologram that is responsive to the infrared light and unresponsive to the visible light, and wherein the wavelength-multiplexed HOE is substantially transparent to environment light.

3. The WHUD of claim 2 wherein the first hologram that is responsive to the visible light converges the visible light to a first exit pupil at the eye of the user and the second hologram that is responsive to the infrared light converges the infrared light to a second exit pupil at the eye of the user.

4. The WHUD of claim 1 wherein the non-transitory processor-readable storage medium further stores data and/or processor-executable instructions that, when executed by the processor, cause the WHUD to project visible display content in the field of view of the eye of the user by the scanning laser projector, and wherein the data and/or processor-executable instructions that, when executed by the processor, cause the WHUD to determine, by the processor, a region in a field if view of the eye of the user at which a gaze of the eye is directed based on the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value, cause the WHUD to determine, by the processor, a region of the visible display content at which the gaze of the eye is directed based on the orientation of the at least one scan mirror that corresponds to the at least one detected reflection for which the intensity exceeds the threshold value.

5. The WHUD of claim 4 wherein the at least one infrared photodetector is positioned on the support frame at a periphery of the field of view of the eye of the user when the eye is gazing straight ahead, and wherein the data and/or processor-executable instructions that, when executed by the processor, cause the WHUD to project visible display content in the field of view of the eye of the user by the scanning laser projector, cause the scanning laser projector to position the visible display content away-from-center in the field of view of the eye of the user and towards the position of the at least one infrared photodetector at the periphery of the field of view of the eye of the user.

6. The WHUD of claim 1 wherein the support frame has a general shape and appearance of a pair of eyeglasses.

7. The WHUD of claim 1 wherein the at least one visible light laser diode in the scanning laser projector includes at least one visible light laser diode selected from a group consisting of:

a red laser diode, a green laser diode, a blue laser diode, and any combination of a red laser diode, a green laser diode, and/or a blue laser diode.