Integrated display and photosensor

• US 9,158,113 B2
• Filed: 03/14/2012
• Issued: 10/13/2015
• Est. Priority Date: 03/14/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Amirparviz, Babak, Wang, Chia-Jean
• Primary Examiner(s): HOFFNER, LINH NGUYEN
• Application Number:
  US13/420,316
• Publication Number:
  US 20130241907A1
• Time in Patent Office: 1,308 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/14 Detecting light within display terminals, e.g. using a single or a plurality of photosensors
• CPC Class Codes: G09G 2370/027 Arrangements and methods specific for the display of internet documents
• CPC Class Codes: G09G 3/001 using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background slide projectors per se G03B23/00 = 42 HP
• CPC Class Codes: G09G 3/3413 Details of control of colour illumination sources
• CPC Class Codes: G09G 3/346 based on modulation of the reflection angle, e.g. micromirrors micromirrors devices per se G02B26/0833

Abstract

An apparatus for simultaneously imaging a subject and displaying computer generated image (“CGI”) light to the subject includes a display array and a photodetector array. The display array and the photodetector array are disposed on a same semiconductor die. The display array includes display pixels configured to selectively generate the CGI light to be sent along a forward optical path. The photodetector array is positioned to receive non-visible image light that is reflected by the subject and directed along a reverse optical path. The CGI light to be sent along the forward optical path travels in a substantially opposite direction as the non-visible image light directed along the reverse optical path.

Claims

1. An apparatus for simultaneously imaging a subject and displaying computer generated image (“

CGI”

) light to the subject, the apparatus comprising;

a Liquid Crystal on Silicon (“

LCOS”

) display disposed on a semiconductor die, wherein the LCOS display includes LCOS pixels configured to selectively generate the CGI light to be sent along a forward optical path;

a photodetector array disposed on the semiconductor die and positioned to receive non-visible image light directed along a reverse optical path, wherein the non-visible image light is reflected by the subject and travels along the reverse optical path in a substantially opposite direction as the CGI light to be sent along the forward optical path; and

a filter element disposed in the reverse optical path for substantially filtering out visible light and substantially passing the non-visible image light to the photodetector array, wherein the non-visible image light directed along the reverse optical path travels through at least a portion of one of the LCOS pixels before reaching one of the photodetector pixels.

2. The apparatus of claim 1, wherein photodetector pixels of the photodetector array and the LCOS pixels are arranged in a repeating pattern on the semiconductor die.

3. The apparatus of claim 2, wherein the repeating pattern includes the photodetector pixels uniformly distributed amongst the LCOS pixels.

4. The apparatus of claim 2, wherein the repeating pattern includes columns of the photodetector array interlaced with columns of the LCOS pixels.

5. The apparatus of claim 1, wherein the photodetector array substantially surrounds the LCOS display.

6. The apparatus of claim 1 further comprising a first light source positioned to provide the illumination light to the LCOS display.

7. The apparatus of claim 6, wherein the first light source includes a red-green-blue light emitting diode (“

RGB LED”

) and modulation circuitry configured to individually strobe each color of the RGB LED.

8. The apparatus of claim 1 further comprising a second light source positioned to emit the non-visible image light onto the subject.

9. An apparatus for a head mounted display (“

HMD”

), the apparatus comprising;

a first light source for providing illumination light;

a Liquid Crystal on Silicon (“

LCOS”

) array disposed on a semiconductor die, wherein LCOS pixels of the LCOS array are configured to selectively reflect the illumination light provided by the first light source to generate computer generated image (“

CGI”

) light to be sent along a forward optical path within an eyepiece;

a second light source oriented to emit non-visible light in an eyeward direction of the HMD;

a photodetector array disposed on the semiconductor die and positioned to receive a reflection of the non-visible light emitted in the eyeward direction, the reflection directed along a reverse optical path in the eyepiece in a substantially opposite direction as the CGI light along the forward optical path;

an end reflector disposed at an opposite end of the eyepiece from the LCOS array;

a polarizing beam splitter positioned to direct the illumination light to the LCOS array and disposed within the eyepiece between the end reflector and the LCOS array; and

a beam splitter for directing the reflection of the non-visible light along the reverse optical path in the eyepiece, the beam splitter disposed within the eyepiece between the polarizing beam splitter and the end reflector.

10. The apparatus of claim 9 further comprising a fixed polarization rotator disposed between the beam splitter and the end reflector, wherein the beam splitter is a polarization beam splitter.

11. The apparatus of claim 9, wherein photodetector pixels of the photodetector array and the LCOS pixels are arranged in a repeating pattern on the semiconductor die, and wherein the repeating pattern includes the photodetector pixels uniformly distributed amongst the LCOS pixels.

12. The apparatus of claim 9, wherein first lines of photodetector pixels of the photodetector array are interlaced with second lines of the LCOS pixels, in a repeating pattern on the semiconductor die.

13. The apparatus of claim 9, wherein the reflection travels through liquid crystal of the LCOS pixels before reaching photodetector pixels of the photodetector array.

14. The apparatus of claim 9, further comprising a polarizing filter disposed over the second light source for polarizing the non-visible light.

15. The apparatus of claim 9, wherein the non-visible light is substantially near-infrared light centered between 800 and 1000 nm.

16. The apparatus of claim 9, wherein the eyepiece is mounted to a frame assembly of the HMD, the frame assembly including a forward facing camera oriented to monitor a field of view seen by an eye of a user of the HMD.

17. The apparatus of claim 16, further comprising control circuitry configured to receive eye image data generated from the photodetector array and scene image data generated from the forward facing camera, wherein the control circuitry is further configured to control the LCOS array.

18. A method of operating a head mounted display (“

HMD”

), the method comprising;

acquiring view data corresponding to a field of view of a user of the HMD;

receiving non-visible light reflected off of an eye of the user of the HMD to generate eye image data;

determining a visual point of interest of the user based on the view data and the eye image data; and

projecting refreshed computer generated image (“

CGI”

) light from a Liquid Crystal on Silicon (“

LCOS”

) array onto the eye of the user based on the visual point of interest, wherein a photodetector array receives the non-visible light and the photodetector array and the LCOS array are both disposed on a same semiconductor die, and wherein the non-visible light reflected off of the eye of the user travels through liquid crystal of the LCOS array before being received by the photodetector array.

19. The method of claim 18, wherein the view data includes existing CGI projected onto the eye of the user from the LCOS array, wherein the existing CGI is presently, or has been previously, projected onto the eye of the user.

20. The method of claim 19 further comprising:

controlling a CGI engine to create a new CGI in response to determining the visual point of interest of the user, wherein the new CGI is for projecting as the refreshed CGI light.

21. The method of claim 18, wherein the view data includes scene images from a camera of the HMD oriented to monitor a forward view seen by the user.

22. The method of claim 21 further comprising:

controlling a CGI engine to create a new CGI in response to determining the visual point of interest of the user, wherein the new CGI is for projecting as the refreshed CGI light and the new CGI includes information about the visual point of interest.

23. The method of claim 18 further comprising:

correcting the eye image data based on a transmission characteristic of the liquid crystal.

24. The method of claim 18 further comprising:

emitting the non-visible light onto the eye of the user with a near-infrared light source emitting near-infrared light centered between 800 and 1000 nm.