Augmented retinal display with view tracking and data positioning

• US 6,097,353 A
• Filed: 01/20/1998
• Issued: 08/01/2000
• Est. Priority Date: 01/20/1998
• Status: Expired due to Term

• Current Assignee: Washington University In St Louis
• Original Assignee: University of Washington
• Inventors: Johnston, Richard S., Melville, Charles D.
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US09/009,579
• Time in Patent Office: 924 Days
• Field of Search: 345/7, 345/8, 345/9, 359/629, 359/630, 359/631, 359/634, 359/636, 396/57
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features

Abstract

A scanned beam tracking system is included in a virtual retinal display. An infrared light source generates light for scanning the viewer'"'"'s environment in the direction the viewer is looking. A visible light source generates visible light which is scanned on a viewer'"'"'s retina to generate a virtual image. A common scanning system is used to scan both the non-visible light and the visible light. The visible light is directed into the viewer'"'"'s eye. The non-visible light is directed away from the viewer'"'"'s eye into the environment. Infrared reflectors are positioned in the environment. When the infrared light from the virtual retinal display scans over a reflector the reflector directs the infrared light back toward the virtual retinal display. The current pixel of the scanning cycle when the infrared return light is detected corresponds to the position of the reflector.

Claims

1. An augmented scanning display apparatus with orientation detection, view tracking, the augmented scanning display apparatus receiving an image data signal corresponding to a virtual image for viewing by a viewer'"'"'s eye and receiving background light from a background source for passing a background image to the viewer'"'"'s eye, the augmented scanning display apparatus comprising:

an image light source which modulates image light as a function of the image data signal to output modulated image light which defines the virtual image in a sequence of display pixels;

a tracking light source which generates tracking light;

a scanner receiving the tracking light and the modulated image light, the scanner deflecting the received tracking light and image light along a predetermined pattern;

a mirror receiving the tracking light from the scanner and deflecting the tracking light into an external real environment;

a beamsplitter having a first input that receives the modulated image light, a second input that receives the background light, and an output port, the beamsplitter being configured to output both the background light and the modulated image light at the output port to the viewer'"'"'s eye; and

a detector which generates a first signal in response to detection of tracking light re-entering the augmented scanning display apparatus.

2. The apparatus of claim 1 in combination with a processor, the processor receiving the first signal and correlating the detection of the re-entering tracking light to a display pixel among the sequence of display pixels forming the virtual image.

3. The apparatus of claim 1, wherein the tracking light source is modulated and the detector is synchronized to the modulation of the tracking light source.

4. The apparatus of claim 1 in combination with a tracking light reflector located in the external environment, wherein the tracking light deflected into the external environment is deflected along a raster scanning pattern, and wherein when the reflector is within a field of view of the augmented scanning display apparatus, the tracking light impinges on the reflector and is reflected back into the augmented scanning display apparatus as re-entering tracking light.

5. The apparatus of claim 1, wherein the tracking light is infrared light and the detector is an infrared detector.

6. The apparatus of claim 1, further comprising an eyepiece which defines the exit pupil.

7. The apparatus of claim 1, further comprising a processor operative to identify a predetermined pattern of reflected received light corresponding to a selected feature of an external environment, the processor programmed to determine a field of view in response to the identified pattern.

8. The apparatus of claim 1, further comprising a processor operative to determine a view orientation in response to reflected light, and a controller coupled to the modulator to adjust the modulation in response to the determined view orientation.

9. An augmented virtual retinal display system with orientation detection, the augmented virtual retinal display system receiving an image data signal corresponding to a virtual image for viewing by a viewer'"'"'s eye and receiving background light from a background source for passing a background image to the viewer'"'"'s eye, the augmented virtual retinal display system having a field of view for viewing the background light and the virtual image, the augmented virtual retinal display system comprising:

an image light source which modulates image light as a function of the image data signal to output modulated image light which defines the virtual image in a sequence of display pixels;

a tracking light source which generates tracking light;

a scanner receiving the tracking light and the modulated image light, the scanner deflecting the received tracking light and image light along a predetermined pattern;

a mirror receiving the tracking light from the scanner and deflecting the tracking light into an external environment;

a beamsplitter having a first input that receives the modulated image light, a second input that receives the background light, and an output port, the beamsplitter being configured to output both the background light and the modulated image light at the output port to the viewer'"'"'s eye;

a tracking light reflector located in the real environment, wherein the tracking light deflected into the real environment is deflected along a predetermined scanning pattern, and wherein when the reflector is within a field of view of the scanning pattern, the tracking light impinges on the reflector and is reflected back onto the mirror as re-entering tracking light; and

a detector which generates a first signal in response to detection of the re-entering tracking light deflected from the mirror.

10. The system of claim 9, wherein the tracking light source is modulated and the detector is synchronized to the modulation of the tracking light source.

11. The system of claim 9, in combination with a tracking light reflector located in the external environment, wherein the tracking light deflected into the external environment is deflected along a raster scanning pattern, and wherein when the reflector is within a field of view of the augmented virtual retinal display system, the tracking light impinges on the reflector and is reflected back into the augmented virtual retinal display system as re-entering tracking light.

12. The system of claim 9, wherein the tracking light is infrared light and the detector is an infrared detector.

13. The system of claim 9, further comprising an eyepiece which defines the exit pupil.

14. The system of claim 9, further comprising a processor operative to identify a predetermined pattern of reflected received light corresponding to a selected feature of an external environment, the processor programmed to determine a field of view in response to the identified pattern.

15. The system of claim 9, further comprising a processor operative to determine a view orientation in response to reflected light, and a controller coupled to the modulator to adjust the modulation in response to the determined view orientation.

16. The system of claim 9, wherein the tracking light reflector includes a retroreflective material positioned to receive light from the tracking light source and is configured to reflect a portion of the received light back to the tracking light source.

17. The system of claim 9, wherein the reflector includes a predetermined pattern of reflective and non-reflective areas.

18. The system of claim 9, wherein the reflector comprises a center area reflective to tracking light and a plurality of spaced concentric rings reflective to tracking light, wherein space between the plurality of concentric rings does not reflect tracking light.

19. The system of claim 9, in combination with a processor, the processor receiving the first signal and correlating the detection of the re-entering tracking light to a display pixel among the sequence of display pixels forming the virtual image.

20. The system of claim 19, wherein the processor is programmed to embed display data into the image data signal responsive to the received first signal.

21. The system of claim 20, wherein the embedded display data corresponds to virtual imagery fixed relative to the field of view.

22. The system of claim 20, wherein the embedded display data corresponds to virtual imagery fixed relative to the reflector in the environment.

23. The system of claim 19, wherein the reflector is a first reflector and further comprising a second reflector in the real environment, wherein when the second reflector is within the field of view, the tracking light impinges on the second reflector and is reflected back onto the mirror as re-entering tracking light;

and wherein the processor comprises means for triangulating position of the augmented virtual retinal display system relative to the background environment in response to the first signal resulting from detection of re-entering tracking light reflected from the first reflector and to the first signal resulting from detection of re-entering tracking light reflected from the second reflector.

24. The system of claim 19, wherein the reflector is a first reflector and further comprising a second reflector in the real environment, wherein when the second reflector is within the field of view, the tracking light impinges on the second reflector and is reflected back onto the mirror as re-entering tracking light;

and wherein the processor is programmed to determine a view orientation relative to the background environment in response to the first signal resulting from detection of re-entering tracking light reflected from the first reflector and to the first signal resulting from detection of re-entering tracking light reflected from the second reflector.

25. The system of claim 21, wherein the processor is programmed to embed first display data into the image data signal at a position within the field of view, the position fixed relative to the background environment.

26. The system of claim 21, wherein the processor is programmed to embed second display data into the image data signal at a position within the field of view, the position fixed relative to the field of view.

27. The system of claim 21, wherein the imaging optics and processor are configured to alter the content or position of the displayed image in response to detected position.

28. An augmented virtual retinal display system with orientation detection, the augmented virtual retinal display system receiving an image data signal corresponding to a virtual image for viewing by a viewer'"'"'s eye and receiving background light from a background environment for passing a background image to the viewer'"'"'s eye, the augmented virtual retinal display system having a field of view for viewing the background environment and the virtual image, the augmented virtual retinal display system comprising:

a light source for generating visible light and infrared light;

means for modulating the visible light as a function of the image data signal to define a sequence of display pixels forming the virtual image;

a scanner receiving the infrared light and the modulated visible light, the scanner deflecting the received visible light and infrared along a predetermined pattern;

an infrared mirror receiving the infrared light from the scanner and deflecting the infrared light into an external environment;

a beamsplitter having a first input that receives the modulated image light, a second input that receives the background light, and an output port, the beamsplitter being configured to output both the background light and the modulated image light at the output port to the viewer'"'"'s eye;

an infrared reflector located in the external environment, wherein the infrared light deflected into the external environment is deflected along a predetermined scanning pattern, and wherein when the reflector is within a field of view of the scanning pattern, the infrared light impinges on the reflector and is reflected back onto the infrared mirror as re-entering infrared light; and

an infrared detector which generates a first signal in response to detection of the infrared light re-entering the augmented virtual retinal display system.

29. The system of claim 28, wherein the infrared reflector comprises a center area reflective to infrared light and a plurality of spaced concentric rings reflective to infrared light.

30. The system of claim 28, in combination with a processor, the processor generating the image data signal and receiving the first signal, the processor correlating the detection of the re-entering infrared light to a display pixel among the sequence of display pixels forming the virtual image.

31. The system of claim 30, wherein the processor embeds display data into the image data signal responsive to the received first signal.

32. The system of claim 31, wherein the embedded display data corresponds to virtual imagery fixed relative to the field of view.

33. The system of claim 31, wherein the embedded display data corresponds to virtual imagery fixed relative to the infrared reflector in the environment.

34. The system of claim 30, wherein the infrared reflector is a first infrared reflector and further comprising a second infrared reflector in the real environment, wherein when the second infrared reflector is within the field of view, the infrared light impinges on the second infrared reflector and is reflected back onto the infrared mirror as re-entering infrared light;

and wherein the processor comprises means for triangulating position of the augmented virtual retinal display system relative to the background environment in response to the first signal resulting from detection of re-entering infrared light reflected from the first infrared reflector and to the first signal resulting from detection of re-entering infrared light reflected from the second infrared reflector.

35. The system of claim 34, wherein the processor further comprises means for embedding first display data into the image data signal at a position within the field of view, the position fixed relative to the real environment.

36. The system of claim 34, wherein the processor further comprises means for embedding second display data into the image data signal at a position within a field of view, the position fixed relative to the field of view.

37. A method for overlaying a virtual image onto a background image with an augmented virtual retinal display having orientation detection, the augmented virtual retinal display receiving an image data signal corresponding to pixels of the virtual image for viewing by a viewer'"'"'s eye and receiving background light from an external environment for passing the background image to the viewer'"'"'s eye, the method comprising the steps of:

generating image light and tracking light;

modulating the image light as a function of the image data signal to define a sequence of display pixels forming the virtual image;

deflecting the received image light and tracking light along a predetermined pattern;

receiving the deflected tracking light at a mirror and deflecting the tracking light into the external environment;

receiving the deflected image light at a beamsplitter and concave mirror and further deflecting the image light toward the viewer'"'"'s eye;

receiving the background light at the beamsplitter and passing a portion of the background light toward the viewer'"'"'s eye;

reflecting the tracking light deflected into the external environment at a reflector located in the external environment, the reflector reflecting the received tracking light back to the mirror as re-entering tracking light;

detecting the re-entering tracking light; and

correlating a pixel of the virtual image with the detected re-entering tracking light.

38. The method of claim 37, further comprising the step of embedding display data into the image data signal responsive to the detected re-entering tracking light.

39. The method of claim 38, wherein the embedded display data corresponds to virtual imagery fixed relative to the field of view.

40. The method of claim 38, wherein the embedded display data corresponds to virtual imagery fixed relative to the reflector in the environment.

41. The method of claim 37, further comprising the step of:

triangulating position relative to the background environment in response detecting the re-entering tracking light.

42. The method of claim 37, further comprising the steps of identifying a predetermined pattern of reflected received light corresponding to a selected feature of an external environment;

and determining a field of view in response to the identified pattern.

43. The method of claim 37, further comprising the steps of determining a view orientation in response to reflected light;

and adjusting the modulation in response to the determined view orientation.

44. A method for overlaying a virtual image onto a background image with a scanning display having orientation detection, the scanning display receiving an image data signal corresponding the virtual image for viewing by a viewer'"'"'s eye, the method comprising the steps of:

generating image light;

generating a tracking signal indicative of viewer orientation;

modulating the image light as a function of the image data signal to define a sequence of display pixels forming the virtual image;

deflecting the received image light along a predetermined pattern;

receiving the deflected image light at a beamsplitter and concave mirror and further deflecting the image light toward the viewer'"'"'s eye;

receiving light corresponding to the background image at the beamsplitter and passing the received light corresponding to the background image toward the viewer'"'"'s eye; and

correlating the sequence of display pixels and the tracking signal.

45. The method of claim 44, further comprising the step of embedding display data into the image data signal responsive to the tracking signal.

46. The method of claim 45, wherein the embedded display data corresponds to virtual imagery fixed relative to the field of view.