AUGMENTED REALITY DEVICE FOR PRESENTING VIRTUAL IMAGERY REGISTERED TO A VIEWED SURFACE

• US 20110018903A1
• Filed: 10/04/2010
• Published: 01/27/2011
• Est. Priority Date: 08/03/2004
• Status: Abandoned Application

• Current Assignee: Silverbrook Research Pty Limited
• Original Assignee: Silverbrook Research Pty Limited
• Inventors: Silverbrook, Kia, Lapstun, Paul
• Application Number:
  US12/897,758
• Publication Number:
  US 20110018903A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 26/06 for controlling the phase of light G02B26/08 takes precedence , measuring optical phase difference G01J9/00
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 30/27 involving lenticular arrays
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

An augmented reality device for inserting virtual imagery into a user'"'"'s view of their physical environment. The device comprises: a see-through display device including a wavefront modulator; a camera for imaging a surface in the physical environment; and a controller. The controller is configured for capturing an image of the surface; determining the virtual imagery to be displayed at a predetermined position relative to the surface; determining a position of the surface relative to the augmented reality device; generating an image based on the virtual imagery and on the position of the surface relative to the augmented reality device; and displaying the generated image via the display device. Based on pixel depth information, the controller modulates the wavefront curvature of light emitted for each pixel so that the user sees the virtual imagery at the predetermined position relative to the surface regardless of changes in position of the user'"'"'s eyes with respect to the display device.

Claims

1. An augmented reality device for inserting virtual imagery into a user'"'"'s view of their physical environment, the device comprising:

a see-through display device through which the user can view the physical environment, said display device including a wavefront modulator;

a camera for imaging at least one surface in the physical environment; and

a controller configured for;

capturing, using the camera, at least one image of the surface;

determining, at least partially from the at least one captured image, the virtual imagery to be displayed at a predetermined position relative to the surface;

determining, at least partially from the at least one captured image, a position of the surface relative to the augmented reality device;

generating at least one image based on the virtual imagery and on the position of the surface relative to the augmented reality device, the image including pixel depth information; and

displaying the generated image via the display device, including modulating, based on the pixel depth information, the wavefront curvature of the light emitted for each pixel, so that the user sees the virtual imagery at the predetermined position relative to the surface regardless of changes in position of the user'"'"'s eyes with respect to the display device.

2. An augmented reality device according to claim 1 wherein the display device has two see-through displays, one for each of the user'"'"'s eyes respectively.

3. An augmented reality device according to claim 1 wherein the display device, the camera and the controller are adapted to be worn on the user'"'"'s head.

4. An augmented reality device according to claim 1 wherein the display device has a virtual retinal display (VRD) for each of the user'"'"'s eyes, each of the VRD'"'"'s scanning at least one beam of light into a raster pattern and modulating the or each beam to produce spatial variations in the virtual imagery.

5. An augmented reality device according to claim 4 wherein the VRD scans red, green and blue beams of light to produce color pixels in the raster pattern.

6. An augmented reality device according to claim 5 wherein the VRDs present different images to each of the user'"'"'s eyes, the differences being based on eye separation and the distance to the predetermined position of the virtual imagery so as to create a perception of depth via stereopsis.

7. An augmented reality device according to claim 1 wherein the wavefront modulator uses a deformable membrane mirror, liquid crystal phase corrector, a variable focus liquid lens or a variable focus liquid mirror.

8. An augmented reality device according to claim 1 wherein the virtual imagery is a movie, a computer application interface, computer application output, hand drawn strokes, text, images or graphics.

9. An augmented reality device according to claim 1 wherein the display device has pupil trackers to detect an approximate point of fixation of the user'"'"'s gaze such that a virtual cursor can be projected into the virtual imagery and navigated using gaze direction.

10. An augmented reality device according to claim 1, further comprising an optical range finder for determining range information using time-of-flight measurement, wherein the controller is configured for determining the position of the surface relative to the augmented reality device using the range information in combination with the at least one captured image.

11. An augmented reality device according to claim 1 wherein the surface has a pattern of coded data disposed thereon, and wherein the controller is configured to at least partially identify the virtual imagery to be displayed using at least part of the pattern of coded data contained in the at least one captured image.

12. An augmented reality device according to claim 11, wherein the pattern of coded data is indicative of an identity of the surface, and wherein the controller is configured to at least partially identify the virtual imagery to be displayed based on the identity of the surface.

13. An augmented reality device according to claim 1, wherein the surface has a pattern of coded data disposed thereon, and wherein the controller is configured to at least partially determine the position of the surface relative to the augmented reality device using at least part of the pattern of coded data contained in the at least one captured image.

14. An augmented reality device according to claim 13, wherein the pattern of coded data disposed on the surface includes a grid of target elements.

15. An augmented reality device according to claim 13, wherein the pattern of coded data disposed on the surface is indicative of a plurality of coordinate locations on the surface, and wherein the controller is configured to at least partially determine the position of the surface relative to the augmented reality device using at least one of the coordinate locations.