Augmented reality light guide display

• US 9,558,590 B2
• Filed: 03/28/2012
• Issued: 01/31/2017
• Est. Priority Date: 03/28/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Westerinen, William J., Robbins, Steven John, Badyal, Rajeev, Fleck, Rod G.
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Flora, Nurun N
• Application Number:
  US13/432,372
• Publication Number:
  US 20130257848A1
• Time in Patent Office: 1,770 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1637 Details related to the display arrangement, including those related to the mounting of the display in the housing
• CPC Class Codes: G06F 1/1643 the display being associated to a digitizer, e.g. laptops that can be used as penpads details related to the relative motion of the display enclosure with respect to the body enclosure, e.g. to move between laptop and tablet PC configuration G06F1/1615
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 2203/04804 Transparency, e.g. transparent or translucent windows
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: H04N 1/00129 with a display device, e.g. CRT or LCD monitor with a television apparatus H04N1/00283; arrangements for control of display devices G09G
• CPC Class Codes: H04N 2101/00 Still video cameras

Abstract

Augmented reality light guide display techniques are described. In one or more implementations, an apparatus includes a housing configured in a hand-held form factor, one or more sensors configured to detect a position and orientation of the housing in three dimensions in a physical environment of the housing, a light guide that is at least partially transparent and supported by the housing, a light engine that is optically coupled to the light guide, and one or more modules disposed within the housing and implemented at least partially in hardware. The one or more modules are configured to calculate a position and orientation of an augmentation and cause the light engine to output the augmentation for display using the light guide such that the augmentation is viewable concurrently with at least a portion of the physical environment through the light guide.

Claims

1. An apparatus comprising:

a housing configured in a hand-held form factor such that the housing is movable relative to a user;

one or more sensors configured to detect a position and orientation of the housing in three dimensions in a physical environment of the housing;

a light guide that is at least partially transparent and supported by the housing;

a light engine able to produce images, the light engine optically coupled to the light guide; and

one or more circuits configured to calculate a position and orientation of an augmentation based on the detected position and orientation of the housing in the physical environment and cause the light engine to output the augmentation for display using the light guide such that the augmentation changes based on a distance between the user and the housing as the housing is moved relative to the user and is viewable by the user concurrently with at least a portion of the physical environment through the light guide.

2. An apparatus as described in claim 1, wherein the augmentation is viewable in three dimensions by the user.

3. An apparatus as described in claim 1, wherein the housing is formed as part of a mobile phone or tablet and the one or more modules include wireless communication functionality.

4. An apparatus as described in claim 1, wherein the light guide is configured to display the augmentation along an image plane focused at infinity.

5. An apparatus as described in claim 1, wherein the light guide includes a layer that is selectable by the one or more modules to alter transparency of portions of the light guide such that a first said portion is transparent while a second said portion is not transparent.

6. An apparatus as described in claim 5, wherein the first said portion is used to view the portion of the physical environment and the second said portion is used to view the augmentation.

7. An apparatus as described in claim 1, wherein the one or more sensors are configured as one or more cameras, one or more depth sensing cameras, a gyroscope, one or more accelerometers, or a magnetometer.

8. A method comprising:

capturing one or more images of a user using one or more cameras of a handheld device that is held by a user, the handheld device movable relative to the user;

calculating a location of the user'"'"'s pupils in three dimensional space from the captured one or more images by the handheld device, the location of the user'"'"'s pupils comprising at least a distance from the handheld device to the user'"'"'s pupils;

calculating a location of the handheld device in three dimensional space relative to a physical surroundings of the handheld device from one or more sensors of the handheld device; and

displaying an augmentation on a transparent display of the handheld device based on the calculated location of the user'"'"'s pupils and the calculated location of the handheld device relative to the physical surrounding, the augmentation changing as the handheld device is moved relative to the user and viewable concurrently with at least a portion of the physical surroundings of the handheld device that is viewable through the transparent display.

9. A method as described in claim 8, wherein the transparent display is a light guide that is optically coupled to a light engine, the light engine configured to output the augmentation as specified by the one or more modules.

10. A method as described in claim 9, wherein the light engine does not output a display of the portion of the physical surroundings of the handheld device.

11. A method as described in claim 9, wherein the light guide is configured to display the augmentation along an image plane focused at infinity.

12. A method as described in claim 9, wherein the light guide includes a layer that is selectable to alter transparency of portions of the light guide such that a first said portion is transparent while a second said portion is not transparent.

13. A method as described in claim 12, wherein the first said portion is used to view the portion of the physical environment and the second said portion is used to view the augmentation.

14. A method as described in claim 8, wherein the handheld device is configured as a mobile phone or tablet.

15. A method as described in claim 8, wherein the transparent display includes a first side that is viewable by the user to see the augmentation and a second side opposing the first side, the second side including one or more touch sensors configured to detect one or more gestures that are recognizable by the handheld device to initiate an operation of the handheld device.

16. A method as described in claim 8, wherein the transparent display is configured to display the augmentation to be viewable in three dimensions by the user.

17. An apparatus comprising:

a housing configured in a hand-held form factor such that the housing is movable relative to a user;

one or more cameras positioned in the housing to track one or more eyes of the user;

one or more sensors configured to detect a position and orientation of the housing in three dimensions in a physical environment of the housing;

a light guide that is at least partially transparent and supported by the housing;

a light engine that is optically coupled to the light guide; and

one or more modules disposed within the housing and implemented at least partially in hardware, the one or more modules configured to calculate a position of one or more pupils of the user in three-dimensional space and cause the light engine to output an augmentation for display based on the calculated position of the housing relative to the pupils of the user as well as the position and orientation of the housing in the physical environment using the light guide such that the augmentation changes as the housing is moved relative to the user and is viewable concurrently with at least a portion of the physical environment through the light guide, the position of the one or more pupils of the user comprising at least a distance between the housing and the one or more pupils of the user.

18. An apparatus as described in claim 17, wherein the light guide includes a layer that is selectable to alter transparency of portions of the light guide such that a first said portion that corresponds to the portion of the physical environment is made transparent while a second said portion that corresponds to the augmentation is not made transparent.

19. An apparatus as described in claim 17, wherein the augmentation is viewable in three dimensions by the user.

20. An apparatus as described in claim 17, wherein the one or more sensors are configured as one or more cameras, one or more depth sensing cameras, a gyroscope, one or more accelerometers, or a magnetometer.