Augmented reality virtual monitor

• US 9,497,501 B2
• Filed: 12/06/2011
• Issued: 11/15/2016
• Est. Priority Date: 12/06/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Mount, Brian, Latta, Stephen, Poulos, Adam, McCulloch, Daniel, Bennett, Darren, Hastings, Ryan, Scott, Jason
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Elbinger, Steven
• Application Number:
  US13/312,604
• Publication Number:
  US 20130141421A1
• Time in Patent Office: 1,806 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/41415 involving a public display, viewable by several users in a public space outside their home, e.g. movie theatre, information kiosk
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

A head-mounted display includes a see-through display and a virtual reality engine. The see-through display is configured to visually augment an appearance of a physical space to a user viewing the physical space through the see-through display. The virtual reality engine is configured to cause the see-through display to visually present a virtual monitor that appears to be integrated with the physical space to a user viewing the physical space through the see-through display.

Claims

1. A head-mounted display, comprising:

a see-through display configured to visually augment an appearance of a physical space to a user viewing the physical space through the see-through display;

a virtual reality engine configured to cause the see-through display to visually present a virtual monitor that appears to be integrated with the physical space at a particular physical-space location to the user viewing the physical space through the see-through display, the particular physical-space location mapped to a shared virtual coordinate system useable by other see-through displays to visually present the virtual monitor at the particular physical-space location; and

wherein

the virtual reality engine is further configured to play a video stream on the virtual monitor, and further configured to present the virtual monitor as a moving virtual monitor, the moving virtual monitor shaped to match a non-planar shape of an object moving within the physical space, and rendered so as to move relative to real-world objects within the physical space as perceived by the user viewing the physical space through the see-through display.

2. The head-mounted display of claim 1, further comprising a speaker, and where the virtual reality engine is further configured to cause the speaker to play an audio stream synced to the video stream.

3. The head-mounted display of claim 2, where the virtual reality engine is further configured to modulate a volume of the audio stream inversely proportional to a distance between the see-through display and a physical-space location at which the virtual monitor appears to be located to the user viewing the physical space through the see-through display.

4. The head-mounted display of claim 2, where the virtual reality engine is further configured to modulate a volume of the audio stream in proportion to a directness that the see-through display is viewing a physical-space location at which the virtual monitor appears to be located to the user viewing the physical space through the see-through display.

5. The head-mounted display of claim 1, further comprising a communication subsystem configured to wirelessly receive the video stream.

6. The head-mounted display of claim 1, where the virtual reality engine is further configured to present the virtual monitor with a specific orientation in the physical space, such that the user viewing the physical space through the see-through display perceives images displayed on the virtual monitor based on a location and orientation of the user in the physical space relative to a location and orientation of the virtual monitor in the physical space.

7. The head-mounted display of claim 1, further comprising a sensor subsystem, where the virtual reality engine is configured to control the virtual monitor responsive to commands recognized via the sensor subsystem.

8. The head-mounted display of claim 7, where the sensor subsystem includes a microphone, and where the virtual reality engine is configured to control the virtual monitor responsive to audible commands recognized via the microphone.

9. The head-mounted display of claim 7, where the sensor subsystem includes a camera, and where the virtual reality engine is configured to control the virtual monitor responsive to hand gesture commands recognized via the camera.

10. The head-mounted display of claim 9, wherein a single virtual monitor is split into two virtual monitors responsive to detection of a karate chop gesture performed by the user.

11. The head-mounted display of claim 7, where the sensor subsystem includes a camera, and where the virtual reality engine is configured to control the virtual monitor responsive to eye gesture commands recognized via the camera.

12. The head-mounted display of claim 1, where the virtual monitor displays three-dimensional virtual objects by visually presenting different views to a left eye and a right eye of the user.

13. The head mounted display of claim 1, wherein the moving virtual monitor is not constrained to a stationary real-world object.

14. The head mounted display of claim 1, wherein the moving virtual monitor is constrained to move within boundaries of a moving real-world object.

15. The head-mounted display of claim 1, wherein the moving virtual monitor is rendered to move within the physical space responsive to movement of the user within the physical space, so as to appear to be at a fixed distance and orientation relative to the user.

16. A method of augmenting reality, the method comprising:

receiving observation information of a physical space from a sensor subsystem;

mapping a virtual reality environment to the physical space based on the observation information, the virtual reality environment including a virtual monitor visually presenting a video stream, the virtual monitor shaped to match a non-planar shape of an object moving within the physical space; and

sending augmented reality display information to a see-through display, the augmented reality display information configured to cause the see-through display to display the virtual reality environment mapped to the physical space so that a user viewing the physical space through the see-through display sees the virtual monitor integrated with the physical space, and such that two or more users viewing the physical space through different see-through displays will see the same virtual monitor in a same physical-space location and with the same orientation.

17. The method of claim 16, further comprising sending augmented reality audio information to a speaker, the augmented reality audio information configured to cause the speaker to play an audio stream synced to the video stream.

18. The method of claim 16, wherein the virtual monitor is shaped to match a non-planar shape of another virtual object.

19. A data-holding device holding memory instructions executable by a logic subsystem to:

receive observation information of a physical space from a sensor subsystem;

map a virtual reality environment to the physical space based on the observation information, the virtual reality environment including a virtual monitor visually presenting dynamic display information; and

send augmented reality display information to a see-through display, the augmented reality display information configured to cause the see-through display to display the virtual reality environment mapped to the physical space so that a user viewing the physical space through the see-through display perceives the virtual monitor as a moving virtual monitor, the moving virtual monitor shaped to match a non-planar shape of an object moving within the physical space, and rendered so as to move relative to real-world objects within the physical space as perceived by the user viewing the physical space through the see-through display.

20. The data-holding device of claim 19, further holding instructions executable by a logic sub system to:

present the virtual monitor such that the virtual monitor moves relative to real world objects in the physical space unconstrained by a surface of a real world object.