Augmented reality virtual monitor

• US 10,497,175 B2
• Filed: 09/07/2016
• Issued: 12/03/2019
• Est. Priority Date: 12/06/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Mount, Brian, Latta, Stephen, Poulos, Adam, McCulloch, Daniel, Bennett, Darren, Hastings, Ryan, Scott, Jason
• Primary Examiner(s): Wu, Xiao M
• Assistant Examiner(s): Elbinger, Steven Z
• Application Number:
  US15/258,791
• Publication Number:
  US 20160379417A1
• Time in Patent Office: 1,182 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/41415 involving a public display, viewable by several users in a public space outside their home, e.g. movie theatre, information kiosk
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

A head-mounted display includes a see-through display and a virtual reality engine. The see-through display is configured to visually augment an appearance of a physical space to a user viewing the physical space through the see-through display. The virtual reality engine is configured to cause the see-through display to visually present a virtual monitor that appears to be integrated with the physical space to a user viewing the physical space through the see-through display.

Claims

1. A head-mounted display, comprising:

a see-through display configured to visually augment an appearance of a physical space to a user viewing the physical space through the see-through display;

one or more sensors configured to sense parameters of the physical space and output computer-readable observation information based on the sensed parameters;

a communication subsystem; and

a computer system configured to;

receive a video stream via the communication subsystem;

receive observation information of the physical space from the one or more sensors;

determine a screen-space position on the see-through display at which to present images of a virtual monitor playing the video stream, the screen-space position based on the observation information such that the virtual monitor appears to be integrated with the physical space to the user viewing the physical space through the see-through display;

based on the observation information, present images on the see-through display of a stationary virtual object that appears to be integrated with the physical space to the user viewing the physical space through the see-through display; and

adjust the screen-space position of the images of the virtual monitor on the see-through display so that the virtual monitor appears to move within the user'"'"'s field of view while moving relative to real-world objects within the physical space as perceived by the user viewing the physical space through the see-through display, and so that the virtual monitor is constrained to move within boundaries of the stationary virtual object as perceived by the user viewing the physical space through the see-through display.

2. The head-mounted display of claim 1, wherein the computer system is further configured to:

adjust the screen-space position of the images of the virtual monitor on the see-through display so that the virtual monitor appears to move relative to real-world objects within the physical space irrespective of movement of the user viewing the physical space through the see-through display.

3. The head-mounted display of claim 1, wherein the computer system is further configured to:

adjust the screen-space position of the images of the virtual monitor on the see-through display so that the virtual monitor is constrained to move within boundaries of a stationary real-world object as perceived by the user viewing the physical space through the see-through display.

4. The head-mounted display of claim 1, wherein the computer system is further configured to:

adjust the screen-space position of the images of the virtual monitor on the see-through display so that the virtual monitor is constrained to move within boundaries of a moving real-world object as perceived by the user viewing the physical space through the see-through display.

5. The head-mounted display of claim 1, wherein the computer system is further configured to:

adjust the screen-space position of the images of the virtual monitor on the see-through display so that the virtual monitor appears to move without being constrained to a real-world object as perceived by the user viewing the physical space through the see-through display.

6. The head-mounted display of claim 5, wherein the computer system is further configured to:

based on the observation information, present images on the see-through display of a moving virtual object that appears to be integrated with the physical space to the user viewing the physical space through the see-through display; and

adjust the screen-space position of the images of the virtual monitor on the see-through display so that the virtual monitor is constrained to move within boundaries of the moving virtual object as perceived by the user viewing the physical space through the see-through display.

7. The head-mounted display of claim 1, wherein the computer system is further configured to map a virtual coordinate system to the physical space based on the observation information such that images presented on the see-through display appear to be at particular physical space locations to the user viewing the physical space through the see-through display.

8. The head-mounted display of claim 7, wherein the virtual coordinate system is a shared virtual coordinate system usable by other see-through displays to adjust the screen-space position of images of the virtual monitor on each see-through display so that the virtual monitor appears at a same physical space location as perceived by each user viewing the physical space through a see-through display.

9. The head-mounted display of claim 1, wherein the computer system is configured to control the virtual monitor based on commands received by the one or more sensors.

10. The head-mounted display of claim 1, further comprising a speaker, and wherein the computer system is further configured to:

receive an audio stream via the communication subsystem; and

cause the speaker to play the audio stream synced to the video stream.

11. The head-mounted display of claim 1, wherein the video stream comprises a pre-determined sequence of images.

12. A head-mounted display, comprising:

a see-through display configured to visually augment an appearance of a physical space to a user viewing the physical space through the see-through display;

one or more sensors configured to sense parameters of the physical space and output computer-readable observation information based on the sensed parameters;

a communication subsystem; and

a computer system configured to;

receive a video stream comprising 3D content via the communication subsystem;

receive observation information of the physical space from the one or more sensors;

determine screen-space positions on the see-through display at which to present images of a virtual monitor playing the video stream, the screen-space positions based on the observation information such that the virtual monitor appears to be integrated with the physical space to the user viewing the physical space through the see-through display; and

adjust the screen-space position of the images of the virtual monitor on the see-through display so that the virtual monitor is constrained to move within boundaries of a stationary real-world object as perceived by the user viewing the physical space through the see-through display.

13. The head-mounted display of claim 12, wherein the computer system is further configured to present the video stream such that 3D objects appear to leave a display screen of the virtual monitor, as perceived by the user viewing the physical space through the see-through display.

14. The head-mounted display of claim 12, wherein the computer system is further configured to present images of the virtual monitor as having a non-planar display screen.

15. The head-mounted display of claim 12, wherein the computer system is further configured to map a virtual coordinate system to the physical space based on the observation information such that images presented on the see-through display appear to be at particular physical space locations to the user viewing the physical space through the see-through display.

16. The head-mounted display of claim 15, wherein the virtual coordinate system is a shared virtual coordinate system usable by other see-through displays to adjust the screen-space positions of images of the virtual monitor on each see-through display so that the virtual monitor appears at a same physical space location as perceived by each user viewing the physical space through a see-through display.

17. The head-mounted display of claim 12, wherein the computer system is configured to control the virtual monitor based on commands received by the sensor subsystem.

18. The head-mounted display of claim 12, further comprising a speaker, and wherein the computer system is further configured to cause the speaker to play an audio stream synced to the video stream.

19. A method of augmenting reality, the method comprising:

receiving observation information of a physical space from one or more sensors;

mapping a virtual reality environment to the physical space based on the observation information;

receiving a video stream;

displaying at a screen-space position of a see-through display a virtual monitor playing the video stream, the screen-space position based on the observation information such that the virtual monitor appears to be integrated with the physical space to a user viewing the physical space through the see-through display;

based on the observation information, present images on the see-through display of a moving virtual object that appears to be integrated with the physical space to the user viewing the physical space through the see-through display; and

adjust the screen-space position of the images of the virtual monitor on the see-through display so that the virtual monitor is constrained to move within boundaries of the moving virtual object as perceived by the user viewing the physical space through the see-through display.