Augmented Reality

• US 20170053445A1
• Filed: 04/26/2016
• Published: 02/23/2017
• Est. Priority Date: 08/20/2015
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Taylor, Brandon V., Chen, Henry Yao-Tsu, Swift, Mark Robert, Lee, Austin S., Menezes, Ryan S.
• Application Number:
  US15/138,927
• Publication Number:
  US 20170053445A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

Augmented reality apparatus comprises stereoscopic display apparatus, a computer interface, and a rendering module. The stereoscopic display apparatus is arranged to provide to a user of the augmented reality apparatus a view of a real-world environment in which the user is located. The display apparatus is configured to generate a stereoscopic image that is visible to the user simultaneously with the real-world view. The computer interface is configured to receive from a network externally generated 3D model data of the real-world environment in which the user is located. The rendering module is configured to use the externally generated 3D model data to control the display apparatus to render a virtual element in a manner such that it is perceived by the user as a 3D element located at a desired location in the real-world environment.

Claims

1. Augmented reality apparatus comprising:

stereoscopic display apparatus arranged to provide to a user of the augmented reality apparatus a view of a real-world environment in which the user is located, wherein the display apparatus is configured to generate a stereoscopic image that is visible to the user simultaneously with the real-world view;

a computer interface configured to receive from a network externally generated 3D model data of the real-world environment in which the user is located; and

a rendering module configured to use the externally generated 3D model data to control the display apparatus to render a virtual element in a manner such that it is perceived by the user as a 3D element located at a desired location in the real-world environment.

2. Augmented reality apparatus according to claim 1, further comprising:

memory; and

sensor apparatus for detecting surface structure in the real-world environment, wherein the sensor apparatus is configured to generate in the memory local 3D model data of the real-world environment that embodies surface structure of the real-world environment, and to update the local 3D model data to embody new surface structure as it become detectable due to movement of the sensor apparatus within the environment;

wherein the externally-generated 3D model data received from the network is used to render the virtual element in said manner at least when the surface structure embodied in the local 3D model data is insufficient for the local 3D model data to be used thus.

3. Augmented reality apparatus according to claim 1, wherein the computer interface is configured to retrieve via the network the externally generated 3D model data from a server.

4. Augmented reality apparatus according to claim 3, wherein the externally generated 3D model is retrieved from the server by transmitting a location of the real-world environment from the augmented reality apparatus to the server for comparison thereat with location data associated with the externally generated 3D model data.

5. Augmented reality apparatus according to claim 3, wherein the computer interface is configured to transmit via the network the local 3D model data to the server for use by the server in generating an updated version of the externally generated 3D model data.

6. Augmented reality apparatus according to claim 5, wherein the local 3D model data is transmitted in association with location information and/or timing information pertaining to the local 3D model data.

7. Augmented reality apparatus according to claim 1, wherein the rendering module is configured to receive element location data defining the desired location from the or another network.

8. Augmented reality apparatus according to claim 7, wherein the element location data is received via the or the other network from another user in a real-time communication event between the user and the other user, or retrieved by the augmented reality apparatus from a server.

9. Augmented reality apparatus according to claim 7, wherein the element location data is received in a virtual element data object that defines the virtual element, the data object also comprising:

structural data defining 3D surface structure of the virtual element, and/or image data defining 2D surface texture of the virtual element.

10. Augmented reality apparatus according to claim 1, wherein the augmented reality apparatus is configured determine the desired location based on user inputs received from the user of the augmented reality apparatus.

11. Augmented reality apparatus according to claim 1, wherein the rendering module is configured to apply at least one rendering criteria to the externally generated 3D model to detect surface structure in the real world environment that the user would expect to exert an influence on the virtual element, wherein the rendering of the virtual element is based on said detection.

12. Augmented reality apparatus according to claim 11, wherein the at least one influence criteria is based on a location and/or an orientation of the detected surface structure as indicated by the externally generated 3D model data;

and/or a location and/or an orientation of the display apparatus.

13. Augmented reality apparatus according to claim 11, wherein the rendering of the virtual element based on said detection comprises at least one of:

suppressing part of the virtual element so that surface structure detected at a location between the virtual element and the user is perceived as partially obscuring the virtual element;

applying an animation effect to the virtual element so that it is perceived as interacting dynamically with detected surface structure; and

modifying the desired location at which the virtual element is perceived to be located to account for detected surface structure.

14. Augmented reality apparatus according to claim 1, wherein the element location data defines the desired location relative to a known piece of surface structure embodied by the externally generated 3D model data.

15. Augmented reality apparatus according to claim 1, wherein at least the display apparatus is embodied in a wearable headset.

16. Augmented reality apparatus according to claim 1, wherein the display apparatus comprises two substantially transparent viewing portions through which the real-world is visible and thereby provides the real-world view, each substantially transparent viewing portion being arranged to project a respective image component of the stereoscopic image onto a respective eye of the user.

17. A computer readable medium storing executable code, the code configured to be executed on an augmented reality apparatus that comprises display apparatus configured to provide a view of a real-world environment in which a user of the augmenter reality apparatus is located, the code configured when so executed to implement the following steps:

controlling the stereoscopic display apparatus of the augmented reality apparatus to generate a stereoscopic image that is visible to the user simultaneously with the real-world view;

receiving from a network externally generated 3D model data of the real-world environment in which the user is located; and

using the externally generated 3D model data to control the display apparatus to render a virtual element in a manner such that it is perceived by the user as a 3D element located at a desired location in the real-world environment.

18. An augmented reality system comprising:

a computer interface configured to receive a plurality of local 3D models of a part of a real-world environment, each having been generated by a different augmented reality device when located in the real-world environment; and

a global modelling module configured to combine the local 3D models to generate a global 3D model representing a greater portion of the real-environment than any of the local 3D models individually, and to transmit at least part of the global 3D model to a target augmented reality device currently located in the real-world environment for use in rendering a virtual element thereat.

19. An augmented reality system according to claim 18, wherein the global modelling module is configured to transmit at least part of the global 3D model to a device remote from the real-world environment, thereby allowing the remote device to explore and augment a virtual version of the real-world environment remotely;

wherein the augmented reality system also comprises an augmentation sharing model configured to;

receive rendering data from the remote device, the rendering data for rendering a virtual object at the target augmented reality device, and transmit the rendering data to the target augmented reality device.

20. An augmented reality system according to claim 19, wherein the global 3D model is transmitted to the remote device in response to a download request from the remote device, and/or the rendering data is transmitted to the target augmented reality device on response to a download request from the augmented reality device.