Augmented reality

• US 10,169,917 B2
• Filed: 04/26/2016
• Issued: 01/01/2019
• Est. Priority Date: 08/20/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chen, Henry Yao-Tsu, Taylor, Brandon V., Swift, Mark Robert, Lee, Austin S., Menezes, Ryan S.
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US15/139,095
• Publication Number:
  US 20170053447A1
• Time in Patent Office: 980 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment

Abstract

An augmented reality (AR) system receives a plurality of local 3D models of a part of a real-world environment, each having been generated by a different AR device when located in the real-world environment. The local 3D models are combined to generate a global 3D model, at least part of which is transmitted to a device remote from the real-world environment. The global 3D model represents a greater portion of the real-environment than any of the local 3D models individually. The AR system receives rendering data from the remote device, and transmits it to an AR device when the AR device is located in the real-world environment. Alternatively, the rendering data may be transmitted from the remote device to the AR device via a network directly. The rendering data is for use in rendering a virtual object at the AR device in the real-world environment.

Claims

1. An augmented reality system comprising:

a computer interface configured to receive a plurality of local 3D models of a part of a real-world environment;

a global model generation module configured to combine the local 3D models to generate a global 3D model representing a greater portion of the real-world environment than any of the local 3D models individually, and to-transmit at least part of the global 3D model to a device remote from the real-world environment, thereby allowing the remote device to explore and augment a virtual version of the real-world environment remotely; and

an augmentation sharing module configured to;

receive an augmentation separate from the global 3D model from the remote device, the augmentation including rendering data for use in rendering a 3D virtual object at an augmented reality device in the real-world environment corresponding to a particular location associated with the global 3D model;

transmit at least a portion of the global 3D model to the augmented reality device; and

transmit the augmentation to the augmented reality device when the augmented reality device is located in the real-world environment without modifying the global 3D model to include the augmentation, such that the augmented reality device can utilize the global 3D model to identify the particular location within the real-world environment and render the 3D virtual object at the particular location.

2. An augmented reality system according to claim 1, wherein the rendering data is received from the remote device and transmitted to the augmented reality device in real-time.

3. An augmented reality system according to claim 1, wherein the rendering data is received from the remote device and transmitted to the augmented reality device during a communication event between a user of the remote device and a user of the augmented reality device.

4. An augmented reality system according to claim 3, wherein the computer interface is configured to receive at least one additional local 3D model from at least one additional augmented reality device when located in the real-world environment;

wherein the augmentation sharing module is configured to update the global 3D model, at a time after the communication event has terminated, based on the at least one additional local 3D model, the updated global 3D model for use in a future communication event between users.

5. An augmented reality system according to claim 1, wherein the augmentation sharing module is configured to receive additional rendering data from the augmented reality device and transmit it to the remote device, the additional rendering data for use in rendering another virtual object at the remote device using the global 3D model.

6. An augmented reality system according to claim 5, wherein the additional rendering data is received from the augmented reality device and transmitted to the remote device in real-time.

7. An augmented reality system according to claim 1, wherein the augmentation sharing module is configured to store the rendering data in computer storage of the augmented reality system, and to transmit it to the augmented reality device in response to a download request from the augmented reality device.

8. An augmented reality system according to claim 1, wherein the computer interface is configured to receive at least one additional local 3D model from at least one additional augmented reality device when located in the real-world environment, wherein the augmentation sharing module is configured to update the global 3D model, at a time after the global 3D model has been generated, based on the at least one additional local 3D model.

9. An augmented reality system according to claim 1, wherein each local 3D model is received in association with location information pertaining to that local 3D model, wherein the global model generation module is configured to identify the plurality of local 3D models as pertaining to the same real-world environment based on their associated location information, and select them for use in generating the global 3D model based on said identification.

10. An augmented reality system according to claim 1, wherein each local 3D model is received in association with timing information pertaining to that local 3D model data, wherein the global model generation module is configured to identify the plurality of local 3D models as pertaining to a pre-determined interval of time and select them for use in generating the global 3D model based on said identification.

11. An augmented reality system according to claim 10, wherein the global model generation module is configured to determine an age of at least one of the local 3D models based on its associated timing information and, in response to the age reaching an age threshold, update the global 3D model in order to remove data from the global 3D model that was derived from the at least one local 3D model.

12. An augmented reality system according to claim 5, wherein the additional rendering data associated with the another virtual object is transmitted to the augmented reality device for use in rendering the another virtual object.

13. A method of effecting communication via a network between an augmented reality device in a real-world environment and another device remote from the real-world environment, the method comprising:

receiving a global 3D model, the global 3D model having been generated by combining a plurality of local 3D models of a part of the real-world environment, wherein the global 3D model represents a greater portion of the real-world environment than any of the local 3D models individually;

transmitting at least part of the global 3D model to the remote device, thereby allowing the remote device to explore and augment a virtual version of the real-world environment remotely;

receiving an augmentation separate from the global 3D model from the remote device;

using the global 3D model and the augmentation to generate rendering data for use in rendering a 3D virtual object corresponding to the augmentation at an augmented reality device in the real-world environment corresponding to a particular location associated with the global 3D model; and

transmitting the rendering data and at least a portion of the global 3D model to the augmented reality device via the network when the augmented reality device is located in the real-world environment without modifying the global 3D model to include the augmentation, such that the augmented reality device can utilize the global 3D model to identify the particular location within the real-world environment and render the 3D virtual object at the particular location.

14. A method according to claim 13, wherein the communication is a real-time communication event between a user of the augmented reality device and a user of the remote device conducted via the network, the rendering data being transmitted from the remote device to the augmented reality device in real-time.

15. A method according to claim 14 wherein the rendering data is transmitted to the augmented reality device from the remote device via an augmented reality system, the rendering data being relayed between the devices by the augmented reality system in real-time.

16. A method according to claim 13, wherein the rendering data is transmitted to an augmented reality system for storage thereat.

17. A method according to claim 13, comprising:

receiving at the remote device additional rendering data from the augmented reality device via the network; and

using, by the remote device, the additional rendering data to render an additional virtual element on a display of the remote device.

18. A method according to claim 17, comprising:

receiving a video signal at the remote device from the augmented reality device via the network; and

displaying on the display, by the remote device, a moving image derived from the video signal, wherein the additional virtual element is overlaid on the moving image on the display.

19. A method according to claim 17, wherein the global 3D model is used by the remote device to render the additional virtual element.

20. A communication system comprising:

a device remote from a real-world environment; and

an augmented reality system comprising;

a computer interface configured to receive a plurality of local 3D models of a part of the real-world environment, and a global model generation module configured to combine the local 3D models to generate a global 3D model representing a greater portion of the real-environment than any of the local 3D models individually, and transmit at least part of the global 3D model to the remote device, thereby allowing the remote device to explore and augment a virtual version of the real-world environment remotely;

wherein the remote device is configured to create an augmentation separate from the global 3D model and use the global 3D model to generate rendering data for use in rendering a virtual object corresponding to the augmentation at an augmented reality device in the real-world environment corresponding to a particular location associated with the global 3D model, and to transmit the rendering data and at least a portion of the global 3D model to the augmented reality device when the augmented reality device is located in the real-world environment and/or to the augmented reality system without modifying the global 3D model to include the augmentation, such that the augmented reality device can utilize the global 3D model to identify the particular location within the real-world environment and render the 3D virtual object at the particular location.