Augmented Reality

• US 20190122442A1
• Filed: 12/16/2018
• Published: 04/25/2019
• Est. Priority Date: 08/20/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chen, Henry Yao-Tsu, Taylor, Brandon V., Swift, Mark Robert, Lee, Austin S., Menezes, Ryan S.
• Granted Patent: US 10,643,394 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment

Abstract

In a device including a processor and a memory in communication with the processor is described, the memory includes executable instructions that, when executed by the processor, cause the processor to control the device to perform functions of: generating, based on a plurality of local 3D models, a global 3D model representing a portion of a real-world environment; determining a location of a 3D virtual object in the global 3D model; and generating augmentation data for rendering the 3D virtual object to be seen at a location of the real-world environment corresponding to the location of the 3D virtual object in the global 3D model.

Claims

1. A device comprising:

a processor; and

a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to control the device to perform functions of;

generating, based on a plurality of local 3D models, a global 3D model representing a portion of a real-world environment;

determining a location of a 3D virtual object in the global 3D model; and

generating augmentation data for rendering the 3D virtual object to be seen at a location of the real-world environment corresponding to the location of the 3D virtual object in the global 3D model.

2. The device of claim 1, wherein, to generate the global 3D model, the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform functions of:

receiving, via a communication network, the plurality of local 3D models from a plurality of remote devices, respectively; and

combining the received local 3D models to generate the global 3D model of the portion of the real-world environment.

3. The device of claim 2, wherein the plurality of remote devices is located at a plurality of different locations, respectively, of the real-world environment.

4. The device of claim 2, wherein each local 3D model represents a different portion of the real-world environment captured by each remote device.

5. The device of claim 2, wherein the portion of the real-world environment represented by the global 3D model is greater than a portion of the real-world environment represented by any of the local 3D models.

6. The device of claim 2, wherein the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform functions of:

receiving, from at least one of the remote devices, an updated local 3D model or additional local 3D model; and

updating the global 3D model based on the received updated local 3D model or additional local 3D model.

7. The device of claim 1, wherein the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform a function of receiving, from a remote device via a communication network, local argumentation data for creating the 3D virtual object at a location in a local 3D model generated by the remote device, wherein the location of the 3D virtual object in the global 3D model is determined based on the received local argumentation data.

8. The device of claim 1, wherein the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform functions of:

transmitting, to a remote device via a communication network, at least a portion of the global 3D model; and

transmitting, to the remote device via the communication network, the augmentation data for rendering, at the remote device, the 3D virtual object to be seen at the location of the real-world environment corresponding to the location of the 3D virtual object in the global 3D model.

9. A method comprising:

generating, based on a plurality of local 3D models, a global 3D model representing a portion of a real-world environment;

determining a location of a 3D virtual object in the global 3D model; and

generating augmentation data for rendering the 3D virtual object to be seen at a location of the real-world environment corresponding to the location of the 3D virtual object in the global 3D model.

10. The method of claim 9, wherein the generating the global 3D model comprises:

receiving, via a communication network, the plurality of local 3D models from a plurality of remote devices, respectively; and

combining the received local 3D models to generate the global 3D model of the portion of the real-world environment.

11. The method of claim 10, wherein the plurality of remote devices is located at a plurality of different locations, respectively, of the real-world environment.

12. The method of claim 10, wherein each local 3D model represents a different portion of the real-world environment captured by each remote device.

13. The method of claim 10, wherein the portion of the real-world environment represented by the global 3D model is greater than a portion of the real-world environment represented by any of the local 3D models.

14. The method of claim 10, further comprising:

receiving, from at least one of the remote devices, an updated local 3D model or additional local 3D model; and

updating the global 3D model based on the received updated local 3D model or additional local 3D model.

15. The method of claim 9, further comprising receiving, from a remote device via a communication network, local argumentation data for creating the 3D virtual object at a location in a local 3D model generated by the remote device, wherein the location of the 3D virtual object in the global 3D model is determined based on the received local argumentation data.

16. The method of claim 9, further comprising:

transmitting, to a remote device via a communication network, at least a portion of the global 3D model; and

transmitting, to the remote device via the communication network, the augmentation data for rendering, at the remote device, the 3D virtual object to be seen at the location of the real-world environment corresponding to the location of the 3D virtual object in the global 3D model.

17. A device comprising:

means for generating, based on a plurality of local 3D models, a global 3D model representing a portion of a real-world environment;

means for determining a location of a 3D virtual object in the global 3D model; and

means for generating augmentation data for rendering the 3D virtual object to be seen at a location of the real-world environment corresponding to the location of the 3D virtual object in the global 3D model.

18. The device of claim 17, wherein the generating means comprises:

means for receiving, via a communication network, the plurality of local 3D models from a plurality of remote devices, respectively; and

means for combining the received local 3D models to generate the global 3D model of the portion of the real-world environment.

19. The device of claim 17, further comprising:

means for receiving, from a remote device via a communication network, local argumentation data for creating the 3D virtual object at a location in a local 3D model generated by the remote device, wherein the location of the 3D virtual object in the global 3D model is determined based on the received local argumentation data.

20. The device of claim 17, further comprising:

means for transmitting, to a remote device via a communication network, at least a portion of the global 3D model; and

means for transmitting, to the remote device via the communication network, the augmentation data for rendering, at the remote device, the 3D virtual object to be seen at the location of the real-world environment corresponding to the location of the 3D virtual object in the global 3D model.