Augmented reality

• US 10,643,394 B2
• Filed: 12/16/2018
• Issued: 05/05/2020
• Est. Priority Date: 08/20/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chen, Henry Yao-Tsu, Taylor, Brandon V., Swift, Mark Robert, Lee, Austin S., Menezes, Ryan S.
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US16/221,567
• Publication Number:
  US 20190122442A1
• Time in Patent Office: 506 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment

Abstract

In a device including a processor and a memory in communication with the processor is described, the memory includes executable instructions that, when executed by the processor, cause the processor to control the device to perform functions of: generating, based on a plurality of local 3D models, a global 3D model representing a portion of a real-world environment; determining a location of a 3D virtual object in the global 3D model; and generating augmentation data for rendering the 3D virtual object to be seen at a location of the real-world environment corresponding to the location of the 3D virtual object in the global 3D model.

Claims

1. A device for generating supplemental surface structure data for a first remote device located at a real-world environment, comprising:

a processor; and

a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to control the device to perform functions of;

receiving, from a plurality of second remote devices located at the real-world environment, a plurality of local 3D models of the real-world environment, each local 3D model representing a surface structure of a different portion of the real-world environment captured by each respective second remote device;

generating, based on the plurality of local 3D models, a global 3D model representing a surface structure of the portions of the real-world environment captured by the second remote devices;

generating augmentation data for rendering a 3D virtual object to be placed at a location in the global 3D model; and

sending, to the first remote device located at the real-world environment, (i) the argumentation data for rendering the 3D virtual object and (ii) global 3D model data representing at least a portion of the global 3D model such that the 3D virtual object is rendered by the first remote device based on the global 3D model data even when a surface structure of the real-world environment captured by the first remote device is not sufficient for the first remote device to render the 3D virtual object.

2. The device of claim 1, wherein, for generating the global 3D model, the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform a function of combining the received local 3D models.

3. The device of claim 1, wherein the plurality of second remote devices is located at a plurality of different locations, respectively, of the real-world environment.

4. The device of claim 1, wherein the portion of the real-world environment represented by the global 3D model is greater than a portion of the real-world environment represented by any of the local 3D models.

5. The device of claim 1, wherein the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform functions of:

receiving, from at least one of the second remote devices, an updated local 3D model or additional local 3D model; and

updating the global 3D model based on the received updated local 3D model or additional local 3D model.

6. The device of claim 1, wherein:

for generating the augmentation data, the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform a function of receiving, from the second remote device, local argumentation data for creating the 3D virtual object in the local 3D model generated by the second remote device, and

the location of the 3D virtual object in the global 3D model is determined based on the received local argumentation data.

7. A method of operating a device for generating supplemental surface structure data for a first remote device located at a real-world environment, the method comprising:

receiving, from a plurality of second remote devices located at the real-world environment, a plurality of local 3D models of the real-world environment, each local 3D model representing a surface structure of a different portion of the real-world environment captured by each respective second remote device;

generating, based on the plurality of local 3D models, a global 3D model representing a surface structure of the portions of the real-world environment captured by the second remote devices;

generating augmentation data for rendering a 3D virtual object to be placed at a location in the global 3D model; and

sending, to the first remote device located at the real-world environment, (i) the argumentation data for rendering the 3D virtual object and (ii) global 3D model data representing at least a portion of the global 3D model such that the 3D virtual object is rendered by the first remote device based on the global 3D model data even when a surface structure of the real-world environment captured by the first remote device is not sufficient for the first remote device to render the 3D virtual object.

8. The method of claim 7, wherein generating the global 3D model comprises combining the received local 3D models.

9. The method of claim 7, wherein the plurality of second remote devices is located at a plurality of different locations, respectively, of the real-world environment.

10. The method of claim 7, wherein the portion of the real-world environment represented by the global 3D model is greater than a portion of the real-world environment represented by any of the local 3D models.

11. The method of claim 7, further comprising:

receiving, from at least one of the second remote devices, an updated local 3D model or additional local 3D model; and

updating the global 3D model based on the received updated local 3D model or additional local 3D model.

12. The method of claim 7, wherein:

generating the augmentation data comprises receiving, from the second remote device, local argumentation data for creating the 3D virtual object in the local 3D model generated by the second remote device, and

the location of the 3D virtual object in the global 3D model is determined based on the received local argumentation data.

13. A non-transitory computer readable medium containing instructions for generating supplemental surface structure data for a first remote device located at a real-world environment, the instructions, when executed by a processor, causing a device to perform functions of:

receiving, from a plurality of second remote devices located at the real-world environment, a plurality of local 3D models of the real-world environment, each local 3D model representing a surface structure of a different portion of the real-world environment captured by each respective second remote device;

generating, based on the plurality of local 3D models, a global 3D model representing a surface structure of the portions of the real-world environment captured by the second remote devices;

generating augmentation data for rendering a 3D virtual object to be placed at a location in the global 3D model; and

sending, to the first remote device located at the real-world environment, (i) the argumentation data for rendering the 3D virtual object and (ii) global 3D model data representing at least a portion of the global 3D model such that the 3D virtual object is rendered by the first remote device based on the global 3D model data even when a surface structure of the real-world environment captured by the first remote device is not sufficient for the first remote device to render the 3D virtual object.

14. The non-transitory computer readable medium of claim 13, wherein for generating the global 3D model, the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform a function of combining the received local 3D models.

15. The non-transitory computer readable medium of claim 13, wherein the plurality of second remote devices is located at a plurality of different locations, respectively, of the real-world environment.

16. The non-transitory computer readable medium of claim 13, wherein the portion of the real-world environment represented by the global 3D model is greater than a portion of the real-world environment represented by any of the local 3D models.

17. The non-transitory computer readable medium of claim 13, wherein the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform functions of:

receiving, from at least one of the second remote devices, an updated local 3D model or additional local 3D model; and

updating the global 3D model based on the received updated local 3D model or additional local 3D model.

18. The non-transitory computer readable medium of claim 13, wherein:

for generating the augmentation data, the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform a function of receiving, from the second remote device, local argumentation data for creating the 3D virtual object in the local 3D model generated by the second remote device, and

the location of the 3D virtual object in the global 3D model is determined based on the received local argumentation data.