Constructing augmented reality environment with pre-computed lighting

• US 10,229,544 B2
• Filed: 11/16/2017
• Issued: 03/12/2019
• Est. Priority Date: 11/05/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Steed, Jonathan, Krauss, Aaron, Scavezze, Mike, Zhang, Wei, Tomlin, Arthur, Ambrus, Tony, Mount, Brian, Latta, Stephen, Hastings, Ryan
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US15/815,634
• Publication Number:
  US 20180075663A1
• Time in Patent Office: 481 Days
• Field of Search: None
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering

Abstract

Embodiments related to efficiently constructing an augmented reality environment with global illumination effects are disclosed. For example, one disclosed embodiment provides a method of displaying an augmented reality image via a display device. The method includes receiving image data, the image data capturing an image of a local environment of the display device, and identifying a physical feature of the local environment via the image data. The method further includes constructing an augmented reality image of a virtual structure for display over the physical feature in spatial registration with the physical feature from a viewpoint of a user, the augmented reality image comprising a plurality of modular virtual structure segments arranged in adjacent locations to form the virtual structure feature, each modular virtual structure segment comprising a pre-computed global illumination effect, and outputting the augmented reality image to the display device.

Claims

1. A physical computer-readable storage device comprising stored instructions executable by a computing system to display an augmented reality image by:

receiving image data acquired by an image sensor, the image data capturing an image of a local environment of the image sensor;

identifying a physical surface of the local environment via the image data;

constructing an image of a virtual structure for display over the surface in spatial registration with the surface by fitting a first modular image segment of the virtual structure to the surface based on a geometry of the surface by one or more of scaling, rotating, clipping, and morphing the first modular image segment, and then positioning a second modular image segment of the virtual structure adjacent to the first modular image segment to form an appearance of the virtual structure, each modular image segment having an appearance of a portion of the virtual structure; and

outputting the image for display by a display device.

2. The physical computer-readable storage device of claim 1, wherein the first and second modular image segments comprise a pre-computed global illumination effect.

3. The physical computer-readable storage device of claim 2, wherein the pre-computed global illumination effect comprises a pre-computed directional lighting effect.

4. The physical computer-readable storage device of claim 2, wherein the pre-computed global illumination effect comprises a pre-computed radiance transfer function.

5. The physical computer-readable storage device of claim 1, wherein the instructions executable to identify the physical surface of the local environment are executable to perform a mesh analysis of the local environment.

6. The physical computer-readable storage device of claim 1, wherein the physical surface comprises one or more of a wall and a ceiling.

7. The physical computer-readable storage device of claim 1, wherein the physical surface comprises a non-structural object within the local environment.

8. The physical computer-readable storage device of claim 1, wherein the instructions are further executable to identify a lighting characteristic of the local environment via the image data, and modulate an appearance of the first and second modular image segments based upon the lighting characteristic of the local environment.

9. The physical computer-readable storage device of claim 8, wherein the lighting characteristic comprises a color characteristic of the local environment, and wherein the instructions executable to modulate the appearance of the first and second modular image segments are executable to impart the color characteristic to the modular image segments.

10. The physical computer-readable storage device of claim 8, wherein the lighting characteristic comprises a location of a physical light in the local environment, and wherein the instructions executable to modulate the appearance of the first and second modular image segments are executable to compute a light effect arising from a virtual point light at the location of the physical light.

11. The physical computer-readable storage device of claim 1, wherein the instructions executable to construct the image of the virtual structure are executable to identify a local lighting characteristic, and select a set of modular image segments determined to have a similar pre-computed lighting effect.

12. The physical computer-readable storage device of claim 1, wherein the instructions executable to construct the image of the virtual structure are executable to apply a connectivity constraint to select adjacent modular image segments.

13. A physical computer-readable storage device comprising stored instructions executable by a computing system to display an augmented reality image by:

receiving image data acquired by an image sensor, the image data capturing an image of a local environment of the image sensor;

identifying a physical feature of the local environment via the image data, the physical feature comprising a furniture structure;

constructing an image of a virtual structure for display over the physical feature in spatial registration with the physical feature by fitting a first modular image segment of the virtual structure to the feature based on a geometry of the feature by one or more of scaling, rotating, clipping, and morphing the first modular image segment, and then positioning a second modular image segment of the virtual structure adjacent to the first modular image segment to form an appearance of the virtual structure, each modular image segment having an appearance of a portion of the virtual structure; and

outputting the image for display by a display device.

14. The physical computer-readable storage device of claim 13, wherein the instructions executable to identify a physical feature of the local environment are executable to perform a mesh analysis of the local environment.

15. The physical computer-readable storage device of claim 13, wherein the first and second modular image segments comprise a pre-computed global illumination effect.

16. The physical computer-readable storage device of claim 15, wherein the pre-computed global illumination effect comprises one or more of a pre-computed directional lighting effect and radiance transfer function information.

17. The physical computer-readable storage device of claim 13, wherein the instructions are further executable to detect a lighting characteristic of the local environment via the image data, and modulate an appearance of the first and second modular image segments based upon the lighting characteristic of the local environment.

18. The physical computer-readable storage device of claim 13, wherein the instructions are further executable to detect a local lighting characteristic, and select a set of modular image segments determined to have a similar pre-computed lighting effect as the local lighting characteristic.

19. A system for displaying an augmented reality image, the system comprising:

an image sensor;

a logic machine; and

a storage machine holding instructions executable by the logic machine to

receive image data acquired by the image sensor, the image data capturing an image of a local environment of the image sensor;

identify a physical surface of the local environment via the image data;

construct an image of a virtual structure for display over the surface in spatial registration with the surface by fitting a first modular image segment of the virtual structure to the surface based on a geometry of the surface by one or more of scaling, rotating, clipping, and morphing the first modular image segment, and then positioning a second modular image segment of the virtual structure adjacent to the first modular image segment to form an appearance of the virtual structure, each modular image segment having an appearance of a portion of the virtual structure; and

output the image for display by a display device.

20. The system of claim 19, wherein the instructions are further executable to detect a lighting characteristic of the local environment via the image data, and to modulate an appearance of the first and second modular image segments based upon the lighting characteristic of the local environment.