CONSTRUCTING AUGMENTED REALITY ENVIRONMENT WITH PRE-COMPUTED LIGHTING

• US 20140125668A1
• Filed: 11/05/2012
• Published: 05/08/2014
• Est. Priority Date: 11/05/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Steed, Jonathan, Krauss, Aaron, Scavezze, Mike, Zhang, Wei, Tomlin, Arthur, Ambrus, Tony, Mount, Brian, Latta, Stephen, Hastings, Ryan
• Granted Patent: US 9,524,585 B2
• Time in Patent Office: Days
• US Class Current: 345/426
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering

Abstract

Embodiments related to efficiently constructing an augmented reality environment with global illumination effects are disclosed. For example, one disclosed embodiment provides a method of displaying an augmented reality image via a display device. The method includes receiving image data, the image data capturing an image of a local environment of the display device, and identifying a physical feature of the local environment via the image data. The method further includes constructing an augmented reality image of a virtual structure for display over the physical feature in spatial registration with the physical feature from a viewpoint of a user, the augmented reality image comprising a plurality of modular virtual structure segments arranged in adjacent locations to form the virtual structure feature, each modular virtual structure segment comprising a pre-computed global illumination effect, and outputting the augmented reality image to the display device.

Claims

1. In a display device, a method of displaying an augmented reality image comprising lighting effects, the method comprising:

receiving image data, the image data capturing an image of a local environment of the display device;

identifying a physical feature of the local environment via the image data;

constructing an augmented reality image of a virtual structure for display over the physical feature in spatial registration with the physical feature from a viewpoint of a user, the augmented reality image comprising a plurality of modular virtual structure segments arranged in adjacent locations to form the virtual structure feature, each modular virtual structure segment comprising a pre-computed global illumination effect; and

outputting the augmented reality image to the display device.

2. The method of claim 1, wherein identifying a physical feature of the local environment comprises performing a mesh analysis of the local environment.

3. The method of claim 1, wherein the physical feature comprises one or more of a wall and a ceiling.

4. The method of claim 1, wherein the physical feature comprises a non-structural object within the local environment.

5. The method of claim 1, wherein the physical feature comprises an empty space within the local environment.

6. The method of claim 1, wherein the pre-computed global illumination effect comprises a pre-computed directional lighting effect.

7. The method of claim 1, wherein the pre-computed global illumination effect comprises a pre-computed radiance transfer function.

8. The method of claim 1, further comprising identifying a lighting characteristic of the local environment via the image data, and modulating an appearance of the plurality of modular virtual structure segments based upon the lighting characteristic of the local environment.

9. The method of claim 8, wherein the lighting characteristic comprises a color characteristic of the local environment, and wherein modulating the appearance of the plurality of modular virtual structure segments comprises imparting the color characteristic to the modular virtual structure segments.

10. The method of claim 8, wherein the lighting characteristic comprises a location of a physical light in the local environment, and wherein modulating the appearance of the plurality of modular virtual structure segments comprises computing a light effect arising from a virtual point light at the location of the physical light.

11. The method of claim 1, wherein constructing the augmented reality imagery of the virtual structure comprises identifying a local lighting characteristic, and selecting a set of modular virtual structure segments determined to have a similar pre-computed lighting effect.

12. The method of claim 1, wherein constructing the augmented reality imagery of the virtual structure comprises applying a connectivity constraint to select adjacent modular virtual structure segments.

13. A computing device, comprising:

a see-through display device;

an image sensor;

a logic subsystem; and

a storage subsystem comprising instructions stored thereon that are executable by the logic subsystem to;

acquire via the image sensor image data capturing an image of a local environment of the computing device;

identify a physical feature of the local environment via the image data;

construct an augmented reality image of a virtual structure for display via the see-through display device over the physical feature in spatial registration with the physical feature from a viewpoint of a user, the augmented reality image comprising a plurality of modular virtual structure segments arranged in adjacent locations in the image to form the virtual structure feature, each modular virtual structure segment comprising a pre-computed global illumination effect; and

output the augmented reality image to the see-through display device.

14. The computing device of claim 13, wherein the instructions are executable to identify a physical feature of the local environment by performing a mesh analysis of the local environment, and to detect the physical feature in the form of one or more of a wall and a ceiling.

15. The computing device of claim 13, wherein the see-through display device is a wearable device, and wherein the image sensor comprises a stereo depth imaging system.

16. The computing device of claim 13, wherein the pre-computed global illumination effect comprises one or more of a pre-computed directional lighting effect and radiance transfer function information.

17. The computing device of claim 13, wherein the instructions are further executable to detect a lighting characteristic of the local environment via the image data, and modulate an appearance of the plurality of modular virtual structure segments based upon the lighting characteristic of the local environment.

18. The computing device of claim 13, wherein the instructions are further executable to detect a local lighting characteristic, and select a set of modular virtual structure segments determined to have a similar pre-computed lighting effect as the local lighting characteristic.

19. A wearable augmented reality computing device, comprising:

a see-through display device;

an image sensor;

a logic subsystem; and

a storage subsystem comprising instructions stored thereon that are executable by the logic subsystem to;

acquire via the image sensor image data capturing an image of a local environment of the computing device;

identify a physical feature of the local environment via the image data;

detect a lighting characteristic of the local environment via the image data;

construct an augmented reality image of a virtual structure for display via the see-through display device over the physical feature in spatial registration with the physical feature from a viewpoint of a user, the augmented reality image comprising a plurality of modular virtual structure segments arranged in adjacent locations in the image to form the virtual structure feature, each modular virtual structure segment comprising a pre-computed global illumination effect;

modulate an appearance of the plurality of modular virtual structure features based upon the lighting characteristic of the local environment; and

output the augmented reality image to the see-through display device.

20. The computing device of claim 19, wherein the lighting characteristic comprises one or more of a color characteristic of the local environment and a location of a physical light in the local environment, and wherein modulating the appearance of the plurality of modular virtual structure segments comprises computing a light effect arising from a virtual point light at the location of the physical light.