PHOTOMETRIC REGISTRATION FROM ARBITRARY GEOMETRY FOR AUGMENTED REALITY

• US 20130271625A1
• Filed: 07/31/2012
• Published: 10/17/2013
• Est. Priority Date: 04/12/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Schmalstieg, Dieter, Gruber, Lukas, Richter-Trummer, Thomas
• Granted Patent: US 9,578,226 B2
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 15/80 Shading
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

Photometric registration from an arbitrary geometry for augmented reality is performed using video frames of an environment captured by a camera. A surface reconstruction of the environment is generated. A pose is determined for the camera with respect to the environment, e.g., using model based tracking using the surface reconstruction. Illumination data for the environment is determined from a video frame. Estimated lighting conditions for the environment are generated based on the surface reconstruction and the illumination data. For example, the surface reconstruction may be used to compute the possible radiance transfer, which may be compressed, e.g., using spherical harmonic basis functions, and used in the lighting conditions estimation. A virtual object may then be rendered based on the lighting conditions. Differential rendering may be used with lighting solutions from the surface reconstruction of the environment and a second surface reconstruction of the environment combined with the virtual object.

Claims

1. A method comprising:

receiving a sequence of video frames of an environment at a mobile device;

generating a surface reconstruction of the environment;

determining a pose of the camera with respect to the environment;

generating illumination data of the environment from at least one video frame;

generating estimated lighting conditions of the environment in each video frame based on the surface reconstruction and the illumination data; and

rendering a virtual object over the video frames based on the pose and the estimated lighting conditions.

2. The method of claim 1, wherein generating the estimated lighting conditions comprises:

generating a radiance transfer for the environment based on the surface reconstruction;

generating a compressed transfer function of the radiance transfer; and

using the compressed transfer function of the radiance transfer and the illumination data to generate the estimated lighting conditions.

3. The method of claim 2, wherein generating the compressed transfer function comprises projecting the radiance transfer into a spherical harmonics basis functions.

4. The method of claim 1, wherein rendering the virtual object over the video frames based on the pose and the estimated lighting conditions comprises:

generating a first lighting solution using the estimated lighting conditions and the surface reconstruction of the environment; and

rendering the virtual object over the video frames based on the pose and the first lighting solution.

5. The method of claim 4, wherein rendering the virtual object over the video frames based on the pose and the estimated lighting conditions further comprises:

generating a second surface reconstruction of the environment combined with the virtual object;

generating a second lighting solution using the estimated lighting conditions and the second surface reconstruction; and

rendering the virtual object over the video frames based on the pose, the first lighting solution and the second lighting solution.

6. The method of claim 5, wherein generating the second surface reconstruction of the environment combined with the virtual object comprises:

transforming a voxelized representation of the virtual object based on the pose; and

combining the voxelized representation of the virtual object after transforming with the surface reconstruction of the environment.

7. The method of claim 5, wherein generating the first lighting solution comprises:

generating a first radiance transfer for the environment based on the surface reconstruction;

generating a first compressed transfer function of the first radiance transfer; and

using the first compressed transfer function and the estimated lighting conditions to calculate the first lighting solution;

wherein generating the second lighting solution comprises;

generating a second radiance transfer for the environment combined with the virtual object based on the second surface reconstruction;

generating a second compressed transfer function of the second radiance transfer; and

using the second compressed transfer function and the estimated lighting conditions to calculate the second lighting solution;

wherein rendering the virtual object over the video frames comprises generating a differential rendering based on the first lighting solution and the second lighting solution.

8. The method of claim 1, wherein generating the illumination data comprises:

converting the at least one video frame into intensity components and color components; and

denoising the intensity components to generate the illumination data.

9. The method of claim 1, wherein generating the surface reconstruction of the environment is performed using at least one of multiple video frames, images from a stereo camera, or a depth image from a depth camera.

10. The method of claim 1, wherein determining the pose of the camera with respect to the environment uses the surface reconstruction.

11. An apparatus comprising:

a camera;

a display; and

a processor coupled to receive a sequence of video frames of an environment captured by the camera, wherein the processor is configured to generate a surface reconstruction of the environment;

determine a pose of the camera with respect to the environment;

generate illumination data of the environment from at least one video frame;

generate estimated lighting conditions of the environment in each video frame based on the surface reconstruction and the illumination data; and

render a virtual object over the video frames based on the pose and the estimated lighting conditions.

12. The apparatus of claim 11, wherein the processor is configured to generate the estimated lighting conditions by being configured to generate a radiance transfer for the environment based on the surface reconstruction;

generate a compressed transfer function of the radiance transfer; and

use the compressed transfer function of the radiance transfer and the illumination data to generate the estimated lighting conditions.

13. The apparatus of claim 12, wherein the processor is configured to generate the compressed transfer function by being configured to project the radiance transfer into a spherical harmonics basis function.

14. The apparatus of claim 11, wherein the processor is configured to render the virtual object over the video frames based on the pose and the estimated lighting conditions by being configured to generate a first lighting solution with the estimated lighting conditions and the surface reconstruction of the environment and render the virtual object over the video frames based on the pose and the first lighting solution.

15. The apparatus of claim 14, wherein the processor is configured to render the virtual object over the video frames based on the pose and the estimated lighting conditions by being further configured to generate a second surface reconstruction of the environment combined with the virtual object;

estimate a second lighting solution with the estimated lighting conditions and the second surface reconstruction; and

render the virtual object over the video frames based on the pose, the first lighting solution and the second lighting solution.

16. The apparatus of claim 15, wherein the processor is configured to generate the second surface reconstruction of the environment combined with the virtual object by being configured to transform a voxelized representation of the virtual object based on the pose;

and combine the voxelized representation of the virtual object after transforming with the surface reconstruction of the environment.

17. The apparatus of claim 15, wherein the processor is configured to generate the first lighting solution by being to:

generate a first radiance transfer for the environment based on the surface reconstruction;

generate a first compressed transfer function of the first radiance transfer; and

use the first compressed transfer function and the estimated lighting conditions to calculate the first lighting solution; and

the processor is configured to generate the second lighting solution by being configured to;

generate a second radiance transfer for the environment combined with the virtual object based on the second surface reconstruction;

generate a second compressed transfer function of the second radiance transfer; and

use the second compressed transfer function and the estimated lighting conditions to calculate the second lighting solution;

wherein the processor is configured to render the virtual object over the video frames by being configured to generate a differential rendering based on the first lighting solution and the second lighting solution.

18. The apparatus of claim 11, wherein the processor is configured to generate the illumination data by being configured to convert the at least one video frame into intensity components and color components and to denoise the intensity components to generate the illumination data.

19. The apparatus of claim 11, wherein the processor is configured to generate the surface reconstruction of the environment using at least one of multiple video frames, images from a stereo camera, or a depth image from a depth camera.

20. The apparatus of claim 11, wherein the processor is configured to determine the pose of the camera with respect to the environment using the surface reconstruction.

21. An apparatus comprising:

means for receiving a sequence of video frames of an environment;

means for generating a surface reconstruction of the environment;

means for determining a pose with respect to the environment;

means for generating illumination data of the environment from at least one video frame;

means for generating estimated lighting conditions of the environment in each video frame based on the surface reconstruction and the illumination data; and

means for rendering a virtual object over the video frames based on the pose and the estimated lighting conditions.

22. The apparatus of claim 21, wherein the means for rendering the virtual object over the video frames based on the pose and the estimated lighting conditions generates a first lighting solution using the estimated lighting conditions and the surface reconstruction of the environment and renders the virtual object over the video frames based on the pose and the first lighting solution.

23. The apparatus of claim 22, wherein the means for rendering the virtual object over the video frames based on the pose and the estimated lighting conditions further generates a second surface reconstruction of the environment combined with the virtual object;

generates a second lighting solution using the estimated lighting conditions and the second surface reconstruction; and

renders the virtual object over the video frames based on the pose, the first lighting solution, and the second lighting solution.

24. The apparatus of claim 23, wherein the means for rendering the virtual object over the video frames based on the pose and the estimated lighting conditions generates the second surface reconstruction of the environment combined with the virtual object by transforming a voxelized representation of the virtual object based on the pose;

and combining the voxelized representation of the virtual object after transforming with the surface reconstruction of the environment.

25. The apparatus of claim 23, wherein the means for rendering the virtual object over the video frames based on the pose and the estimated lighting conditions generates the first lighting solution by generating a first radiance transfer for the environment based on the surface reconstruction;

generating a first compressed transfer function of the first radiance transfer; and

using the first compressed transfer function and the estimated lighting conditions to calculate the first lighting solution; and

the means for rendering the virtual object over the video frames based on the pose and the estimated lighting conditions estimates the second lighting solution by generating a second radiance transfer for the environment combined with the virtual object based on the second surface reconstruction;

generating a second compressed transfer function of the second radiance transfer; and

using the second compressed transfer function of the second radiance transfer and the estimated lighting conditions to generate the second lighting solution; and

wherein the means for rendering the virtual object over the video frames generates a differential rendering based on the first lighting solution and the second lighting solution.

26. The apparatus of claim 21, wherein the means for generating the illumination data converts the at least one video frame into intensity components and color components and denoises the intensity components to generate the illumination data.

27. A storage medium including program code stored thereon, comprising:

program code to generate a surface reconstruction of an environment using at least one video frame of the environment captured with a camera;

program code to determine a pose of the camera with respect to the environment;

program code to generate illumination data of the environment from the at least one video frame;

program code to generate estimated lighting conditions of the environment in each video frame based on the surface reconstruction and the illumination data; and

program code to render a virtual object over the video frames based on the pose and the estimated lighting conditions.

28. The storage medium of claim 27, wherein the program code to render the virtual object over the video frames based on the pose and the estimated lighting conditions comprises:

program code to generate a first lighting solution using the estimated lighting conditions and the surface reconstruction; and

program code to render the virtual object over the video frames based on the pose and the first lighting solution.

29. The storage medium of claim 28, wherein the program code to render the virtual object over the video frames based on the pose and the estimated lighting conditions further comprises:

program code to generate a second surface reconstruction of the environment combined with the virtual object;

program code to generate a second lighting solution using the estimated lighting conditions and the second surface reconstruction; and

program code to render the virtual object over the video frames based on the pose, the first lighting solution, and the second lighting solution.

30. The storage medium of claim 29, wherein the program code to generate the first lighting solution comprises:

program code to generate a first radiance transfer for the environment based on the surface reconstruction;

program code to generate a first compressed transfer function of the first radiance transfer; and

program code to use the first compressed transfer function and the estimated lighting conditions to calculate the first lighting solution;

wherein the program code to generate the second lighting solution comprises

program code to generate a second radiance transfer for the environment combined with the virtual object based on the second surface reconstruction;

program code to generate a second compressed transfer function of the second radiance transfer; and

program code to use the second compressed transfer function and the estimated lighting conditions to calculate the second lighting solution;

wherein the program code to render the virtual object over the video frames comprises program code to generate a differential rendering based on the first lighting solution and the second lighting solution.

31. The storage medium of claim 27, wherein the program code to generate the illumination data comprises program code to convert the at least one video frame into intensity components and color components and program code to noise the intensity components to generate the illumination data.