APPARATUS AND METHOD FOR RADIANCE TRANSFER SAMPLING FOR AUGMENTED REALITY

• US 20140320530A1
• Filed: 02/25/2014
• Published: 10/30/2014
• Est. Priority Date: 04/24/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Schmalstieg, Dieter, GRUBER, Lukas Jr.
• Granted Patent: US 10,062,210 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 15/55 Radiosity
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 15/80 Shading
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering

Abstract

Methods, systems, computer-readable media, and apparatuses for radiance transfer sampling for augmented reality are presented. In some embodiments, a method includes receiving at least one video frame of an environment. The method further includes generating a surface reconstruction of the environment. The method additionally includes projecting a plurality of rays within the surface reconstruction of the environment. Upon projecting a plurality of rays within the surface reconstruction of the environment, the method includes generating illumination data of the environment from the at least one video frame. The method also includes determining a subset of rays from the plurality of rays in the environment based on areas within the environment needing refinement. The method further includes rendering the virtual object over the video frames based on the plurality of rays excluding the subset of rays.

Claims

1. A method, comprising:

receiving at least one video frame of an environment;

generating a surface reconstruction of the environment;

projecting a plurality of rays within the surface reconstruction of the environment;

generating illumination data of the environment from the at least one video frame;

determining a subset of rays from the plurality of rays in the environment based on areas within the environment needing refinement; and

rendering a virtual object over the at least one video frame based on the plurality of rays excluding the subset of rays.

2. The method of claim 1 wherein the areas within the environment needing refinement comprise shadow areas, surface edges, and shadow edges.

3. The method of claim 1 further comprising generating estimated real world lighting data of the environment in the at least one video frame based on the surface reconstruction and the illumination data.

4. The method of claim 1 wherein the determining a subset of rays from the plurality of rays comprises applying lighting and differential rendering techniques to the environment and analyzing shadow information as a result thereof.

5. The method of claim 1 wherein the determining a subset of rays from the plurality of rays comprises applying adaptive sampling techniques based on at least one of a 4-D subdivision of a ray space comprising the plurality of rays and an approximated variance analysis.

6. The method of claim 1 further comprising storing temporal coherence information associated with the subset of rays in a two-level hierarchical cache, wherein the first level stores the information in a screen space and the second level stores the information in an object space.

7. The method of claim 1 wherein generating the illumination data comprises:

converting the at least one video frame into intensity components and color components; and

denoising the intensity components to extract the illumination data.

8. The method of claim 1 wherein generating the surface reconstruction of the environment is performed using the at least one video frame.

9. An apparatus, comprising:

a camera;

a memory; and

a processor coupled the memory to receive at least one video frame of an environment captured by the camera, wherein the processor is configured to;

generate a surface reconstruction of the environment;

project a plurality of rays within the surface reconstruction of the environment;

generate illumination data of the environment from the at least one video frame;

determine a subset of rays from the plurality of rays in the environment based on areas within the environment needing refinement; and

render a virtual object over the at least one video frame based on the plurality of rays excluding the subset of rays.

10. The apparatus of claim 9 wherein the areas within the environment needing refinement comprise shadow areas, surface edges, and shadow edges.

11. The apparatus of claim 9 wherein the processor is further configured to generate estimated real world lighting data of the environment in the at least one video frame based on the surface reconstruction and the illumination data.

12. The apparatus of claim 9 wherein the determining a subset of rays from the plurality of rays comprises applying lighting and differential rendering techniques to the environment and analyzing shadow information as a result thereof.

13. The apparatus of claim 9 wherein the determining a subset of rays from the plurality of rays comprises applying adaptive sampling techniques based on at least one of a 4-D subdivision of a ray space comprising the plurality of rays and an approximated variance analysis.

14. The apparatus of claim 9 wherein the processor is further configured to store temporal coherence information associated with the subset of rays in a two-level hierarchical cache, wherein the first level stores the information in a screen space and the second level stores the information in an object space.

15. The apparatus of claim 9 wherein generating the illumination data comprises:

converting the at least one video frame into intensity components and color components; and

denoising the intensity components to extract the illumination data.

16. The apparatus of claim 9 wherein generating the surface reconstruction of the environment is performed using the at least one video frame.

17. An apparatus, comprising:

means for receiving a at least one video frame of an environment;

means for generating a surface reconstruction of the environment;

means for projecting a plurality of rays within the surface reconstruction of the environment;

means for generating illumination data of the environment from the at least one video frame;

means for determining a subset of rays from the plurality of rays in the environment based on areas within the environment needing refinement; and

means for rendering a virtual object over the at least one video frame based on the plurality of rays excluding the subset of rays.

18. The apparatus of claim 17 wherein the areas within the environment needing refinement comprise shadow areas, surface edges, and shadow edges.

19. The apparatus of claim 17 further comprising means for generating estimated real world lighting data of the environment in the at least one video frame based on the surface reconstruction and the illumination data.

20. The apparatus of claim 17 wherein the means for determining a subset of rays from the plurality of rays comprises applying lighting and differential rendering techniques to the environment and analyzing shadow information as a result thereof.

21. The apparatus of claim 17 wherein the means for determining a subset of rays from the plurality of rays comprises applying adaptive sampling techniques based on at least one of a 4-D subdivision of a ray space comprising the plurality of rays and an approximated variance analysis.

22. The apparatus of claim 17 further comprising means for storing temporal coherence information associated with the subset of rays in a two-level hierarchical cache, wherein the first level stores the information in a screen space and the second level stores the information in an object space.

23. The apparatus of claim 17 wherein the means for generating the illumination data comprises:

means for converting the at least one video frame into intensity components and color components; and

means for denoising the intensity components to extract the illumination data.

24. A processor-readable non-transitory medium comprising processor readable instructions configured to cause a processor to:

receive a at least one video frame of an environment;

generate a surface reconstruction of the environment;

project a plurality of rays within the surface reconstruction of the environment;

generate illumination data of the environment from the at least one video frame;

determine a subset of rays from the plurality of rays in the environment based on areas within the environment needing refinement; and

render a virtual object over the at least one video frame based on the plurality of rays excluding the subset of rays.

25. The processor-readable non-transitory medium of claim 24 wherein the areas within the environment needing refinement comprise shadow areas, surface edges, and shadow edges.

26. The processor-readable non-transitory medium of claim 24 wherein the processor readable instructions are further configured to cause the processor to generate estimated real world lighting data of the environment in the at least one video frame based on the surface reconstruction and the illumination data.

27. The processor-readable non-transitory medium of claim 24 wherein the determining a subset of rays from the plurality of rays comprises applying lighting and differential rendering techniques to the environment and analyzing shadow information as a result thereof.

28. The processor-readable non-transitory medium of claim 24 wherein the determining a subset of rays from the plurality of rays comprises applying adaptive sampling techniques based on at least one of a 4-D subdivision of a ray space comprising the plurality of rays and an approximated variance analysis.

29. The processor-readable non-transitory medium of claim 24 wherein the processor readable instructions are further configured to cause the processor to store temporal coherence information associated with the subset of rays in a two-level hierarchical cache, wherein the first level stores the information in a screen space and the second level stores the information in an object space.

30. The processor-readable non-transitory medium of claim 24 wherein generating the illumination data comprises:

converting the at least one video frame into intensity components and color components; and

denoising the intensity components to extract the illumination data.