Methods, Apparatus, and Computer-Readable Storage Media for Blended Rendering of Focused Plenoptic Camera Data

• US 20130120605A1
• Filed: 11/30/2010
• Published: 05/16/2013
• Est. Priority Date: 03/03/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Georgiev, Todor G., Lumsdaine, Andrew
• Granted Patent: US 8,860,833 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/21 involving computational photography

Abstract

Methods, apparatus, and computer-readable storage media for rendering focused plenoptic camera data. A rendering with blending technique is described that blends values from positions in multiple microimages and assigns the blended value to a given point in the output image. A rendering technique that combines depth-based rendering and rendering with blending is also described. Depth-based rendering estimates depth at each microimage and then applies that depth to determine a position in the input flat from which to read a value to be assigned to a given point in the output image. The techniques may be implemented according to parallel processing technology that renders multiple points of the output image in parallel. In at least some embodiments, the parallel processing technology is graphical processing unit (GPU) technology.

Claims

1. A method, comprising:

obtaining a flat comprising a plurality of microimages of separate portions of an image of a scene, wherein each of the plurality of microimages is in a separate region of the flat, wherein the flat is a two-dimensional (2D) representation of a four-dimensional (4D) light-field that captures both spatial and angular information of the scene;

rendering an output image of the scene from the plurality of separate microimages, wherein said rendering comprises, for each point of the output image;

determining a target microimage in the flat for this point;

calculating a position in the target microimage from which a value is to be sampled;

for each microimage in a neighborhood including the target image, reading a value from the calculated position in the respective microimage and adding the value to a value accumulator; and

normalizing the value in the value accumulator and assigning the normalized value to the point in the output image.

2. The method as recited in claim 1, wherein said rendering an output image of the scene from the plurality of separate microimages comprises performing said rendering according to a parallel processing technique that renders multiple points of the output image in parallel.

3. The method as recited in claim 2, wherein the parallel processing technique is implemented according to graphical processing unit (GPU) technology.

4. The method as recited in claim 1, wherein said reading a value from the calculated position in the respective microimage and adding the value to a value accumulator comprises weighting the value read from the respective microimage according to position of the respective microimage in the neighborhood prior to said adding the value to the value accumulator.

5. The method as recited in claim 4, wherein said normalizing the value in the value accumulator comprises normalizing the value in the value accumulator according to sum of the weights of the microimages in the neighborhood.

6. The method as recited in claim 1, wherein said calculating a position in the target microimage from which a value is to be sampled comprises calculating the position within an M×

M region within the microimage.

7. The method as recited in claim 1, further comprising:

prior to said rendering the output image, for each of the plurality of microimages, estimating a depth of a respective portion of the image of the scene;

wherein, in said calculating a position in the target microimage from which a value is to be sampled, the program instructions are computer-executable to implement calculating the position according to the estimated depth corresponding to the target microimage.

8. A computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

obtaining a flat comprising a plurality of microimages of separate portions of an image of a scene, wherein each of the plurality of microimages is in a separate region of the flat, wherein the flat is a two-dimensional (2D) representation of a four-dimensional (4D) light-field that captures both spatial and angular information of the scene;

rendering an output image of the scene from the plurality of separate microimages, wherein said rendering comprises, for each point of the output image;

determining a target microimage in the flat for this point;

calculating a position in the target microimage from which a value is to be sampled;

for each microimage in a neighborhood including the target image, reading a value from the calculated position in the respective microimage and adding the value to a value accumulator; and

normalizing the value in the value accumulator and assigning the normalized value to the point in the output image.

9. The computer-readable storage medium as recited in claim 8, wherein, in said rendering an output image of the scene from the plurality of separate microimages, the program instructions are computer-executable to implement performing said rendering according to a parallel processing technique that renders multiple points of the output image in parallel.

10. The computer-readable storage medium as recited in claim 9, wherein the parallel processing technique is implemented according to graphical processing unit (GPU) technology.

11. The computer-readable storage medium as recited in claim 8, wherein, in said reading a value from the calculated position in the respective microimage and adding the value to a value accumulator, the program instructions are computer-executable to implement weighting the value read from the respective microimage according to position of the respective microimage in the neighborhood prior to said adding the value to the value accumulator.

12. The computer-readable storage medium as recited in claim 11, wherein, in said normalizing the value in the value accumulator, the program instructions are computer-executable to implement normalizing the value in the value accumulator according to sum of the weights of the microimages in the neighborhood.

13. The computer-readable storage medium as recited in claim 8, wherein, in said calculating a position in the target microimage from which a value is to be sampled, the program instructions are computer-executable to implement calculating the position within an M×

M region within the microimage.

14. The computer-readable storage medium as recited in claim 8, wherein the program instructions are computer-executable to implement:

prior to said rendering the output image, for each of the plurality of microimages, estimating a depth of a respective portion of the image of the scene;

wherein, in said calculating a position in the target microimage from which a value is to be sampled, the program instructions are computer-executable to implement calculating the position according to the estimated depth corresponding to the target microimage.

15. A system, comprising

at least one processor;

and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

obtain a flat comprising a plurality of microimages of separate portions of an image of a scene, wherein each of the plurality of microimages is in a separate region of the flat, wherein the flat is a two-dimensional (2D) representation of a four-dimensional (4D) light-field that captures both spatial and angular information of the scene;

render an output image of the scene from the plurality of separate microimages, wherein, to render an output image, the program instructions are computer-executable to implement, for each point of the output image;

determine a target microimage in the flat for this point;

calculate a position in the target microimage from which a value is to be sampled;

for each microimage in a neighborhood including the target image, read a value from the calculated position in the respective microimage and add the value to a value accumulator; and

normalize the value in the value accumulator and assign the normalized value to this point in the output image.

16. The system as recited in claim 15, wherein the at least one processor includes at least one graphical processing unit (GPU), and wherein, to render an output image of the scene from the plurality of separate microimages, the program instructions are computer-executable to implement said rendering according to a parallel processing technique that renders multiple points of the output image in parallel on the at least one GPU.

17. The system as recited in claim 15, wherein, to read a value from the calculated position in the respective microimage and add the value to a value accumulator, the program instructions are executable by the at least one processor to weight the value read from the respective microimage according to position of the respective microimage in the neighborhood prior to said adding the value to the value accumulator.

18. The system as recited in claim 17, wherein, to normalize the value in the value accumulator, the program instructions are executable by the at least one processor to normalize the value in the value accumulator according to sum of the weights of the microimages in the neighborhood.

19. The system as recited in claim 15, wherein, to calculate a position in the target microimage from which a value is to be sampled, the program instructions are executable by the at least one processor to calculate the position within an M×

M region within the microimage.

20. The system as recited in claim 15, wherein the program instructions are executable by the at least one processor to:

prior to said rendering the output image, for each of the plurality of microimages, estimate a depth of a respective portion of the image of the scene;

wherein, to calculate a position in the target microimage from which a value is to be sampled, the program instructions are executable by the at least one processor to calculate the position according to the estimated depth corresponding to the target microimage.