Methods and Apparatus for Rendering Focused Plenoptic Camera Data using Super-Resolved Demosaicing

• US 20130128068A1
• Filed: 11/30/2010
• Published: 05/23/2013
• Est. Priority Date: 08/27/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Georgiev, Todor G., Chunev, Georgi N.
• Granted Patent: US 8,749,694 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

A super-resolved demosaicing technique for rendering focused plenoptic camera data performs simultaneous super-resolution and demosaicing. The technique renders a high-resolution output image from a plurality of separate microimages in an input image at a specified depth of focus. For each point on an image plane of the output image, the technique determines a line of projection through the microimages in optical phase space according to the current point and angle of projection determined from the depth of focus. For each microimage, the technique applies a kernel centered at a position on the current microimage intersected by the line of projection to accumulate, from pixels at each microimage covered by the kernel at the respective position, values for each color channel weighted according to the kernel. A value for a pixel at the current point in the output image is computed from the accumulated values for the color channels.

Claims

1. A method, comprising:

rendering an output image of a scene from a plurality of separate microimages of an image of the scene at a depth of focus that defines an angle of projection in an optical phase space, the rendering comprising, for a point on an image plane of the output image;

determining a line of projection through the microimages in optical phase space according to the point and the angle of projection determined from the depth of focus;

for each microimage, applying a kernel centered at a position on the microimage intersected by the line of projection to accumulate, from pixels at each of the plurality microimage covered by the kernel at the respective position, values for each color channel accumulated calculated separately according to the kernel; and

computing a value for a pixel at the point in the output image from the values for the color channels that were accumulated and calculated separately.

2. The method as recited in claim 1, wherein the values for each color channel at each microimage are weighted according to distance of the pixels covered by the kernel from the position on the microimage intersected by the line of projection.

3. The method as recited in claim 1, wherein said computing a value for a pixel at the point in the output image from the accumulated values for the color channels comprises computing a value for each color channel of the pixel in the output image from the accumulated value for a corresponding color channel.

4. The method as recited in claim 3, wherein said computing a value for each color channel of the pixel in the output image from the accumulated value for a corresponding color channel comprises normalizing the accumulated values according to a normalization metric for the kernel.

5. The method as recited in claim 1, wherein the kernel is a Gaussian kernel.

6. The method as recited in claim 1, wherein the input image is captured according to a photosensor technology that captures a pattern of pixels in multiple separate color channels.

7. The method as recited in claim 6, wherein the photosensor is a Bayer array photosensor.

8. The method as recited in claim 1, wherein the pixels in the input image are RGB technology pixels.

9. The method as recited in claim 1, further comprising applying a deconvolution technique to the output image.

10. The method as recited in claim 1, further comprising:

obtaining a different depth of focus, wherein the different depth of focus determines a different angle of projection in optical phase space; and

repeating said rendering at the different depth of focus to generate a second output image at the different depth of focus.

11. A computer-readable storage medium excluding signals and comprising program instructions stored thereon, the program instructions are computer-executable to cause operations to be performed comprising:

rendering an output image of a scene from a plurality of separate microimages of an image of the scene at a depth of focus that defines an angle of projection in an optical phase space, the rendering comprising, for point on an image plane of the output image comprising;

determining a line of projection through the microimages in optical phase space according to the point and the angle of projection determined from the depth of focus;

for each microimage, applying a kernel centered at a position on the microimage intersected by the line of projection to accumulate, from pixels at each of the plurality of microimages covered by the kernel at the respective position, values for each color channel accumulated and calculated separately according to the kernel; and

computing a value for a pixel at the point in the output image from the values for the color channels that were accumulated and calculated separately.

12. The computer-readable storage medium as recited in claim 11, wherein the values for each color channel at each microimage are weighted according to distance of the pixels covered by the kernel from the position on the microimage intersected by the line of projection.

13. The computer-readable storage medium as recited in claim 11, wherein, in said computing a value for a pixel at the point in the output image from the accumulated values for the color channels, the program instructions are computer-executable to implement computing a value for each color channel of the pixel in the output image from the accumulated value for a corresponding color channel.

14. The computer-readable storage medium as recited in claim 11, wherein the input image is captured according to a photosensor technology that captures a pattern of pixels in multiple separate color channels.

15. The computer-readable storage medium as recited in claim 11, wherein the program instructions are computer-executable to implement:

obtaining a different depth of focus, wherein the different depth of focus determines a different angle of projection in optical phase space; and

repeating said rendering at the different depth of focus to generate a second high resolution output image at the different depth of focus.

16. A system, comprising

at least one processor;

and

a memory comprising program instructions that are executable by the at least one processor to;

obtain an input image comprising a plurality of separate microimages of an image of a scene, the input image includes pixels in multiple separate color channels;

obtain a depth of focus for an output image to be rendered from the input image, the depth of focus defining an angle of projection in optical phase space; and

render an output image of the scene from the plurality of separate microimages at the depth of focus, the rendering comprising, for a point on an image plane of the output image;

determine a line of projection through the microimages in optical phase space according to the and the angle of projection determined from the depth of focus;

for each microimage, apply a kernel centered at a position on the microimage intersected by the line of projection to accumulate, from pixels at each of the plurality of microimages covered by the kernel at the respective position, values for each color channel accumulated and calculated separately according to the kernel; and

compute a value for a pixel at the point in the output image from the values for the color channels that were accumulated and calculated separately.

17. The system as recited in claim 16, wherein the values for each color channel at each microimage are weighted according to distance of the pixels covered by the kernel from the position on the microimage intersected by the line of projection.

18. The system as recited in claim 16, wherein, to compute a value for a pixel at the current point in the output image from the accumulated values for the color channels, the program instructions are executable by the at least one processor to compute a value for each color channel of the pixel in the output image from the accumulated value for a corresponding color channel.

19. The system as recited in claim 16, wherein the program instructions are executable by the at least one processor to:

obtain a different depth of focus, wherein the different depth of focus determines a different angle of projection in optical phase space; and

repeat said rendering at the different depth of focus to generate a second output image at the different depth of focus.

20. The system as recited in claim 16, wherein the at least one processor includes at least one graphics processing unit (GPU).