Super-resolution with the focused plenoptic camera

• US 8,315,476 B1
• Filed: 01/20/2010
• Issued: 11/20/2012
• Est. Priority Date: 01/20/2009
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Georgiev, Todor G., Lumsdaine, Andrew
• Primary Examiner(s): Seth, Manav
• Application Number:
  US12/690,869
• Time in Patent Office: 1,035 Days
• Field of Search: None
• US Class Current: 382/276
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Methods and apparatus for super-resolution in focused plenoptic cameras. By examining the geometry of data capture for super-resolution with the focused plenoptic camera, configurations for which super-resolution is realizable at different modes in the focused plenoptic camera are generated. A focused plenoptic camera is described in which infinity is super resolved directly, with registration provided by the camera geometry and the microlens pitch. In an algorithm that may be used to render super-resolved images from flats captured with a focused plenoptic camera, a high-resolution observed image is generated from a flat by interleaving pixels from adjacent microlens images. A deconvolution method may then be applied to the high-resolution observed image to deblur the image.

Claims

1. A camera, comprising:

a photosensor configured to capture light projected onto the photosensor, wherein the photosensor comprises a plurality of pixels;

an objective lens, wherein the objective lens is configured to refract light from a scene located in front of the camera to form an image of the scene at an image plane of the objective lens; and

a microlens array positioned between the objective lens and the photosensor, wherein the microlens array comprises a plurality of microlenses, wherein the plurality of microlenses are focused on the image plane and not on the objective lens, and wherein each microlens samples a respective region of the image of the scene formed at the image plane by the objective lens, and wherein each region of the image of the scene sampled by the microlenses partially overlaps at least one other region of the image of the scene sampled by the microlenses;

wherein each microlens of the microlens array is configured to project a respective region of the image of the scene sampled by the microlens onto a separate location on the photosensor, wherein each pixel of each said location on the photosensor samples radiance from a particular area of the image of the scene, and wherein the areas of the image of the scene sampled by the pixels of each said location on the photosensor partially overlap the areas of the image of the scene sampled by the pixels of at least one other location on the photosensor onto which an overlapping region of the image of the scene is projected;

wherein the photosensor is configured to capture a flat that includes each of the regions of the image of the scene projected onto the photosensor by the microlenses in a separate microimage in the flat; and

wherein the flat is configured to be processed by a super-resolution technique that interleaves pixels in neighboring microimages according to a subpixel shift that indicates the amount of overlap of the pixels to render a high-resolution image of the scene.

2. The camera as recited in claim 1, wherein the super-resolution technique applies deconvolution to the high-resolution image of the scene to deblur the high-resolution image of the scene.

3. The camera as recited in claim 1, wherein the subpixel shift is determined according to configuration values of the focused plenoptic camera.

4. The camera as recited in claim 3, wherein the configuration values include pitch of the microlenses in the microlens array, distance from the microlenses to the photosensor, and distance from the microlenses to the image plane of the objective lens.

5. The camera as recited in claim 1, wherein the image plane of the objective lens is at one focal length from the objective lens and at depth infinity relative to the microlenses.

6. The camera as recited in claim 1, wherein the camera is configured to store the captured flat to a memory device.

7. A method, comprising:

obtaining a flat comprising a plurality of separate portions of an image of a scene, wherein each of the plurality of portions is in a separate region of the flat, wherein each portion of the image of the scene partially overlaps at least one other portion of the image of the scene, and wherein areas of the image of the scene sampled at pixels of each said portion partially overlap areas of the image sampled by pixels of at least one other portion that partially overlaps said portion;

determining a subpixel shift that indicates the amount of overlap of the pixels in the portions of the image; and

interleaving pixels in neighboring portions of the image according to the subpixel shift to render a high-resolution image of the scene.

8. The method as recited in claim 7, further comprising applying deconvolution to the interleaved image of the scene to deblur the high-resolution image of the scene.

9. The method as recited in claim 7, wherein said determining a subpixel shift that indicates the amount of overlap of the pixels in the portions of the image comprises determining the subpixel shift according to configuration values of a focused plenoptic camera used to capture the flat.

10. The method as recited in claim 9, wherein the configuration values include pitch of microlenses in a microlens array of the focused plenoptic camera, distance from the microlenses to a photosensor of the focused plenoptic camera, and distance from the microlenses to an image plane of an objective lens of the focused plenoptic camera at which the flat was captured.

11. The method as recited in claim 10, wherein the image plane of the objective lens is at one focal length from the objective lens and at depth infinity relative to the microlenses.

12. The method as recited in claim 7, wherein said determining a subpixel shift that indicates the amount of overlap of the pixels in the portions of the image comprises determining the subpixel shift according to an analysis of at least two of the separate regions of the flat that include overlapping portions of the image of the scene.

13. The method as recited in claim 7, wherein said interleaving pixels in neighboring portions of the image according to the subpixel shift to render a high-resolution image of the scene comprises:

for each portion of the image, interleaving the pixels of said portion with the pixels of (L_S²−

1) neighboring portions that overlap said portion to generate an interleaved subimage, where L_S is a level of super-resolution determined according to the subpixel shift; and

appropriately assembling the interleaved subimages to generate the high-resolution image of the scene.

14. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

obtaining a flat comprising a plurality of separate portions of an image of a scene, wherein each of the plurality of portions is in a separate region of the flat, wherein each portion of the image of the scene partially overlaps at least one other portion of the image of the scene, and wherein areas of the image of the scene sampled at pixels of each said portion partially overlap areas of the image sampled by pixels of at least one other portion that partially overlaps said portion;

determining a subpixel shift that indicates the amount of overlap of the pixels in the portions of the image; and

interleaving pixels in neighboring portions of the image according to the subpixel shift to render a high-resolution image of the scene.

15. The non-transitory computer-readable storage medium as recited in claim 14, wherein the program instructions are computer-executable to implement applying deconvolution to the interleaved image of the scene to deblur the high-resolution image of the scene.

16. The non-transitory computer-readable storage medium as recited in claim 14, wherein, in said determining a subpixel shift that indicates the amount of overlap of the pixels in the portions of the image, the program instructions are computer-executable to implement determining the subpixel shift according to configuration values of a focused plenoptic camera used to capture the flat.

17. The non-transitory computer-readable storage medium as recited in claim 16, wherein the configuration values include pitch of microlenses in a microlens array of the focused plenoptic camera, distance from the microlenses to a photosensor of the focused plenoptic camera, and distance from the microlenses to an image plane of an objective lens of the focused plenoptic camera at which the flat was captured.

18. The non-transitory computer-readable storage medium as recited in claim 17, wherein the image plane of the objective lens is at one focal length from the objective lens and at depth infinity relative to the microlenses.

19. The non-transitory computer-readable storage medium as recited in claim 14, wherein, in said determining a subpixel shift that indicates the amount of overlap of the pixels in the portions of the image, the program instructions are computer-executable to implement determining the subpixel shift according to an analysis of at least two of the separate regions of the flat that include overlapping portions of the image of the scene.

20. The non-transitory computer-readable storage medium as recited in claim 14, wherein, in said interleaving pixels in neighboring portions of the image according to the subpixel shift to render a high-resolution image of the scene, the program instructions are computer-executable to implement:

for each portion of the image, interleaving the pixels of said portion with the pixels of (L_S²−

1) neighboring portions that overlap said portion to generate an interleaved subimage, where L_S is a level of super-resolution determined according to the subpixel shift; and

appropriately assembling the interleaved subimages to generate the high-resolution image of the scene.