Methods and apparatus for full-resolution light-field capture and rendering

• US 8,189,065 B2
• Filed: 05/28/2009
• Issued: 05/29/2012
• Est. Priority Date: 01/23/2008
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Georgiev, Todor G., Lumsdaine, Andrew
• Primary Examiner(s): Perkey, W. B.
• Application Number:
  US12/474,112
• Publication Number:
  US 20090295829A1
• Time in Patent Office: 1,097 Days
• Field of Search: 348/222.1, 382282-284
• US Class Current: 348/222.1
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G03B 15/00 Special procedures for taking photographs; Apparatus therefor
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 35/24 using apertured or refractive resolving means on screens or between screen and eye
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

Method and apparatus for full-resolution light-field capture and rendering. A radiance camera is described in which the microlenses in a microlens array are focused on the image plane of the main lens instead of on the main lens, as in conventional plenoptic cameras. The microlens array may be located at distances greater than f from the photosensor, where f is the focal length of the microlenses. Radiance cameras in which the distance of the microlens array from the photosensor is adjustable, and in which other characteristics of the camera are adjustable, are described. Digital and film embodiments of the radiance camera are described. A full-resolution light-field rendering method may be applied to flats captured by a radiance camera to render higher-resolution output images than are possible with conventional plenoptic cameras and rendering methods.

Claims

1. A computer-implemented method, comprising:

obtaining a flat comprising a plurality of separate portions of an image of a scene, wherein each of the plurality of separate portions is in a separate region of the flat, wherein each of the plurality of separate portions comprises a plurality of pixels, and wherein the flat is a 2D representation of a 4D light-field that captures both spatial and angular information of the scene;

generating a plurality of m₁×

m₂ pixel subregions from the plurality of separate portions, wherein said generating comprises cropping each of the plurality of separate portions to an m₁×

m₂ pixel subregion of the respective portion, where at least one of m₁ and m₂ is an integer greater than or equal to two; and

assembling the plurality of subregions to produce a single image of the scene.

2. The method as recited in claim 1, where m₁ is equal to m₂.

3. The method as recited in claim 1, wherein each of the plurality of separate portions is substantially circular.

4. The method as recited in claim 1, wherein said assembling the plurality of subregions comprises moving the subregions together so that features of the image of the scene from any given subregion substantially match with features of adjacent subregions.

5. The method as recited in claim 1, wherein said assembling the plurality of subregions comprises enlarging each of the subregions so that features of the image of the scene from any given subregion substantially match with features of adjacent subregions.

6. The method as recited in claim 1, further comprising, prior to said cropping, for each of two or more areas of the flat:

examining each of two or more of the plurality of separate portions in the area to determine a direction of movement of edges in the two or more separate portions in the area relative to centers of the portions, wherein said examining starts at a portion in the area and proceeds in a direction across the area, and wherein an edge is a feature of the image of the scene that appears in one or more of the portions; and

if the direction of movement of the edges in the two or more separate portions in the area is the same as the direction across the area at which the two or more separate portions are examined, inverting each of the two or more separate portions in the area relative to their respective centers.

7. The method as recited in claim 1, further comprising, prior to said cropping:

examining each of two or more of the plurality of separate portions to determine a direction of movement of edges within the two or more separate portions, wherein said examining is performed in a direction, and wherein an edge is a feature of the image of the scene that appears in one or more of the portions;

detecting that the direction of movement of edges in the two or more separate portions is the same as the direction in which said examining is performed; and

inverting at least the two or more separate portions relative to their respective centers in response to said detecting.

8. The method as recited in claim 1, wherein said obtaining a flat comprises:

receiving light from the scene at an objective lens of a camera;

refracting light from the objective lens to form an image of the scene at an image plane of the objective lens;

receiving light from the image plane at a microlens array located between the objective lens and a photosensor of the camera, wherein the microlens array comprises a plurality of microlenses, and wherein the plurality of microlenses are focused on the image plane and not on the objective lens;

receiving light from the microlens array at the photosensor, wherein the photosensor receives a separate portion of the image of the scene formed at the image plane by the objective lens from each microlens of the microlens array at a separate location on the photosensor; and

capturing the flat of the scene at the photosensor, wherein the flat includes each of the plurality of separate portions of the image of the scene in a separate region of the flat.

9. A system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

obtain a flat comprising a plurality of separate portions of an image of a scene, wherein each of the plurality of separate portions is in a separate region of the flat, wherein each of the plurality of separate portions comprises a plurality of pixels, and wherein the flat is a 2D representation of a 4D light-field that captures both spatial and angular information of the scene;

crop each of the plurality of separate portions to an m₁×

m₂ pixel subregion of the respective portion to generate a plurality of m₁×

m₂ pixel subregions from the plurality of separate portions, where at least one of m₁ and m₂ is an integer greater than or equal to two; and

assemble the plurality of subregions to produce a single image of the scene.

10. The system as recited in claim 9, where m₁ is equal to m₂.

11. The system as recited in claim 9, wherein each of the plurality of separate portions is substantially circular.

12. The system as recited in claim 9, wherein, to assemble the plurality of subregions, the program instructions are executable by the at least one processor to move the subregions together so that features of the image of the scene from any given subregion substantially match with features of adjacent subregions.

13. The system as recited in claim 9, wherein, to assemble the plurality of subregions, the program instructions are executable by the at least one processor to enlarge each of the subregions so that features of the image of the scene from any given subregion substantially match with features of adjacent subregions.

14. The system as recited in claim 9, wherein the program instructions are executable by the at least one processor to, prior to said crop, for each of two or more areas of the flat:

examine each of two or more of the plurality of separate portions in the area to determine a direction of movement of edges in the two or more separate portions in the area relative to centers of the portions, wherein said examining starts at a portion in the area and proceeds in a direction across the area, and wherein an edge is a feature of the image of the scene that appears in one or more of the portions;

if the direction of movement of the edges in the two or more separate portions in the area is the same as the direction across the area at which the two or more separate portions are examined, invert each of the two or more separate portions in the area relative to their respective centers.

15. The system as recited in claim 9, wherein the program instructions are executable by the at least one processor to, prior to said crop:

examine each of two or more of the plurality of separate portions to determine a direction of movement of edges within the two or more separate portions, wherein said examining is performed in a direction, and wherein an edge is a feature of the image of the scene that appears in one or more of the portions;

detect that the direction of movement of edges in the two or more separate portions is the same as the direction in which said examining is performed; and

invert at least the two or more separate portions relative to their respective centers in response to said detecting.

16. The system as recited in claim 9, wherein the system further comprises:

a camera, comprising;

a photosensor configured to capture light projected onto the photosensor;

an objective lens configured to refract light from in front of the camera to form an image at an image plane of the objective lens; and

a microlens array positioned between the objective lens and the photosensor, wherein the microlens array comprises a plurality of microlenses, wherein the plurality of microlenses are focused on the image plane and not on the objective lens, wherein each microlens of the microlens array is configured to project a separate portion of the image formed at the image plane by the objective lens onto a separate location on the photosensor;

wherein the flat is a single flat of the scene captured by the photosensor of the camera.

17. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

obtaining a flat comprising a plurality of separate portions of an image of a scene, wherein each of the plurality of separate portions is in a separate region of the flat, wherein each of the plurality of separate portions comprises a plurality of pixels, and wherein the flat is a 2D representation of a 4D light-field that captures both spatial and angular information of the scene;

generating a plurality of m₁×

m₂ pixel subregions from the plurality of separate portions, wherein said generating comprises cropping each of the plurality of separate portions to an m₁×

m₂ pixel subregion of the respective portion, where at least one of m₁ and m₂ is an integer greater than or equal to two; and

assembling the plurality of subregions to produce a single high-resolution image of the scene.

18. The non-transitory computer-readable storage medium as recited in claim 17, where m₁ is equal to m₂.

19. The non-transitory computer-readable storage medium as recited in claim 17, wherein each of the plurality of separate portions is substantially circular.

20. The non-transitory computer-readable storage medium as recited in claim 17, wherein, in said assembling the plurality of subregions, the program instructions are computer-executable to implement moving the subregions together so that features of the image of the scene from any given subregion substantially match with features of adjacent subregions.

21. The non-transitory computer-readable storage medium as recited in claim 17, wherein, in said assembling the plurality of subregions, the program instructions are computer-executable to implement enlarging each of the subregions so that features of the image of the scene from any given subregion substantially match with features of adjacent subregions.

22. The non-transitory computer-readable storage medium as recited in claim 17, wherein the program instructions are computer-executable to implement, prior to said cropping, for each of two or more areas of the flat:

examining each of two or more of the plurality of separate portions in the area to determine a direction of movement of edges in the two or more separate portions in the area relative to centers of the portions, wherein said examining starts at a portion in the area and proceeds in a direction across the area, and wherein an edge is a feature of the image of the scene that appears in one or more of the portions;

if the direction of movement of the edges in the two or more separate portions in the area is the same as the direction across the area at which the two or more separate portions are examined, inverting each of the two or more separate portions in the area relative to their respective centers.

23. The non-transitory computer-readable storage medium as recited in claim 17, wherein the program instructions are computer-executable to implement, prior to said cropping:

examining each of two or more of the plurality of separate portions to determine a direction of movement of edges within the two or more separate portions, wherein said examining is performed in a direction, and wherein an edge is a feature of the image of the scene that appears in one or more of the portions;

detecting that the direction of movement of edges in the two or more separate portions is the same as the direction in which said examining is performed; and

inverting at least the two or more separate portions relative to their respective centers in response to said detecting.

24. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

obtaining a flat comprising a plurality of separate portions of an image of a scene, wherein each of the plurality of separate portions is in a separate region of the flat, wherein each of the plurality of separate portions comprises a plurality of pixels, and wherein the flat is a 2D representation of a 4D light-field that captures both spatial and angular information of the scene;

examining each of two or more of the plurality of separate portions to determine a direction of movement of edges within the two or more separate portions, wherein said examining is performed in a direction, and wherein an edge is a feature of the image of the scene that appears in one or more of the portions;

detecting that the direction of movement of edges in the two or more separate portions is the same as the direction in which said examining is performed;

inverting at least the two or more separate portions relative to their respective centers in response to said detecting; and

assembling the plurality of separate portions to produce a single high-resolution image of the scene.