Methods and apparatus for rich image capture with focused plenoptic cameras

• US 8,345,144 B1
• Filed: 11/02/2010
• Issued: 01/01/2013
• Est. Priority Date: 07/15/2009
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Georgiev, Todor G., Lumsdaine, Andrew
• Primary Examiner(s): Moe, Aung S
• Assistant Examiner(s): DAGNEW, MEKONNEN D
• Application Number:
  US12/917,984
• Time in Patent Office: 791 Days
• Field of Search: 348/335, 348345-351, 348/340, 396/89, 396/268, 396/104, 359/368, 359/619, 250/208.1
• US Class Current: 348/340
• CPC Class Codes: G02B 3/005 arranged along a single direction only, e.g. lenticular sheets G02B3/0043 takes precedence
• CPC Class Codes: G03B 11/00 Filters or other obturators specially adapted for photographic purposes
• CPC Class Codes: G03B 17/00 Details of cameras or camera bodies; Accessories therefor lens hoods or caps G03B11/04
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/84 for processing colour signals
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules

Abstract

Methods and apparatus for rich image capture using focused plenoptic camera technology. A radiance camera employs focused plenoptic camera technology and includes sets of modulating elements that may be used to modulate the sampling of different aspects of the range of plenoptic data. The radiance camera, via the modulating elements, may capture a particular property of light, such as luminance, color, polarization, etc., differently in different microimages or in different portions of microimages. With the focused plenoptic camera technology, the microimages are captured at the same time in a single image. Thus, multiple microimages of the same image of a scene may be captured at different exposures, different colors, different polarities, and so on, in a single image at the same time. Captured images may be used, for example, in High Dynamic Range (HDR) imaging, spectral imaging, polarization imaging, 3D imaging, and other imaging applications.

Claims

1. A camera, comprising:

a photosensor configured to capture light projected onto the photosensor;

an objective lens, wherein the objective lens is configured to refract light from a scene located in front of the camera to form an image of the scene at an image plane of the objective lens;

a microlens array positioned between the objective lens and the photosensor, wherein the microlens array comprises a plurality of microlenses, wherein the plurality of microlenses are focused on the image plane and not on the objective lens, wherein each microlens of the microlens array is configured to project a separate microimage of the image of the scene formed at the image plane by the objective lens onto a separate location on the photosensor; and

a modulating element array comprising two or more modulating elements, wherein the modulating element array affects a property of light passing through the modulating array, wherein at least one of the modulating elements in the modulating element array affects the property of light differently than at least one other modulating element in the modulating element array to thus modulate the property of light in the separate microimages projected onto the separate locations on the photosensor by the microlens array.

2. The camera as recited in claim 1, wherein the modulating element array is proximate to the objective lens of the camera so that the modulating element array affects the property of light proximate to the objective lens.

3. The camera as recited in claim 2, wherein the modulating element array is located in front of the objective lens between the objective lens and the scene.

4. The camera as recited in claim 2, wherein the modulating element array is located behind the objective lens between the objective lens and the image plane.

5. The camera as recited in claim 2, wherein each of the separate microimages of the image of the scene projected onto the separate locations on the photosensor by the microlens array includes two or more modulated portions, wherein each modulated portion corresponds to a respective one of the modulating elements in the modulating element array.

6. The camera as recited in claim 1, wherein the modulating element array is integrated in the objective lens, wherein the objective lens is divided into two or more portions, and wherein each portion of the objective lens corresponds to a respective one of the modulating elements.

7. The camera as recited in claim 1, wherein each of the plurality of microlenses includes an aperture, wherein the apertures are approximately identical, wherein the modulating element array is located proximate to the microlens array so that the modulating element array affects the property of light proximate to the microlens array, and wherein each modulating element in the modulating element array corresponds to a particular one of the plurality of microlenses.

8. The camera as recited in claim 7, wherein the modulating element array is located between the microlens array and the photosensor.

9. The camera as recited in claim 1, wherein the modulating element array is the microlens array, wherein the modulating elements are the microlenses in the array, and wherein the modulation is provided by differently positioning different ones of the microlenses relative to a main optical axis of the camera or by interleaving microlenses with differing optical characteristics in the microlens array.

10. The camera as recited in claim 1, wherein the property of light is luminance, and wherein the modulating elements are neutral density filters that filter the light at two or more different levels.

11. The camera as recited in claim 1, wherein the property of light is color, and wherein the modulating elements are color filters that filter the light according to two or more different wavelengths.

12. The camera as recited in claim 1, wherein the property of light is polarity, and wherein the modulating elements are polarizing filters that filter the light according to two or more different polarities.

13. The camera as recited in claim 1, wherein the property of light is focal length, and wherein the modulating elements provide different focal lengths so that different portions of the image of the scene are in focus at the image plane.

14. The camera as recited in claim 1, wherein the modulating elements are sheets of glass with different refractive indexes to thus provide different optical paths for the light passing through the different sheets of glass.

15. The camera as recited in claim 1, wherein the photosensor is configured to capture a flat comprising the separate microimages, wherein each of the separate microimages is in a separate region of the flat, and wherein at least two of the separate microimages or at least two portions of each microimage differ according to the different effects of the modulating elements.

16. The camera as recited in claim 1, wherein the camera is a small camera with an aperture of approximately 1 millimeter or less at the objective lens.

17. The camera as recited in claim 16, wherein the property of light is luminance, wherein the modulating elements are neutral density filters that filter the light at two or more different levels, wherein the camera is configured to capture a flat comprising the separate microimages, and wherein the captured flat is configured to be processed according to one or more rendering methods to generate a high dynamic range (HDR) image of the scene.

18. A method, comprising:

receiving light from a scene at an objective lens of a camera;

refracting light from the objective lens to form an image of the scene at an image plane of the objective lens;

receiving light from the image plane at a microlens array located between the objective lens and a photosensor of the camera and parallel to the image plane, wherein the microlens array comprises a plurality of microlenses, wherein each of the plurality of microlenses is focused on the image plane and not on the objective lens;

receiving light from the microlens array at the photosensor, wherein the photosensor receives a separate microimage of the image of the scene formed at the image plane by the objective lens from each microlens of the array at a separate location on the photosensor; and

modulating a property of the light at a modulating element array comprising two or more modulating elements, wherein at least one of the modulating elements in the modulating element array affects the property of light differently than at least one other modulating element in the modulating element array to thus modulate the property of light in the separate microimages projected onto the separate locations on the photosensor by the microlens array.

19. The method as recited in claim 18, wherein the modulating element array is proximate to the objective lens of the camera so that the modulating element array modulates the property of the light proximate to the objective lens.

20. The method as recited in claim 18, wherein each of the plurality of microlenses includes an aperture, wherein the apertures are approximately identical, wherein the modulating element array is located proximate to the microlens array so that the modulating element array modulates the property of light proximate to the microlens array, and wherein each modulating element in the modulating element array corresponds to a particular one of the plurality of microlenses.