Methods and apparatus for light-field imaging

• US 8,290,358 B1
• Filed: 06/23/2008
• Issued: 10/16/2012
• Est. Priority Date: 06/25/2007
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Georgiev, Todor G.
• Primary Examiner(s): Mahoney, Christopher
• Assistant Examiner(s): Smith, Linda B
• Application Number:
  US12/144,411
• Time in Patent Office: 1,576 Days
• Field of Search: 396/326, 396/327, 396/113, 396/333
• US Class Current: 396/326
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses

Abstract

Methods and apparatus for light-field imaging. Light-field camera designs are described that produce higher spatial resolution than conventional plenoptic camera designs, while trading-off the light-field'"'"'s angular sampling density. This lower angular resolution may be compensated for by a light-field image processing method that inserts data synthesized by view interpolation of the measured light-field. In one embodiment, a light-field image processing method that performs three-view morphing may be used to interpolate the missing angular samples of radiance. The light-field camera designs may be implemented in hand-held light-field cameras that may capture a light-field with a single exposure. Some of the light-field camera designs are internal to the camera, while others are external to the camera. One light-field camera design includes a single, relatively large lens and an array of negative lenses that are placed in front of (external to) the main lens of a conventional camera.

Claims

1. An apparatus, comprising:

an optical system comprising;

an objective lens, wherein the objective lens is configured to refract light from a scene located in front of an aperture of the optical system; and

a multidimensional optical array, wherein the optical array comprises a plurality of optical elements, wherein each optical element in the optical array is configured to refract light from the objective lens;

a photosensor configured to capture light projected onto the photosensor from the optical system, wherein the photosensor is positioned proximate to an image plane of the optical system;

wherein the optical array is positioned between the objective lens and the photosensor in the apparatus and proximate to the objective lens so that distance between the objective lens and the optical array is small relative to distance from the optical system to the image plane of the optical system such that the objective lens and the optical array are located at substantially a same optical plane to form the image plane as an optical unit; and

wherein each optical element of the optical array is configured to project a separate image of the scene onto a separate location on the photosensor;

wherein the separate images of the scene projected onto the photosensor by the optical array form a multidimensional array of the separate images that includes radiance information, wherein the radiance information indicates distribution of intensity of the light in multiple directions at multiple points.

2. The apparatus as recited in claim 1, wherein the optical elements in the optical array are positive lenses.

3. The apparatus as recited in claim 1, wherein the optical elements in the optical array are negative lenses.

4. The apparatus as recited in claim 1, wherein the optical elements in the optical array are prisms.

5. The apparatus as recited in claim 1, wherein the photosensor is a charge-coupled device (CCD).

6. The apparatus as recited in claim 1, wherein the photosensor is configured to capture a light-field image that includes each of the separate images of the scene projected onto the photosensor by the plurality of optical elements in a separate region of the light-field image.

7. The apparatus as recited in claim 1, wherein the apparatus is a hand-held camera.

8. An apparatus, comprising:

an optical system comprising;

a single lens, wherein the single lens is configured to refract light from a scene located in front of the single lens;

an optical array, wherein the optical array comprises a plurality of optical elements, wherein each optical element in the optical array is configured to refract light from the single lens; and

an objective lens positioned proximate to an image plane of the optical array, wherein the objective lens is configured to refract light from the optical array;

wherein the optical array is positioned between the single lens and the objective lens and proximate to the single lens so that distance between the single lens and the optical array is small relative to distance from the optical array to the image plane of the optical array such that the single lens and the optical array are located at substantially a same optical plane to form the image plane as an optical unit;

a photosensor configured to capture light projected onto the photosensor from the optical system;

wherein the optical system is configured to project a plurality of separate images of the scene onto the photosensor, wherein each of the plurality of separate images is projected onto a separate location on the photosensor;

wherein the plurality of separate images of the scene projected onto the photosensor by the optical system form an array of the separate images that includes radiance information, wherein the radiance information indicates distribution of intensity of the light in multiple directions at multiple points.

9. The apparatus as recited in claim 8, wherein the optical elements in the optical array are negative lenses.

10. The apparatus as recited in claim 8, wherein the photosensor is a charge-coupled device (CCD).

11. The apparatus as recited in claim 8, wherein the photosensor is configured to capture a light-field image that includes each of the separate images of the scene projected onto the photosensor by the optical system in a separate region of the light-field image.

12. The apparatus as recited in claim 8, wherein the apparatus is a hand-held camera.

13. A method for capturing light-field images, comprising:

receiving light from a scene at an objective lens of a camera;

receiving light from the objective lens at a multidimensional optical array located between the objective lens and a photosensor of the camera, wherein the optical array comprises a plurality of optical elements, and wherein the optical array is proximate to the objective lens so that distance between the objective lens and the optical array is small relative to distance from the optical array to an image plane of the optical array such that the objective lens and the optical array are located at substantially a same optical plane to form the image plane as an optical unit;

receiving light from the optical array at the photosensor, wherein the photosensor is positioned proximate to the image plane, and wherein the photosensor receives a different image of the scene from each optical element in the array at a separate location on the photosensor; and

capturing the different images of the scene at the photosensor as a light-field image;

wherein the different images of the scene projected onto the photosensor by the optical array form a multidimensional array of the different images that includes radiance information, wherein the radiance information indicates distribution of intensity of the light in multiple directions at multiple points.

14. The method as recited in claim 13, wherein the optical elements in the optical array are positive lenses.

15. The method as recited in claim 13, wherein the optical elements in the optical array are negative lenses.

16. The method as recited in claim 13, wherein the optical elements in the optical array are prisms.

17. The method as recited in claim 13, wherein the photosensor is a charge-coupled device (CCD).

18. A method for capturing light-field images, comprising:

receiving light from a scene at a single lens;

receiving light from the single lens at an optical array located between the single lens and an objective lens of a camera, wherein the optical array comprises a plurality of optical elements, and wherein the optical array is proximate to the single lens so that distance between the single lens and the optical array is small relative to distance from the optical array to an image plane of the optical array such that the single lens and the optical array are located at substantially a same optical plane to form the image plane as an optical unit;

receiving light from the optical array at the objective lens of the camera, wherein the objective lens is positioned proximate to the image plane of the optical array;

receiving light from the objective lens at a photosensor of the camera, wherein the photosensor receives a different image of the scene from each optical element in the array; and

capturing the different images of the scene at the photosensor as a light-field image;

wherein the different images of the scene projected onto the photosensor by the optical array form an array of the different images that includes radiance information, wherein the radiance information indicates distribution of intensity of the light in multiple directions at multiple points.

19. The method as recited in claim 18, wherein the optical elements in the optical array are negative lenses.

20. The method as recited in claim 18, wherein the photosensor is a charge-coupled device (CCD).

21. An apparatus, comprising:

a lens, wherein the lens is configured to refract light from a scene located in front of the lens; and

a multidimensional optical array, wherein the optical array comprises a plurality of optical elements, wherein each optical element in the optical array is configured to refract light from the lens;

wherein the optical array is positioned with respect to the lens to produce via light received through the lens a plurality of separate images of the scene at an image plane, and wherein distance between the lens and the optical array is small relative to distance from the optical array to the image plane such that the lens and the optical array are located at substantially a same optical plane to form the image plane as an optical unit;

wherein the separate images of the scene produced at the image plane form a multidimensional array of the separate images that includes radiance information, wherein the radiance information indicates distribution of intensity of the light in multiple directions at multiple points.

22. The apparatus as recited in claim 21, wherein the optical elements in the optical array are positive lenses.

23. The apparatus as recited in claim 21, wherein the optical elements in the optical array are negative lenses.

24. The apparatus as recited in claim 21, wherein the optical elements in the optical array are prisms.

25. The apparatus as recited in claim 21, wherein the apparatus is configured to attach to a camera comprising an objective lens and a photosensor so that the objective lens is positioned proximate to the image plane of the apparatus.

26. The apparatus as recited in claim 25, wherein the apparatus is configured to attach to the camera so that the optical array is positioned between the lens of the apparatus and the objective lens of the camera and the objective lens of the camera is positioned between the apparatus and the photosensor.

27. The apparatus as recited in claim 21, wherein the apparatus is configured to attach to a camera comprising a photosensor, wherein the optical array is positioned between the lens and the photosensor and proximate to the lens, wherein the lens of the apparatus acts as the objective lens of the camera, and wherein the photosensor is positioned proximate to the image plane of the apparatus.

28. An apparatus, comprising:

a multidimensional objective lens array comprising a plurality of discrete objective lenses, wherein each lens in the objective lens array is configured to refract light received directly from a scene located in front of the camera; and

a photosensor configured to capture light projected onto the photosensor from the objective lens array, wherein the lenses in the objective lens array are positioned such that the optical axes of the lenses are parallel to each other and perpendicular to a light-capturing surface of the photosensor;

wherein each lens of the objective lens array receives light from the scene from a different angle than the other lenses in the objective lens array and consequently projects a view of the scene onto the photosensor, whereby the photosensor receives a different view of the scene from each lens in the objective lens array, wherein each different view of the scene is received at a separate location on the photosensor to produce an array of different views of the scene at the photosensor;

wherein the array of different views of the scene includes radiance information, wherein the radiance information indicates distribution of intensity of the light in multiple directions at multiple points; and

wherein the objective lens array and the photosensor are arranged within a single camera; and

a processing module operable to render multiple different output images of the scene from the array of different views of the scene received at the photosensor, each different output image focused at a different depth in the scene or with a different depth of field.

29. The apparatus as recited in claim 28, wherein the processing module is further operable to generate additional different views of the scene from the array of different views of the scene according to a view morphing technique, wherein the additional different views of the scene are combined with the array of different views to render the multiple different output images.

30. The apparatus as recited in claim 28, wherein the plurality of discrete objective lenses are arranged in the objective lens array in two or more rows, wherein adjacent rows are shifted relative to each other such that columns formed by the lenses in the array are non-orthogonal to the rows.

31. The apparatus as recited in claim 28, wherein the lenses are of a geometric shape other than rectangular or square.