4D light field cameras

• US 20080187305A1
• Filed: 02/06/2007
• Published: 08/07/2008
• Est. Priority Date: 02/06/2007
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Raskar, Ramesh, Agrawal, Amit Kumar
• Granted Patent: US 7,792,423 B2
• Time in Patent Office: Days
• US Class Current: 396/268
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus

Abstract

A camera acquires a 4D light field of a scene. The camera includes a lens and sensor. A mask is arranged in a straight optical path between the lens and the sensor. The mask including an attenuation pattern to spatially modulate the 4D light field acquired of the scene by the sensor. The pattern has a low spatial frequency when the mask is arranged near the lens, and a high spatial frequency when the mask is arranged near the sensor.

Claims

1. An apparatus for acquiring a 4D light field of a scene, comprising:

a lens;

a sensor; and

a mask arranged in a straight optical path between the lens and the sensor, the mask including an attenuation pattern to spatially modulate the 4D light field acquired of the scene by the sensor.

2. The apparatus of claim 1, in which the pattern has a low spatial frequency and is arranged near the lens.

3. The apparatus of claim 1, in which the pattern has a high spatial frequency and is arranged near the sensor.

4. The apparatus of claim 3, in which the mask heterodynes the 4D light field.

5. The apparatus of claim 1, in which the sensor produces an input image, and further comprising:

means for reconstructing the 4D light field from the input image.

6. The apparatus of claim 5, in which the 4D light field is reconstructed by Fourier-domain decoding.

7. The apparatus of claim 2, in which the sensor produces a defocused blur input image.

8. The apparatus of claim 2, in which digital refocusing can be achieved without a loss of resolution. The resolution of the refocused image is equal to the pixel resolution of the sensor.

9. The apparatus of claim 1, in which the pattern in printed on the mask.

10. The apparatus of claim 1, in which the mask is active, and further comprising:

means for changing a position of the mask and the spatially frequency of the pattern continuously.

11. The apparatus of claim 5, further comprising:

means for refocusing the input image from the reconstructing 4D light field to produce an in-focus output image, without a loss of resolution.

12. The apparatus of claim 11, in which the output image has an increased depth of field.

13. The apparatus of claim 5, further comprising:

means for synthesizing a novel refocused image from the reconstructed 4D light field.

14. The apparatus of claim 5, further comprising:

means for generating a high resolution image from the reconstructed 4D light field.

15. The apparatus of claim 1, in which the pattern includes colors.

16. The apparatus of claim 1, in which the mask is placed at an angle with respect to the optical path.

17. The apparatus of claim 1, in which the mask has a cosine pattern.

18. The apparatus of claim 1, in which the mask uses polarizing effects.

19. The apparatus of claim 1, in which the pattern has a high spatial frequency and is arranged near the lens to precisely sense depth in the scene.

20. The apparatus of claim 7, further comprising:

means for deblurring the input image to remove out of focus blur.

21. The apparatus of claim 1, in which the sensor produces an input image encoding the 4D light field, and further comprising:

a microprocessor connected to the sensor for generating an output image from the input image.

22. The apparatus of claim 1, in which the sensor acquires a sequence of images of the scene.

33. The apparatus of claim 5, further comprising the step of:

refocusing the input image from the reconstructed 4D light field to produce an in-focus output image.

34. The method of claim 33, in which the output image has an increased depth of field.

23. A method for acquiring a 4D light field of a scene, comprising the steps of:

passing a 4D light field through a lens to a sensor along a straight optical path; and

modulating the 4D light field with an attenuating mask arranged in the optical path between the lens and the sensor.

24. The method of claim 23, in which the pattern has a low spatial frequency and is arranged near the lens.

25. The method of claim 23, in which the pattern has a high spatial frequency and is arranged near the sensor.

26. The method of claim 25, in which the mask heterodynes the 4D light field.

27. The method of claim 23, in which the sensor produces an input image, and further comprising the step of:

reconstructing the 4D light field from the input image.

28. The method of claim 27, in which the 4D light field is reconstructed by Fourier-domain decoding.

29. The method of claim 24, in which the sensor produces a defocused blur input image.

30. The method of claim 29, in which the refocused image obtained by deblurring the input image has same resolution as the pixel resolution of the sensor.

31. The method of claim 23, in which the pattern in printed on the mask.

32. The method of claim 23, in which the mask is active, and further comprising the step:

changing a position of the mask and the spatially frequency of the pattern continuously.

35. The method of claim 27, further comprising the step of:

synthesizing a novel image from the reconstructed 4D light field.

36. The method of claim 27, further comprising the step of:

generating a high resolution image from the reconstructed 4D light field.

37. The method of claim 23, in which the pattern includes colors.

38. The method of claim 23, in which the mask is placed at an angle with respect to the optical path.

39. The method of claim 23, in which the mask has a cosine pattern.

40. The method of claim 23, in which the mask uses polarizing effects.

41. The method of claim 23, in which the pattern has a high spatial frequency and is arranged near the lens to precisely sense depth in the scene.

42. The method of claim 29, further comprising the step of:

deburring the input image.

43. The method of claim 23, in which the sensor produces an input image encoding the 4D light field, and further comprising the step of:

generating an output image from an input image produced by the sensor.

44. The method of claim 23, in which the sensor acquires a sequence of images of the scene.

45. The method of claim 23, further comprising the steps of:

applying a Fourier transform to a 2D signal sensed by the sensor to obtain 2D tiles;

re-assembling the 2D tiles into a 4D stack of planes; and

applying an inverse Fourier transform to the 4D stack of tiles to reconstruct the 4D light field.

46. The method of claim 23, in which the mask is arranged in an aperture, and in which a spectrum of the 4D light field of the scene lacks variation in an angular dimensions, and the mask is arranged in an aperture and a spectrum of the mask lacks variation in the spatial dimensions and the sensor senses a convolution of the two spectrums.

47. The method of claim 45, in which the Fourier transform of the 4D light field l after passing through the aperture is given by a modulation of the Fourier transform of the incoming light field with the Fourier transform of the aperture modulation function.

48. The method of claim 27, further comprising:

determining a re focused image at a different depth from the input image.

49. The method of claim 27, further comprising:

synthesizing a novel image from a different viewpoint from the reconstructed 4D light field.

50. The method of claim 23, in which multiple masks are arranged in the optical path.

52. The method of claim 23, in which the pattern is an analytical 2D function.

53. The method of claim 23, in which the pattern is a binary function.

54. The method of claim 23, in which the pattern is a continuous 2D function.

55. The method of claim 50, in which the sensor acquires a sequence of images of the scene with the multiple masks.

56. A method for acquiring a 4D light field of a scene, comprising the steps of:

modulating a 4D light field with an attenuating mask arranged in a straight optical path between a lens and a sensor; and

sensing the modulated 4D light field by the sensor to produce an input image encoding the 4D light field.

57. The method of claim 56, further comprising:

demodulating the input image to recover the 4D light field.