4D light field cameras

• US 7,792,423 B2
• Filed: 02/06/2007
• Issued: 09/07/2010
• Est. Priority Date: 02/06/2007
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Raskar, Ramesh, Agrawal, Amit Kumar
• Primary Examiner(s): Perkey; W. B.
• Assistant Examiner(s): Hancock; Dennis
• Application Number:
  US11/671,701
• Publication Number:
  US 20080187305A1
• Time in Patent Office: 1,309 Days
• Field of Search: 398/172, 398/204, 324/76.21, 324/76.41, 348/499, 348/E13, 359/559, 382/277, 382/280, 382/279, 382/263, 382/264, 356/310, 396/268, 396/89, 250/237.R
• US Class Current: 396/268
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus

Abstract

A camera acquires a 4D light field of a scene. The camera includes a lens and sensor. A mask is arranged in a straight optical path between the lens and the sensor. The mask including an attenuation pattern to spatially modulate the 4D light field acquired of the scene by the sensor. The pattern has a low spatial frequency when the mask is arranged near the lens, and a high spatial frequency when the mask is arranged near the sensor.

Claims

1. An apparatus for acquiring a 4D light field of a scene, comprising:

a lens;

a sensor; and

a mask arranged in a straight optical path between the lens and the sensor, the mask including an attenuation pattern to spatially modulate the 4D light field acquired of the scene by the sensor, in which the pattern has a high spatial frequency, and in which the mask is arranged on a front surface of the sensor to acquire the 4D light field of the scene.

2. The apparatus of claim 1, in which the mask heterodynes the 4D light field.

3. The apparatus of claim 1, in which the sensor produces an input image, and further comprising:

means for reconstructing the 4D light field from the input image.

4. The apparatus of claim 3, in which the 4D light field is reconstructed by Fourier-domain decoding.

5. The apparatus of claim 3, further comprising:

means for refocusing the input image from the reconstructed 4D light field to produce an in-focus output image, without a loss of resolution.

6. The apparatus of claim 5, in which the output image has an increased depth of field.

7. The apparatus of claim 3, further comprising:

means for synthesizing a novel refocused image from the reconstructed 4D light field.

8. The apparatus of claim 3, further comprising:

means for generating a high resolution image from the reconstructed 4D light field.

9. The apparatus of claim 1, in which the sensor produces a defocused blur input image.

10. The apparatus of claim 9, further comprising:

means for deblurring the input image to remove out of focus blur.

11. The apparatus of claim 1, in which digital refocusing can be achieved without a loss of resolution.

12. The apparatus of claim 1, in which the pattern is printed on the mask.

13. The apparatus of claim 1, in which the mask is active, and further comprising:

means for changing a position of the mask and the spatial frequency of the pattern continuously.

14. The apparatus of claim 1, in which the pattern includes colors.

15. The apparatus of claim 1, in which the mask is placed at an angle with respect to the optical path.

16. The apparatus of claim 1, in which the mask has a cosine pattern.

17. The apparatus of claim 1, in which the mask uses polarizing effects.

18. The apparatus of claim 1, in which the sensor produces an input image encoding the 4D light field, and further comprising:

a microprocessor connected to the sensor for generating an output image from the input image.

19. The apparatus of claim 1, in which the sensor acquires a sequence of images of the scene.

20. A method for acquiring a 4D light field of a scene, comprising the steps of:

passing a 4D light field through a lens to a sensor along a straight optical path; and

modulating the 4D light field with a mask arranged in the optical path between the lens and the sensor, in which the mask includes an attenuation pattern to spatially modulate the 4D light field acquired of the scene by the sensor, in which the pattern has a high spatial frequency, and in which the mask is arranged on a front surface of the sensor to acquire the 4D light field of the scene.

21. The method of claim 20, in which the mask heterodynes the 4D light field.

22. The method of claim 20, in which the sensor produces an input image, and further comprising the step of:

reconstructing the 4D light field from the input image.

23. The method of claim 22, in which the 4D light field is reconstructed by Fourier-domain decoding.

24. The method of claim 22, further comprising the step of:

refocusing the input image from the reconstructed 4D light field to produce an in-focus output image.

25. The method of claim 24, in which the output image has an increased depth of field.

26. The method of claim 22, further comprising the step of:

synthesizing a novel image from the reconstructed 4D light field.

27. The method of claim 22, further comprising the step of:

generating a high resolution image from the reconstructed 4D light field.

28. The method of claim 22, further comprising:

determining a refocused image at a different depth from the input image.

29. The method of claim 22, further comprising:

synthesizing a novel image from a different viewpoint from the reconstructed 4D light field.

30. The method of claim 20, in which the sensor produces a defocused blur input image.

31. The method of claim 30, in which the refocused image obtained by deblurring the input image has the same resolution as the pixel resolution of the sensor.

32. The method of claim 30, further comprising the step of:

deblurring the input image.

33. The method of claim 20, in which the pattern is printed on the mask.

34. The method of claim 20, in which the mask is active, and further comprising the step of:

changing a position of the mask and the spatial frequency of the pattern continuously.

35. The method of claim 20, in which the pattern includes colors.

36. The method of claim 20, in which the mask is placed at an angle with respect to the optical path.

37. The method of claim 20, in which the mask has a cosine pattern.

38. The method of claim 20, in which the mask uses polarizing effects.

39. The method of claim 20, in which the sensor produces an input image encoding the 4D light field, and further comprising the step of:

generating an output image from an input image produced by the sensor.

40. The method of claim 20, in which the sensor acquires a sequence of images of the scene.

41. The method of claim 20, further comprising the steps of:

applying a Fourier transform to a 4D signal sensed by the sensor to obtain 2D tiles;

re-assembling the 2D tiles into a 4D stack of planes; and

applying an inverse Fourier transform to the 4D stack of tiles to reconstruct the 4D light field.

42. The method of claim 20, in which the Fourier transform of the 4D light field after passing through the aperture is given by a modulation of the Fourier transform of the incoming light field with the Fourier transform of the aperture modulation function.

43. The method of claim 20, in which multiple masks are arranged in the optical path.

44. The method of claim 43, in which the sensor acquires a sequence of images of the scene with the multiple masks.

45. The method of claim 20, in which the pattern is an analytical 2D function.

46. The method of claim 20, in which the pattern is a binary function.

47. The method of claim 20, in which the pattern is a continuous 2D function.

48. A method for acquiring a 4D light field of a scene, comprising the steps of:

modulating a 4D light field with a mask arranged in a straight optical path between a lens and a sensor, the mask including an attenuation pattern to spatially modulate the 4D light field acquired of the scene by the sensor, in which the pattern has a high spatial frequency, and in which the mask is arranged on a front surface of the sensor to acquire the 4D light field of the scene; and

sensing the modulated 4D light field by the sensor to produce an input image encoding the 4D light field.

49. The method of claim 48, further comprising the steps of:

demodulating the input image to recover the 4D light field.