Method and apparatus for deblurring images

• US 7,756,407 B2
• Filed: 05/08/2006
• Issued: 07/13/2010
• Est. Priority Date: 05/08/2006
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Raskar, Ramesh
• Primary Examiner(s): FULLER, RODNEY EVAN
• Application Number:
  US11/430,233
• Publication Number:
  US 20070258707A1
• Time in Patent Office: 1,527 Days
• Field of Search: 396/52, 396/55, 348/208.99, 348/208.4, 348/208.6, 348/208.13
• US Class Current: 396/55
• CPC Class Codes: G03B 17/00 Details of cameras or camera bodies; Accessories therefor lens hoods or caps G03B11/04
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/73 by influencing the exposure time

Abstract

A method and system deblurs images acquired of a scene by a camera. A light field acquired of a scene is modulated temporally according to a sequence of ons and offs. The modulated light field is integrated by a sensor of a camera during an exposure time to generate an encoded input image. The encoded input image is decoded according to a pseudo-inverse of a smearing matrix to produce a decoded output image having a reduced blur.

Claims

1. A method for reducing blur in an image of a scene acquired by a camera, comprising the steps of:

temporally modulating light of a scene acquired and integrated by a camera image sensor according to a temporally coded pattern during an exposure time to generate an encoded input image, in which the camera image sensor is a solid state sensor and includes an array of pixels, and in which the modulating results in a well-posed invertible Fourier transform of a resultant point spread function; and

decoding the encoded input image according to an inverse of the coded pattern to produce a decoded output image having a reduced blur.

2. The method of claim 1, in which the blur is caused by camera motion.

3. The method of claim 1, in which the blur is caused by object motion.

4. The method of claim 1, in which the blur is caused by a combination of camera and object motion.

5. The method of claim 1, in which the modulating is discrete.

6. The method of claim 1, in which the modulating is continuous.

7. The method of claim 1, in which the modulating is selected from the group comprising:

amplitude modulation, frequency modulation, phase modulation, pulse width modulation, opacity modulation, polarization modulation and combinations thereof.

8. The method of claim 1, in which the modulating is performed by repeatedly opening and closing a camera shutter during the exposure time.

9. The method of claim 1, in which the modulating is performed by repeatedly changing an opacity of a camera shutter during the exposure time.

10. The method of claim 1, in which the modulating is performed by repeatedly turning integration by the camera image sensor on and off during the exposure time.

11. The method of claim 1, in which the modulating is performed by repeatedly turning a flash unit illuminating the scene on and off while keeping a camera shutter open during the exposure time.

12. The method of claim 1, in which the modulating is performed by both repeatedly opening and closing a camera shutter and repeatedly turning a flash unit on and off during the exposure time.

13. The method of claim 1, in which the camera image sensor is a single photosensor.

14. The method of claim 1, in which the camera is a video camera configured to generate a sequence of input images and a sequence of deblurred output images.

15. The method of claim 14, in which different temporal modulating is used for different images in the sequence.

16. The method of claim 1, in which the modulating maximizes a coverage in a Fourier domain of a resultant point spread function.

17. The method of claim 16, in which the point spread function is an integration of the modulated light and spatial motion of a moving object in the scene.

18. The method of claim 1, in which the decoding is performed by a process operating in the camera.

19. The method of claim 1, in which the temporal modulation is selected to preserve relatively higher spatial frequencies of a moving object in the scene and to recover the relatively higher spatial frequencies during the decoding.

20. The method of claim 1, in which the modulating is according to a pseudo-random sequence of opens and closes of a camera shutter.

21. The method of claim 1, in which the decoding is performed by a deconvolution process.

22. The method of claim 1, in which the decoding is performed by a linear algebra process using a least-square estimation to estimate the output image.

23. An apparatus for deblurring an image of a scene including a moving object, comprising:

means for temporally modulating a light field of a scene including a moving object acquired by a sensor of a camera during an exposure time to generate an encoded input image, wherein the modulating is according to a binary on and off sequence of the light field, in which the sensor is a solid state sensor and includes an array of pixels, and in which the modulating results in a well-posed invertible Fourier transform of a resultant point spread function; and

means for decoding the encoded input image to produce to a decoded output image having a reduced blur.

24. The apparatus of claim 23, in which the means for modulating is a shutter of the camera.

25. The apparatus of claim 23, in which the means for modulating is an opacity filter of the camera.

26. The apparatus of claim 23, in which the means for modulating is a flash unit of the camera.

27. The apparatus of claim 23, in which the means for modulating turns the sensor on and off while acquiring the light field.

28. The apparatus of claim 23, in which the means for modulating modulates the light field in such a way that a resultant point spread function has a maximum coverage in a Fourier domain.

29. The apparatus of claim 23, in which the input image is a convolution between a temporal integration of the modulated light field and spatial object motion.

30. The apparatus of claim 23, in which the modulating is continuous.

31. The apparatus of claim 23, in which the modulating is discrete.

32. The apparatus of claim 23, in which the binary on and off sequence is

1010000111000001010000110011110111010111001001100111.

33. The apparatus of claim 23, in which a motion of the camera is recovered.

34. A method for reducing blur in an image of a scene, comprising the steps of:

temporally modulating an energy field of a scene acquired and integrated by a sensor according to a temporally coded pattern during an exposure time to generate an encoded input image, in which the camera image sensor is a solid state sensor and includes an array of pixels, and in which the modulating results in a well-posed invertible Fourier transform of a resultant point spread function; and

decoding the encoded input image according to an inverse of the coded pattern to produce a decoded output image having a reduced blur.

35. The method of claim 34, in which the energy field is electromagnetic radiation.

36. The method of claim 34, in which the energy field is acoustic radiation.

37. The method of claim 34, in which the modulating is continuous.

38. The method of claim 34, in which the modulating is discrete.

39. The method of claim 34, in which the modulation is selected from the group consisting of:

amplitude modulation, frequency modulation, phase modulation, pulse width modulation, opacity modulation, polarization modulation and combinations thereof of a shutter between the scene and the sensor.