Method for deblurring images using optimized temporal coding patterns

• US 7,580,620 B2
• Filed: 05/08/2006
• Issued: 08/25/2009
• Est. Priority Date: 05/08/2006
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Agrawal, Amit, Raskar, Ramesh, Tumblin, Jack
• Primary Examiner(s): FULLER, RODNEY EVAN
• Application Number:
  US11/429,694
• Publication Number:
  US 20070258706A1
• Time in Patent Office: 1,205 Days
• Field of Search: 396/52, 396/55, 348/208.99, 348/208.4, 348/208.6, 348/208.13
• US Class Current: 396/55
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/75 by influencing optical camera components

Abstract

A method and system deblurs images acquired of a scene by a camera. A light field acquired of a scene is modulated temporally according to a sequence of ons and offs. An optimal coding pattern is selected. The modulated light field is integrated by a sensor of a camera during an exposure time to generate an encoded input image. The encoded input image is decoded according to a pseudo-inverse of a smearing matrix to produce a decoded output image having a reduced blur.

Claims

1. A method for reducing blur in an image of a scene acquired by a camera, comprising the steps of:

selecting an optimal temporally coded pattern;

temporally modulating light of a scene acquired and integrated by a camera image sensor according to the temporally coded pattern during an exposure time to generate an encoded input image, wherein the exposure time has an arbitrary duration, and the temporally coded pattern is a binary on and off sequence with an arbitrary length; and

decoding the encoded input image according to an inverse of the coded pattern to produce a decoded output image having a reduced blur.

2. The method of claim 1, in which the pattern is binary and has a maximum coverage in a Fourier domain.

3. The method of claim 1, in which the temporally coded pattern is an optimal binary code.

4. The method of claim 1, in which the exposure time is about 200 milliseconds, and the modulating opens and closes a camera shutter about 52 times during the exposure time.

5. The method of claim 1, in which the modulating is selected adaptively according to a sensed optical flow in the scene.

6. The method of claim 5, in which the optical flow is sensed in the camera.

7. The method of claim 5, in which the optical flow is sensed using an ultrasound sensor.

8. The method of claim 5, in which motion of the camera is sensed with an inertial sensor or a gyroscope.

9. The method of claim 1, in which temporally coded pattern opens and closes a shutter of the camera according to a binary sequence 1010000111000001010000110011110111010111001001100111.

10. The method of claim 1, further comprising:

scaling the input image by a factor of m/k;

determining an estimated output image; and

scaling the estimated output image by a factor of k/m to obtain an output image, in which m is the number of ons and offs in the binary sequence, and k is the length of a point spread function vector equal to a number of pixels in a scan line of the camera image sensor.

11. The method of claim 1, further comprising:

determining automatically a color of a background region and a color of a moving object in the input image.

12. The method of claim 1, further comprising:

determining a static background region in the input image; and

determining a foreground region in the input image including a moving object.

13. The method of claim 12, in which the foreground region is specified by a rectangular bounding box.

14. The method of claim 12, in which the foreground region is specified by an irregularly shaped cut-out.

15. The method of claim 12, further comprising:

reducing an effect of the static background by excluding pixels in the input image along motion lines having a relatively high gradient.

16. The method of claim 1, further comprising:

removing from the input image a static object occluding a blurred moving object.

17. The method of claim 1, in which temporal blur in the input image is modified during a post-processing step to achieve novel synthesis effects.