Method for de-blurring images of moving objects

• US 7,440,634 B2
• Filed: 06/17/2004
• Issued: 10/21/2008
• Est. Priority Date: 06/17/2003
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Nayar, Shree K., Ben-Ezra, Moshe
• Primary Examiner(s): Mehta; Bhavesh
• Assistant Examiner(s): Strege; John B
• Application Number:
  US10/870,814
• Publication Number:
  US 20050047672A1
• Time in Patent Office: 1,587 Days
• Field of Search: 382/255, 382/103, 382/284
• US Class Current: 382/255
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Systems and methods for providing methods are provided for deblurring images, which are blurred due to the movement of objects in the imaged scene. The systems and methods involve separating a moving object component from a static or background component of a primary image. The moving object component is deblurred using suitable image processing algorithms, and then superimposed on the static or background component to obtain a deblurred primary image.

Claims

1. A method for deblurring images of a scene that includes a moving object, the method comprising,

obtaining a blurred primary image I of the scene that includes the moving object at a first resolution;

obtaining at least two secondary image of the scene at a second resolution in addition to the blurred primary image of the scene at the first resolution;

analyzing the at least two secondary images of the scene at the second resolution for information on the movement of the object; and

estimating a point spread function S that models the movement of the moving object based on information on the movement of the object in the at least two secondary images of the scene;

obtaining a background image B of the scene;

obtaining a shape of the moving object;

computing a mask M from the shape of the moving object and the point spread function from the secondary images;

applying the mask to the background image B to isolate a static background component not associated with the moving object;

isolating a moving object component of the primary image I;

deblurring the moving object component by image deconvolution with the point spread function; and

superimposing the deblurred moving object component and a background component to obtain a deblurred primary image of the scene.

2. The method of claim 1 wherein obtaining at least two secondary images of the scene in addition to the blurred primary image of the scene comprises using a hybrid imaging system with multiple detectors to capture the images of the scene.

3. The method of claim 2 wherein obtaining at least one secondary image of the scene in addition to the deblurred primary image of the scene comprises obtaining a secondary image at a lower spatial resolution than that the primary image.

4. The method of claim 1 wherein obtaining a point spread function S that models the movement of he moving object, comprises estimating a continuous point spread function.

5. The method of claim 1 wherein obtaining a point spread function S that models the movement of he moving object, comprises computing a point spread function using kinetic theory.

6. The method of claim 1 wherein obtaining a clear background image B of the scene comprises capturing several high-resolution images of the scene and median filtering the brightness level at each image pixel.

7. The method of claim 1 wherein isolating a moving object component of the primary image I comprises subtracting the static background component from the primary image I.

8. The method of claim 1 wherein obtaining a background image B comprises deblurring a motion blurred background image.

9. An image processor for deblurring a primary image I of a scene captured with a primary detector, wherein the primary image I is blurred due to the movement of an object in the scene, the processor comprising:

a first arrangement for separating a moving object component from the primary image I;

a second arrangement for deblurring the moving object component by image deconvolution; and

a third arrangement for superimposing the deblurred moving object component on a background image of the scene to obtain a deblurred primary image of the scene,

wherein the second arrangement for deblurring the moving object component by image deconvolution, comprises;

a hybrid imaging system having a secondary detector associated with the primary detector with which the prim image I is captured, wherein the secondary detector is configured to capture second images of the scene for motion analysis at a second resolution; and

an arrangement for estimating a point spread function from the secondary images of the scene for motion analysis at a second resolution and deconvolving it with the moving object component.

10. The image processor of claim 9, wherein the motion analysis comprises estimating point spread function which models the movements of the moving object in the imaged scene.

11. The image processor of claim 9 wherein the background image of the scene comprises an image the portion of the scene that is stationary.

12. The image processor of claim 9 wherein the first arrangement for separating a moving object component from the primary image I comprises a segmentation mask.

13. The image processor of claim 12 wherein the segmentation mask is constructed from a shape oft object and a point spread function which models the movements of the moving object in the imaged scene.

14. The image processor of claim 9 wherein at least one of the three arrangements is disposed in a digital photography camera.

15. The image processor of claim 9 wherein at least one of the three arrangements is disposed in a personal computer to which primary images are downloaded for processing.

16. The image processor of claim 9 wherein the background image of the scene comprises a deblurred image of the portion of the scene that is stationary.