Method for de-blurring images of moving objects

• US 20050047672A1
• Filed: 06/17/2004
• Published: 03/03/2005
• Est. Priority Date: 06/17/2003
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Nayar, Shree K., Ben-Ezra, Moshe
• Granted Patent: US 7,440,634 B2
• Time in Patent Office: Days
• US Class Current: 382/255
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Systems and methods for providing methods are provided for deblurring images, which are blurred due to the movement of objects in the imaged scene. The systems and methods involve separating a moving object component from a static or background component of a primary image. The moving object component is deblurred using suitable image processing algorithms, and then superimposed on the static or background component to obtain a deblurred primary image.

Claims

1. A method for deblurring images of a scene that includes a moving object, the method comprising,

obtaining a blurred primary image I of the scene that includes the moving object;

obtaining a point spread function S that models the movement of the moving object;

obtaining a background image B of the scene;

obtaining a shape of the moving object;

computing a mask M from the shape of the moving object and the point spread function;

applying the mask to the background image B to isolate a static background component not associated with the moving object;

isolating a moving object component of the primary image I;

deblurring the moving object component by image deconvolution with the point spread function; and

superimposing the deblurred moving object component and a background component to obtain a deblurred primary image of the scene.

2. The method of claim 1 wherein obtaining a point spread function S that models the movement of the moving object, comprises:

obtaining at least two secondary image of the scene in addition to the blurred primary image of the scene;

analyzing the secondary images of the scene for information on the movement of the object; and

estimating the point spread function S based on information on the movement of the object.

3. The method of claim 2 wherein obtaining at least one secondary image of the scene in addition to the blurred primary image of the scene comprises using a hybrid imaging system with multiple detectors to capture the images of the scene. (Shape require single image from the secondary detector whereas PSF requires at least two, (if not calculated using kinetic data)]

4. The method of claim 3 wherein obtaining at least one secondary image of the scene in addition to the blurred primary image of the scene comprises obtaining a secondary image at a lower spatial resolution than that the primary image.

5. The method of claim 1 wherein obtaining a point spread function S that models the movement of the moving object, comprises estimating a continuous point spread function:

6. The method of claim 1 wherein obtaining a point spread function S that models the movement of the moving object, comprises computing a point spread function using kinetic theory.

7. The method of claim 1 wherein obtaining a clear background image B of the scene comprises capturing several high-resolution images of the scene and median filtering the brightness level at each image pixel.

8. The method of claim 1 wherein isolating a moving object component of the primary image I comprises subtracting the static background component from the primary image I.

9. The method of claim 1 wherein obtaining a background image B comprises deblurring a motion blurred background image.

10. An image processor for deblurring a primary image I of a scene captured with a primary detector, wherein the primary image I is blurred due to the movement of an object in the scene, the processor comprising:

a first arrangement for separating a moving object component from the primary image I;

a second arrangement for deblurring the moving object component by image deconvolution; and

a third arrangement for superimposing the deblurred moving object component on a background image of the scene to obtain a deblurred primary image of the scene.

11. The image processor of claim 10 wherein the second arrangement for deblurring the moving object component by image deconvolution, comprises an arrangement for estimating a point spread function and deconvolving it with the moving object component.

12. The image processor of claim 10, further comprising a hybrid imaging system having a secondary detector associated with the primary detector with which the primary image I is captured, wherein the secondary detector is configured to capture secondary images of the scene for motion analysis.

13. The image processor of claim 12, wherein the motion analysis comprises estimating a point spread function which models the movements of the moving object in the imaged scene.

14. The image processor of claim 10 wherein the background image of the scene comprises an image the portion of the scene that is stationary.

15. The image processor of claim 10 wherein the first arrangement for separating a moving object component from the primary image I comprises a segmentation mask.

16. The image processor of claim 15 wherein the segmentation mask is constructed from a shape of the object and a point spread function which models the movements of the moving object in the imaged scene.

17. The image processor of claim 10 wherein at least one of the three arrangements is disposed in a digital photography camera.

18. The image processor of claim 10 wherein at least one of the three arrangements is disposed in a personal computer to which primary images are downloaded for processing.

19. The image processor of claim 10 wherein the background image of the scene comprises a deblurred image of the portion of the scene that is stationary.