Systems and methods for de-blurring motion blurred images

• US 20060119710A1
• Filed: 11/03/2004
• Published: 06/08/2006
• Est. Priority Date: 06/21/2002
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Nayar, Shree K., Ben-Ezra, Moshe
• Granted Patent: US 7,619,656 B2
• Time in Patent Office: Days
• US Class Current: 348/208.990
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Systems and methods for providing a substantially de-blurred image of a scene from a motion blurred image of the scene are disclosed. An exemplary system includes a primary detector for sensing the motion blurred image and generating primary image information representing the blurred image, a secondary detector for sensing two or more secondary images of the scene and for generating secondary image information representing the two or more secondary images, and a processor for determining motion information from the secondary image information, estimating a point spread function for the motion blurred image from the motion information, and applying the estimated point spread function to the primary image information to generate information representing the substantially de-blurred image.

Claims

1. A system for providing a substantially de-blurred image of a scene from a motion blurred image of said scene, comprising:

(a) a primary detector for sensing said motion blurred image of said scene at a first predetermined resolution and generating primary image information representing said blurred image;

(b) a secondary detector, coupled to said primary detector, for sensing two or more secondary images of said scene at a second predetermined resolution which is less than or equal to said first predetermined resolution, and for generating secondary image information representing said two or more secondary images; and

(c) a processor, coupled to said primary detector and receiving said primary image information therefrom, and coupled to said secondary detector and receiving said secondary image information therefrom, adapted to determine motion information from said secondary image information, estimate a point spread function for said motion blurred image from said motion information, and apply said estimated point spread function to said primary image information to thereby generate information representing said substantially de-blurred image.

2. The system of claim 1, further comprising a first camera housing said primary detector, a second camera housing said secondary detector, and a rigid member connecting said first camera to said second camera.

3. The system of claim 1, further comprising a camera housing said primary and said secondary detectors.

4. The system of claim 3, further comprising a beam splitter having one input area and first and second output areas, said beam splitter being optically coupled to said scene at said input area, optically coupled to said primary detector at said first output area, and optically coupled to said secondary detector at said second output area.

5. The system of claim 4, wherein said beam splitter comprises an asymmetric beam splitter adapted to output greater than 50% of an input image through said first output area.

6. The system of claim 5, wherein said beam splitter is adapted to output greater than 80% of an input image through said first output area.

7. The system of claim 4, wherein said beam splitter is adapted to output substantially all visible light corresponding to an input image through said first output area, and at least some non-visible electromagnetic radiation corresponding to an input image through said second output area.

8. The system of claim 3, wherein said primary detector comprises a first portion of a dual-resolution sensor, and said secondary detector comprises a second portion of said dual-resolution sensor.

9. The system of claim 1, wherein said first predetermined resolution is at least twice as great as said second predetermined.

10. A method for providing a substantially de-blurred image of a scene from a motion blurred image of said scene, comprising the steps of:

(a) sensing said motion blurred image of said scene at a first predetermined resolution;

(b) sensing two or more secondary images of said scene at a second predetermined resolution which is less than or equal to said first predetermined resolution while said motion blurred image is sensed;

(c) generating primary image information representing said blurred image and secondary image information representing said two or more secondary images;

(d) determining motion information from said secondary imageS information;

(e) estimating a point spread function for said motion blurred image from said motion information; and

(f) applying said estimated point spread function to said primary image information to thereby generate information representing said substantially de-blurred image.

11. The method of claim 10, wherein said step (b) comprises sensing five or more secondary images of said scene at said second predetermined resolution, and said step (c) comprises generating primary image information representing said blurred image and secondary image information representing said five or more secondary images.

12. The method of claim 10, wherein step (d) comprises the step of determining global motion information from said secondary image information.

13. The method of claim 12, wherein step (e) comprises the step of estimating a continues point spread function for said motion blurred image from said global motion information.

14. The method of claim 13, wherein step (f) comprises the step of deconvolving said primary image information with estimated point spread function to thereby generate information representing said substantially de-blurred image.

15. A method for providing a substantially de-blurred image of a scene from primary image information representing a motion blurred image of said scene captured at a first predetermined resolution, and secondary image information representing two or more secondary images of said scene captured at a second predetermined resolution which is less than or equal to said first predetermined resolution, comprising the steps of:

(a) determining motion information from said secondary image information;

(b) estimating a point spread function for said motion blurred image from said motion information; and

(c) applying said estimated point spread function to said primary image information to thereby generate information representing said substantially de-blurred image.

16. The method of claim 15, wherein step (a) comprises the step of determining global motion information from said secondary image information.

17. The method of claim 16, wherein step (b) comprises the step of estimating a continuous point spread function for said motion blurred image from said global motion information.

18. The method of claim 17, wherein step (c) comprises the step of deconvolving said primary image information with estimated point spread function to thereby generate information representing said substantially de-blurred image.

19. A method for estimating a point spread function for an image which was blurred due to motion of an image sensor which captured said image, comprising the steps of:

(a) receiving discrete motion information indicative of said motion of said image sensor;

(b) determining a continuous two-dimensional path corresponding to said discrete motion points by interpolating missing data;

(c) partitioning said determined continuous two-dimensional path into a plurality of frames; and

(d) constructing equal-area regions within each of said partitioned frames to determine a mean energy value for each of said frames.

20. The method of claim 19, wherein step (b) comprises the step of determining said continuous two-dimensional path corresponding to said discrete motion points by Spline interpolation.

21. The method of claim 19, wherein step (c) comprises the step of partitioning said determined continuous two-dimensional path into said plurality of frames by Voronoi Tessellation.

22. The method of claim 19, further comprising the step of normalizing said determined mean energy values.

23. A method for interactively determining a de-blurred image corresponding to a motion blurred image of a scene, comprising the steps of:

(a) selecting one or more features within said motion blurred image;

(b) forming one or more de-blurred image feature models, each of which corresponds to one of said one or more selected features;

(c) estimating a point spread function using said selected one or more features and said corresponding one or more de-blurred image feature models; and

(d) determining said de-blurred image using said estimated point spread function and said motion blurred image.

24. The method of claim 23, wherein at least one of said one of said one or more selected features comprises a point, and said step (b) comprises the step of using a point tool to form at least one of said one or more de-blurred image feature models.

25. The method of claim 23, wherein at least one of said one of said one or more selected features comprises a line, and said step (b) comprises the step of using a line tool to form at least one of said one or more de-blurred image feature models.

26. The method of claim 23, wherein at least one of said one of said one or more selected features comprises a ramp, and said step (b) comprises the step of using a ramp tool to form at least one of said one or more de-blurred image feature models.

27. The method of claim 23, wherein at least one of said one of said one or more selected features comprises a comer, and said step (b) comprises the step of using a comer tool to form at least one of said one or more de-blurred image feature models.

28. A method for measuring a point spread function for an image of a scene which was blurred due to motion of an image sensor which captured said image, comprising the steps of:

(a) aiming a stabilized light beam at said scene to thereby form a light point in said scene;

(b) capturing a blurred image of said scene, including a blurred image of said light point, using a non-stabilized image sensor; and

(c) determining said point spread function from said blurred image of said light point.

29. The method of claim 28, wherein said stabilized light beam originates from a laser attached to said non-stabilized image sensor.

30. The method of claim 28, wherein said stabilized light beam originates from a laser positioned independently from said non-stabilized image sensor.

31. The method of claim 28, further comprising the step of generating a de-blurred image corresponding to said blurred image of said scene.