Image deblurring using a spatial image prior

• US 8,390,704 B2
• Filed: 10/16/2009
• Issued: 03/05/2013
• Est. Priority Date: 10/16/2009
• Status: Expired due to Fees

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Wang, Sen, Miller, Rodney L., Tingbo, Hou
• Primary Examiner(s): Fan, Chieh M
• Assistant Examiner(s): Trehan, Akshay
• Application Number:
  US12/580,569
• Publication Number:
  US 20110090352A1
• Time in Patent Office: 1,236 Days
• Field of Search: 348/208.6, 348/208.4, 348/241, 348/78, 348/248, 348/208.12, 348/208.13, 382/255, 382/275, 382/264
• US Class Current: 348/248
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 25/583 with different integration times

Abstract

A method for determining a deblurred image, the method implemented at least in part by a data processing system and comprising: receiving a sharp image of a scene captured with a short exposure time; receiving a blurred image of the scene captured with a longer exposure time than the sharp image, wherein the blurred image has a higher level of motion blur and a lower level of image noise than the sharp image; determining a blur kernel responsive to the sharp image and the blurred image; determining one or more reference differential images responsive to the sharp image; determining a deblurred image responsive to the blurred image, the blur kernel and the one or more reference differential images; and storing the deblurred image in a processor-accessible memory system.

Claims

1. A method for determining a deblurred image, the method implemented at least in part by a data processing system and comprising:

a) receiving a sharp image of a scene captured with a short exposure time;

b) receiving a blurred image of the scene captured with a longer exposure time than the sharp image, wherein the blurred image has a higher level of motion blur than the sharp image;

c) determining a blur kernel responsive to the sharp image and the blurred image;

d) determining one or more reference differential images responsive to the sharp image, wherein each pixel of a particular reference differential image is determined by computing a difference between the corresponding pixel in the sharp image and a nearby pixel in the sharp image at a predefined relative position, wherein each differential digital image is associated with a different predefined relative position;

e) defining an energy function that includes an image differential term that is a function of differences between the reference differential images and a corresponding set of candidate differential images computed for a candidate deblurred image, wherein the candidate differential images for the candidate deblurred image are computed using the predefined relative positions associated with the corresponding reference differential images;

f) determining a deblurred image responsive to the blurred image and the blur kernel by evaluating candidate deblurred images using the energy function; and

g) storing the deblurred image in a processor-accessible memory system.

2. The method of claim 1 wherein the blur kernel is determined by using an optimization method to minimize an energy function which includes a term that is a function of a difference between the blurred image and a candidate blurred image determined by convolving the sharp image with a candidate blur kernel.

3. The method of claim 2 wherein the blur kernel is determined by analyzing only a subset of image pixels in the sharp image and a corresponding subset of image pixels in the blurred image, wherein the subsets are determined to contain significant image detail.

4. The method of claim 3 wherein the subset of the image pixels in the sharp image is selected by:

determining a Laplacian image from either the sharp image or the blurred image or both; and

selecting the subset of image pixels in the sharp image and the corresponding subset of image pixels in the blurred image by analyzing the Laplacian image to determine a subset of image pixels containing significant image detail.

5. The method of claim 4 wherein the subset of image pixels include pixels in the Laplacian image that exceed a predefined threshold.

6. The method of claim 2, wherein a denoising algorithm or a tone scale adjustment algorithm or both are applied to the sharp image before it is used to determine the blur kernel.

7. The method of claim 2, wherein the blur kernel is determined by analyzing multiple resized versions of the sharp image and the blurred image having different resolutions.

8. The method of claim 1 wherein the one or more reference differential images include a horizontal differential image corresponding to a predefined relative position with a horizontal offset and a vertical differential image corresponding to a predefined relative position with a vertical offset.

9. The method of claim 1 wherein the deblurred image is determined using a Bayesian inference method.

10. The method of claim 9 wherein the Bayesian inference method includes evaluating candidate deblurred images using an energy function.

11. The method of claim 10 wherein the energy function includes:

an image fidelity term which is a function of the difference between the blurred image and a candidate blurred image determined by convolving a candidate deblurred image with the blur kernel.

12. The method of claim 11 wherein the energy function further includes a term which is a function of a gradient of the candidate deblurred image.

13. The method of claim 10 wherein the energy function is optimized using a conjugate gradient algorithm.

14. The method of claim 1 wherein the deblurred image is determined using a frequency domain deconvolution method.

15. The method of claim 1 further including applying an affine transform to the sharp image or the blurred image to better align the sharp image with the blurred image.

16. The method of claim 1 wherein the sharp image and the blurred image are captured sequentially using a digital camera in response to a single button press.

17. The method of claim 16 wherein the motion blur is at least partially due to motion of the digital camera during the time that the sharp image and the blurred image are captured.

18. The method of claim 16 wherein the motion blur is at least partially due to motion of objects in the scene during the time that the sharp image and the blurred image are captured.

19. The method of claim 1 wherein the sharp image and the blurred image are frames of a digital video.

20. A digital camera system comprising:

an image sensor for capturing an image of a scene;

an lens system for imaging the scene onto the image sensor;

a processor-accessible memory system;

a data processing system for performing the steps of;

receiving a sharp image of a scene captured with a short exposure time;

receiving a blurred image of the scene captured with a longer exposure time than the sharp image, wherein the blurred image has a higher level of motion blur and a lower level of image noise than the sharp image;

determining a blur kernel responsive to the sharp image and the blurred image;

determining one or more reference differential images responsive to the sharp image, wherein each pixel of a particular reference differential image is determined by computing a difference between the corresponding pixel in the sharp image and a nearby pixel in the sharp image at a predefined relative position, wherein each reference differential image is associated with a different predefined relative position;

defining an energy function that includes an image differential term that is a function of differences between the reference differential images and a corresponding set of candidate differential images computed for a candidate deblurred image, wherein the candidate differential images for the candidate deblurred image are computed using the predefined relative positions associated with the corresponding reference differential images;

determining a deblurred image responsive to the blurred image and the blur kernel by evaluating candidate deblurred images using the energy function; and

storing the deblurred image in a processor-accessible memory system.

21. A computer program product for determining a deblurred image comprising an executable software application stored in a non-transitory processor readable medium for causing a data processing system to perform the steps of

a) receiving a sharp image of a scene captured with a short exposure time;

b) receiving a blurred image of the scene captured with a longer exposure time than the sharp image, wherein the blurred image has a higher level of motion blur than the sharp image;

c) determining a blur kernel responsive to the sharp image and the blurred image;

d) determining one or more reference differential images responsive to the sharp image, wherein each pixel of a particular reference differential image is determined by computing a difference between the corresponding pixel in the sharp image and a nearby pixel in the sharp image at a predefined relative position, wherein each reference differential image is associated with a different predefined relative position;

e) defining an energy function that includes an image differential term that is a function of differences between the reference differential images and a corresponding set of candidate differential images computed for a candidate deblurred image, wherein the candidate differential images for the candidate deblurred image are computed using the predefined relative positions associated with the corresponding reference differential images;

f) determining a deblurred image responsive to the blurred image and the blur kernel by evaluating candidate deblurred images using the energy function; and

g) storing the deblurred image in a processor-accessible memory system.