IMAGE DEBLURRING USING PANCHROMATIC PIXELS

• US 20110090378A1
• Filed: 10/16/2009
• Published: 04/21/2011
• Est. Priority Date: 10/16/2009
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Hou, Tingbo, Wang, Sen, Miller, Rodney L.
• Granted Patent: US 8,203,615 B2
• Time in Patent Office: Days
• US Class Current: 348/242
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 5/208 for compensating for attenuation of high frequency components, e.g. crispening, aperture distortion correction

Abstract

A method for determining a deblurred image from images captured using an image sensor comprising a two-dimensional array of light-sensitive pixels including panchromatic pixels and color pixels, the method comprising: receiving image sensor data from the image sensor for a sharp image of a scene captured with a short exposure time; receiving image sensor data from the image sensor for a blurred image of the scene captured with a longer exposure time than the sharp image; determining an interpolated sharp grayscale image from the sharp image; determining an interpolated blurred color image from the blurred image; determining an interpolated blurred grayscale image from the blurred image; determining a blur kernel responsive to the sharp grayscale image and the blurred grayscale image; determining a deblurred image responsive to the blurred color image and the blur kernel; and storing the deblurred image in a processor-accessible memory system.

Claims

1. A method for determining a deblurred image from images captured using an image sensor comprising a two-dimensional array of light-sensitive pixels including panchromatic pixels and color pixels, the pixels being arranged in a rectangular minimal repeating unit, the method implemented at least in part by a data processing system and comprising:

a) receiving image sensor data from the image sensor for a sharp image of a scene captured with a short exposure time;

b) receiving image sensor data from the image sensor for a blurred image of the scene captured with a longer exposure time than the sharp image, wherein the blurred image has a higher level of motion blur than the sharp image;

c) determining an interpolated sharp grayscale image from the image sensor data for the sharp image;

d) determining an interpolated blurred color image from the image sensor data for the blurred image;

e) determining an interpolated blurred grayscale image from the image sensor data for the blurred image;

f) determining a blur kernel responsive to the sharp grayscale image and the blurred grayscale image;

g) determining a deblurred image responsive to the blurred color image and the blur kernel; and

h) storing the deblurred image in a processor-accessible memory system.

2. The method of claim 1 wherein the blur kernel is determined by using an optimization method to minimize an energy function which includes a term that is a function of a difference between the blurred grayscale image and a candidate blurred image determined by convolving the sharp grayscale image with a candidate blur kernel.

3. The method of claim 2 wherein the blur kernel is determined by analyzing only a subset of the image pixels in the sharp grayscale image and a corresponding subset of the image pixels in the blurred grayscale image, wherein the subsets are determined to contain significant image detail.

4. The method of claim 3 wherein the subset of the image pixels in the sharp grayscale image is selected by:

determining a Laplacian image from either the sharp grayscale image or the blurred grayscale image or both; and

selecting the subset of image pixels in the sharp grayscale image and the corresponding subset of image pixels in the blurred grayscale image by analyzing the Laplacian image to determine a subset of image pixels containing significant image detail.

5. The method of claim 2, wherein a denoising algorithm or a tone scale adjustment algorithm or both are applied to the sharp grayscale image before it is used to determine the blur kernel.

6. The method of claim 2, wherein the blur kernel is determined by analyzing multiple resized versions of the sharp grayscale image and the blurred grayscale image having different resolutions.

7. The method of claim 1 wherein the deblurred image is determined using a Bayesian inference method.

8. The method of claim 7 wherein the Bayesian inference method includes evaluating candidate deblurred images using an energy function.

9. The method of claim 8 wherein the energy function includes an image fidelity term which is a function of the difference between the blurred color image and a candidate blurred image determined by convolving a candidate deblurred image with the blur kernel.

10. The method of claim 9 wherein the energy function further includes an image differential term which is a function of differences between one or more reference differential images determined from the interpolated sharp grayscale image and one or more candidate differential images determined from the candidate deblurred image

11. The method of claim 9 wherein the energy function further includes a term which is a function of a gradient of the candidate deblurred image.

12. The method of claim 8 wherein the energy function is optimized using a conjugate gradient method.

13. The method of claim 1 wherein the deblurred image is determined using a frequency domain deconvolution method.

14. The method of claim 1 wherein the blurred grayscale image is determined from the blurred color image.

15. The method of claim 1 further including determining one or more reference differential images responsive to the sharp grayscale image;

and wherein the step of determining the deblurred image is further responsive to the one or more reference differential images.

16. The method of claim 15 wherein the one or more reference differential images include a horizontal differential image representing differences between neighboring pixels in the horizontal direction and a vertical differential image representing differences between neighboring pixels in the vertical direction.

17. The method of claim 1 further including applying an affine transform to the sharp grayscale image or the blurred grayscale image to better align the sharp grayscale image with the blurred grayscale image.

18. The method of claim 1 wherein the sharp image and the blurred image are captured sequentially using a digital camera in response to a single button press.

19. The method of claim 1 wherein the motion blur is at least partially due to motion of the digital camera during the time that the sharp image and the blurred image are captured.

20. The method of claim 1 wherein the motion blur is at least partially due to motion of objects in the scene, during the time that the sharp image and the blurred image are captured.

21. The method of claim 1 wherein the blurred image has a lower level of image noise than the sharp image.

22. The method of claim 1 wherein the sharp image and the blurred image are frames of a digital video.

23. A digital camera system comprising:

an image sensor for capturing an image of a scene, the image sensor comprising a two-dimensional array of light-sensitive pixels including panchromatic pixels and color pixels, the pixels being arranged in a rectangular minimal repeating unit;

a lens system for imaging the scene onto the image sensor;

a processor-accessible memory system

a data processing system for performing the steps of;

receiving image sensor data from the image sensor for a sharp image of a scene captured with a short exposure time;

receiving image sensor data from the image sensor for a blurred image of the scene captured with a longer exposure time than the sharp image, wherein the blurred image has a higher level of motion blur than the sharp image;

determining an interpolated sharp grayscale image from the image sensor data for the sharp image;

determining an interpolated blurred color image from the image sensor data for the blurred image;

determining an interpolated blurred grayscale image from the image sensor data for the blurred image;

determining a blur kernel responsive to the sharp grayscale image and the blurred grayscale image;

determining a deblurred image responsive to the blurred color image and the blur kernel; and

storing the deblurred image in the processor-accessible memory system.

24. A computer program product for determining a deblurred image from images captured using an image sensor comprising a two-dimensional array of light-sensitive pixels including panchromatic pixels and color pixels, the pixels being arranged in a rectangular minimal repeating unit, the computer program product comprising an executable software application for causing a data processing system to perform the steps of:

a) receiving image sensor data from the image sensor for a sharp image of a scene captured with a short exposure time;

b) receiving image sensor data from the image sensor for a blurred image of the scene captured with a longer exposure time than the sharp image, wherein the blurred image has a higher level of motion blur than the sharp image;

c) determining an interpolated sharp grayscale image from the image sensor data for the sharp image;

d) determining an interpolated blurred color image from the image sensor data for the blurred image;

e) determining an interpolated blurred grayscale image from the image sensor data for the blurred image;

f) determining a blur kernel responsive to the sharp grayscale image and the blurred grayscale image;

g) determining a deblurred image responsive to the blurred color image and the blur kernel; and

h) storing the deblurred image in a processor-accessible memory system.