Image deblur based on two images

• US 7,680,354 B2
• Filed: 05/22/2006
• Issued: 03/16/2010
• Est. Priority Date: 03/22/2006
• Status: Active Grant

• Current Assignee: ArcSoft, Inc.
• Original Assignee: ArcSoft, Inc.
• Inventors: Mei, Li, Wang, Jin
• Primary Examiner(s): Mehta; Bhavesh M
• Assistant Examiner(s): ZARKA, DAVID PETER
• Application Number:
  US11/419,733
• Publication Number:
  US 20070223831A1
• Time in Patent Office: 1,394 Days
• Field of Search: 382/255
• US Class Current: 382/255
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

A method includes (a) performing image deblur at the original size of underexposed and normally exposed images when their original image size is small. Otherwise, the method includes (b) downsizing the images and performing image deblur on the downsized images. The image deblur on downsized images includes (c) applying a global color transfer between the images if their difference of average gray values is small. Otherwise, the method includes four successive sub-procedures: (d) histogram customizing of the underexposed image based on the normally exposed image; (e) region segmentation for the histogram-customized underexposed image and local alignment of regions in the histogram-customized underexposed image with the normally exposed image; (f) color tuning for the histogram-customized underexposed image based on the local alignment and a global alignment between the underexposed and the normally exposed images; and (g) local color transfer with multi-size neighbor windows from between the images.

Claims

1. A computer-readable storage medium encoded with executable instructions for execution by a processor to generate a final image from a first image with a first exposure and a second image with a second exposure longer than the first exposure, the instructions comprising:

determining if the first and the second images are smaller than a threshold size;

when the first and the second images are smaller than the threshold size, using a deblurring procedure to generate the final image from the first and the second images;

when the first and the second images are not smaller than the threshold size;

saving the first and the second images at a size smaller than their original size as third and fourth images, respectively;

using the deblurring procedure to generate a fifth image from the third and the fourth images; and

transferring color from the fifth image to the first image based on the third image to generate the final image;

displaying the final image.

2. The computer-readable storage medium of claim 1, wherein the first and the second images are smaller than the threshold size when a maximum length of the first and the second images is less than a threshold dimension, the maximum length being a greater of a width and a height of the first and the second images.

3. The computer-readable storage medium of claim 1, wherein said using the deblurring procedure comprises:

receiving first and second input images, wherein;

the first and the second input images are the first and the second images when the first and the second images are smaller than the threshold size; and

the first and the second input images are the third and the fourth images when the first and the second images are not smaller than the threshold size;

determining if the first input image is brighter than the second input image;

when the first input image is brighter than the second input image, transferring color from the second input image to the first input image to generate the final image;

when the first input image is not brighter than the second input image;

globally aligning the first and the second input images;

adjusting saturation and luminance of the first input image based on the second input image to generate a first intermediate image;

customizing a histogram of the first intermediate image based on the second input image to generate a second intermediate image;

segmenting the second intermediate image into visually related regions;

locally aligning the regions in the second intermediate image to regions in the second input image;

adjusting color of the second intermediate image based on results of said globally aligning and said locally aligning to generate a third intermediate image; and

transferring color from the third intermediate image to the first input image to generate the final image.

4. The computer-readable storage medium of claim 3, wherein said determining if the first input image is brighter than the second input image comprises:

determining a first mean of gray values of the first input image and a second mean of gray values of the second input images;

determining if the first mean of gray values is greater than a fixed percentage of the second mean of gray values;

wherein the first input image is brighter than the second input image when the first mean of gray values is greater than the fixed percentage of the second mean of gray values.

5. The computer-readable storage medium of claim 3, wherein said transferring color from the second input image to the first input image comprises:

converting the first and the second input images from their original color space to lα

β

color space where l represents an achromatic luminance channel, α

represents a yellow-blue channel, and β

represents a red-green channel;

determining mean and variance values of each color channel in the first and the second input images;

determining mapping tables based on the mean and the variance values;

mapping the first input image with the mapping tables;

convening the first input image from the lα

β

color space back to the original color space;

saving the first input image as the final image.

6. The computer-readable storage medium of claim 5, wherein said determining mapping tables based on the mean and the variance values comprises generating a mapping table for each color channel based on a color transfer formula defined as follows:

7. The computer-readable storage medium of claim 3, prior to said globally aligning the first and the second input images, the instructions further comprise:

determining a gamma correction factor from a first mean of gray values of the first input image and a second mean of gray values of the second input image;

gamma correcting the first input image and the first mean of gray values using the gamma correction factor; and

adjusting gray values of the second input image so the second mean of gray values is equal to the first mean of the gray values.

8. The computer-readable storage medium of claim 7, wherein:

the gamma correction factor is defined as;

9. The computer-readable storage medium of claim 7, wherein said adjusting gray values of the second input image comprises adding a difference of the first mean of gray values from the second mean of gray values to the gray values of the second input image.

10. The computer-readable storage medium of claim 3, wherein said globally aligning the first and the second input images comprises:

looping through each step level, each step level comprising a step size and a search region;

looping through each displacement based on the step size in the search region;

determining an absolute gray error between the first and the second input images;

setting a displacement with a smallest absolute gray error as a best alignment for a current step level and a starting point for a next level.

11. The computer-readable storage medium of claim 3, wherein said adjusting saturation and luminance of the first input image comprises:

determining gamma correction factors for the saturation and the luminance, wherein the gamma correction factors comprises;

12. The computer-readable storage medium of claim 3, wherein said customizing a histogram of the first intermediate image comprises:

normalizing color channels of the first intermediate image and the second input image in their overlapping areas determined from said globally aligning;

determining histograms of the color channels of the first intermediate image and the second input image in the overlapping areas;

smoothing the histograms;

building channel mapping tables that map the histograms of the second input image to the histograms of the first intermediate image by histogram specification;

inverse normalizing the channel mapping tables; and

mapping the first intermediate image with the channel mapping tables to generate the second intermediate image.

13. The computer-readable storage medium of claim 3, wherein said locally aligning the regions in the second intermediate image to regions in the second input image comprises:

looping through each step level, each step level comprising a step size and a search region size;

looping through each displacement based on the step size in the search region;

looping through each region in the second intermediate image;

determining an absolute gray error between the region in the second intermediate image and a corresponding region in the second input image;

setting a displacement with a smallest absolute gray error as a best alignment of the region.

14. The computer-readable storage medium of claim 3, wherein said adjusting color of the second intermediate image comprises:

looping through each pixel in the second intermediate image;

determining a first absolute color error between the pixel and a first corresponding pixel in the second input image locally aligned with the pixel by said locally aligning;

determining if the first absolute color error is less than a first threshold;

when the first absolute color error is less than the first threshold;

determining a first weak-sorting color error between the pixel and the first corresponding pixel;

determining if a first combination of the first absolute color error and the first weak-sorting color error is less than a second threshold;

when the first combination of the first absolute color error and the first weak-sorting color error is less than the second threshold, setting the color of the pixel equal to the color of the first corresponding pixel.

15. The computer-readable storage medium of claim 14, further comprising:

when the first combination of the first absolute color error and the first weak-sorting color error is not less than the second threshold;

determining a second absolute color error between the pixel and a second corresponding pixel in the second input image globally aligned with the pixel by said globally aligning;

determining if the second absolute color error is less than the first threshold;

when the second absolute color error is less than the first threshold;

determining a second weak-sorting color error between the pixel and the second corresponding pixel;

determining if a second combination of the second absolute color error and the second weak-sorting color error is less than the second threshold;

when the second combination of the second absolute color error and the second weak-sorting color error is less than the second threshold, setting the color of the pixel equal to the color of the second corresponding pixel.

16. The computer-readable storage medium of claim 15, the instructions further comprise:

when the first absolute color error is not less than the first threshold;

determining the second absolute color error between the pixel and the second corresponding pixel;

determining if the second absolute color error is less than the first threshold;

when the second absolute color error is less than the first threshold;

determining the second weak-sorting color error between the pixel and the second corresponding pixel;

determining if the second combination of the second absolute color error and the second weak-sorting color error is less than the second threshold;

when the second combination of the second absolute color error and the second weak-sorting color error is less than the second threshold, setting the color of the pixel equal to the color of the second corresponding pixel.

17. The computer-readable storage medium of claim 15, wherein the first and the second weak-sorting color errors are each defined as:

18. The computer-readable storage medium of claim 17, wherein the first and the second combinations are each defined as:

E_total=(3−

E2)*E1;

where E_total is a combination of the first or the second absolute error and the first or the second weak-sorting color error, E1 is the first or the second absolute error, and E2 is the first or the second weak-sorting color error.

19. The computer-readable storage medium of claim 3, wherein said transferring color from the third intermediate image to the first input image to generate the final image comprises:

converting the first input image and the third intermediate image from their original color space to lα

β

color space where l represents an achromatic luminance channel, α

represents a yellow-blue channel, and β

represents a red-green channel;

looping through each size of a neighborhood-window;

looping through each pixel of a plurality of pixels in the first input image and the third intermediate image;

determining mean and variance values of each color channel from neighboring pixels in the neighborhood-window in the first input image and corresponding neighboring pixels in the neighborhood-window in the third intermediate image;

applying a color transfer based on the mean and the variance values to the pixel;

convening the first input image from the lα

β

color space back to the original color space;

saving the first input image as the final image.

20. The computer-readable storage medium of claim 19, wherein the size of the neighborhood-window ranges from 7*7 to 5*5 to 3*3.

21. The computer-readable storage medium of claim 19, wherein said applying a color transfer comprises:

22. The computer-readable storage medium of claim 1, wherein said transferring color from the fifth image to the first image comprise:

convening the first image, the third image, and the fifth image from their original color space to lα

β

color space where l represents an achromatic luminance channel, α

represents a yellow-blue channel, and β

represents a red-green channel;

looping through each pixel of a plurality of pixels in the first image;

determining mean and variance values of each color channel of neighboring pixels of a corresponding pixel in the third image and neighboring pixels of a corresponding pixel in the fifth image;

determining a mean difference between the pixel and its four closest neighboring pixels in the third image;

when the mean difference is less than a threshold, determining new mean and new variance values based on bilinear interpolation of mean and variance values of the four closest neighbors in the third and the fifth images;

when the mean difference is not less than the threshold, determining the new mean and the new variance values based on 4-neighborhood interpolation of mean and variance values of the four closest neighbors in the third and the fifth images;

applying a color transfer based on the new mean and the new variance values to the pixel;

converting the first image from the lα

β

color space back to the original color space; and

saving the first image as the final image.

23. The computer-readable storage medium of claim 22, wherein the mean difference is defined as:

mean difference=|l_x+1,y+l_x,y+1l_x+1,y+1−

3l_x,y|+|α

_x+1,y+α

_x,y+1+α

a_x+1,y+1−

3*α

_x,y|+|β

_x+1,y+β

_x,y+1+β

_x+1,y+1−

3*β

_x,y|,

where (l_x,y, α

_x,y, β

_x,y), (l_x+1, α

_x+1,y, β

_{x+1, y}), (l_x,y+1, α

_x,y+1, β

_x,y+1), and (l_{x+1, y+1}, α

_{x+1, y+1}, β

_x+1,y+1) are four mean values of four neighboring pixel windows centered about the four closest neighboring pixels.

24. The computer-readable storage medium of claim 23, wherein said applying a color transfer comprises: