Image Deblur Based on Two Images

• US 20070223831A1
• Filed: 05/22/2006
• Published: 09/27/2007
• Est. Priority Date: 03/22/2006
• Status: Active Grant

• Current Assignee: ArcSoft, Inc.
• Original Assignee: ArcSoft, Inc.
• Inventors: Wang, Jin, Mei, Li
• Granted Patent: US 7,680,354 B2
• Time in Patent Office: Days
• US Class Current: 382/260
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/003 Deblurring; Sharpening vibration or motion blur correction for cameras comprising an electronic image sensor H04N23/682
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

A method includes (a) performing image deblur at the original size of underexposed and normally exposed images when their original image size is small. Otherwise, the method includes (b) downsizing the images and performing image deblur on the downsized images. The image deblur on downsized images includes (c) applying a global color transfer between the images if their difference of average gray values is small. Otherwise, the method includes four successive sub-procedures: (d) histogram customizing of the underexposed image based on the normally exposed image; (e) region segmentation for the histogram-customized underexposed image and local alignment of regions in the histogram-customized underexposed image with the normally exposed image; (f) color tuning for the histogram-customized underexposed image based on the local alignment and a global alignment between the underexposed and the normally exposed images; and (g) local color transfer with multi-size neighbor windows from between the images.

Claims

1. A method of generating a final image from a first image with a short exposure and a second image with a long exposure, comprising:

determining if the first and the second images are small images;

if the first and the second images are small images, using a deblurring procedure to generate the final image from the first and the second images;

if the first and the second images are not small images;

saving the first and the second images at a smaller size as third and fourth images, respectively;

using the deblurring procedure to generate a fifth image from the third and the fourth images; and

transferring color from the fifth image to the first image based on the third image to generate the final image;

displaying the final image.

2. The method of claim 1, wherein the first and the second images are small images if a maximum length of the first and the second images is less than a threshold, the maximum length being a greater of a width and a height of the first and the second images.

3. The method of claim 1, wherein said using the deblurring procedure comprises:

receiving first and second input images, wherein the first input images has the short exposure and the second input image has the long exposure;

determining if the first input image is relatively bright when compared to the second input image;

if the first input image is relatively bright, transferring color from the second input image to the first input image to generate an output image;

if the first input image is not relatively bright;

globally aligning the first and the second input images;

adjusting saturation and luminance of the first input image based on the second input image to generate a first intermediate image;

customizing a histogram of the first intermediate image based on the second input image to generate a second intermediate image;

segmenting the second intermediate image into visually related regions;

locally aligning the regions in the second intermediate image to regions in second input image;

adjusting color of the second intermediate image based on results of said globally aligning and said locally aligning to generate a third intermediate image; and

transferring color from the third intermediate image to the first input image to generate the output image.

4. The method of claim 3, wherein said determining if the first input image is relatively bright when compared to the second input image comprises:

determining a first mean of gray values of the first input image and a second mean of gray values of the second input images;

determining if the first mean of gray values is greater than a fixed percentage of the second mean of gray values;

wherein the first input image is relatively bright if the first mean of gray values is greater than the fixed percentage of the second mean of gray values.

5. The method of claim 3, wherein said transferring color from the second input image to the first input image comprises:

converting the first and the second input images from their original color space to 1α

β

color space;

determining mean and variance values of each color channel in the first and the second input images;

determining mapping tables based on the mean and the variance values;

mapping the first input image with the mapping tables;

converting the first input image from the 1α

β

color space back to the original color space;

saving the first input image as the output image.

6. The method of claim 5, wherein said determining mapping tables from the mean and the variance values comprises generating a mapping table for each color channel based on a color transfer formula defined as follows:

7. The method of claim 3, prior to said globally aligning the first and the second input images, further comprises:

determining a gamma correction factor from a first mean of gray values of the first input image and a second mean of gray values of the second input image;

gamma correcting the first input images and the first mean of gray values using the gamma correction factor; and

adjusting the gray values of the second input image so the second mean of gray values is equal to the first mean of the gray values.

8. The method of claim 7, wherein:

the gamma correction factor is defined as;

$\mathrm{\gamma <br><br>}=1.0+\frac{{\mathrm{Mean}}_{\mathrm{IH}}-{\mathrm{Mean}}_{\mathrm{IL}}}{{\mathrm{Mean}}_{\mathrm{IL}}\times <br><br>20}+\frac{\mathrm{Max}<br><br>\left(1.0,\frac{\left(42-{\mathrm{Mean}}_{\mathrm{IL}}\right)*{\mathrm{Mean}}_{\mathrm{IH}}}{{\mathrm{Mean}}_{\mathrm{IL}}}\right)}{200},$

where γ

is the gamma correction factor, Mean_IL is the first mean of gray

values of the first input image, and Mean_IH is the second mean of gray values of the second input image; and

said gamma correcting the first input image and the first mean of gray values comprises using a table based on;

$\mathrm{Gamma}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{Table}<br><br>\left[i\right]=\frac{\left(i^{\frac{1}{y}}\right)}{{255}^{\frac{1}{y}}}\times <br><br>255,\left(i=0,1,\mathrm{\dots <br><br>}<br><br>\text{\hspace{1em}<br><br>},255\right),$

where GammaTable [i] represents a value in the table after gamma correction, and i represents an original value prior to gamma correction.

9. The method of claim 7, wherein said adjusting gray values of the second input image comprises adding a difference of the first mean of gray values from a second mean of gray values to gray values of the pixels in the second input image.

10. The method of claim 3, wherein said globally aligning the first and the second input images comprises:

looping through each step level, each step level comprising a step size and a search region;

looping through each displacement based on the step size in the search region;

determining an absolute gray error between the first and the second input images;

setting a displacement with a smallest absolute gray error as a best alignment for a current step level and a starting point for a next level.

11. The method of claim 3, wherein said adjusting saturation and luminance of the first input image comprises:

determining gamma correction factors for the saturation and the luminance,

wherein the gamma correction factors comprises;

$\mathrm{\gamma <br><br>}=1.0+\frac{{\mathrm{Mean}}_{\mathrm{IH}}-{\mathrm{Mean}}_{\mathrm{IL}}}{{\mathrm{Mean}}_{\mathrm{IL}}\times <br><br>20}+\frac{\mathrm{Max}<br><br>\left(1.0,\frac{\left(42-{\mathrm{Mean}}_{\mathrm{IL}}\right)*{\mathrm{Mean}}_{\mathrm{IH}}}{{\mathrm{Mean}}_{\mathrm{IL}}}\right)}{200},<br><br>{\mathrm{\gamma <br><br>}}_{\mathrm{saturation}}=\min <br><br>\left(1.23,\frac{\mathrm{\gamma <br><br>}-1.0}{3.0}+1.0\right),<br><br>{\mathrm{\gamma <br><br>}}_{\mathrm{luminance}}=1.0+\frac{{\mathrm{Mean}}_{\mathrm{IH}}-{\mathrm{Mean}}_{\mathrm{IL}}}{{\mathrm{Mean}}_{\mathrm{IH}}\times <br><br>1.25},<br><br>$

where γ

is a gamma correction factor, γ

_saturation is the gamma correction factor for the saturation, γ

is the gamma correction factor for the

luminance, Mean_IL is a mean of gray values of the first input image, and Mean_IH is a mean of gray values of the second input image;

converting the first input image from its original color space to HSL color space;

applying gamma correction to the first input image using a table based on;

$\mathrm{Gamma}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{Table}<br><br>\left[i\right]=\frac{\left(i^{\frac{1}{y}}\right)}{{255}^{\frac{1}{y}}}\times <br><br>255,\left(i=0,1,\mathrm{\dots <br><br>}<br><br>\text{\hspace{1em}<br><br>},255\right),$

where GammaTable [i] represents a value in the table after gamma correction, and i represents an original value prior to gamma correction;

converting the first input image from the HSL color space to the original color space and saving the first input image as the first intermediate image.

12. The method of claim 3, wherein said customizing a histogram of the first intermediate image comprises:

normalizing color channels of the first intermediate image and the second input image in their overlapping areas determined from said globally aligning;

determining histograms of the color channels of the first intermediate image and the second input image in the overlapping areas;

smoothing the histograms;

building channel mapping tables that map the histograms of the second input image to the histograms of the first intermediate image by histogram specification;

inverse normalizing the channel mapping tables; and

mapping the first intermediate image with the mapping tables to generate the second intermediate image.

13. The method of claim 3, wherein said locally aligning the regions in the second intermediate image to regions in second input image comprises:

looping through each step level, each step level comprising a step size and a search region size;

looping through each possible displacement based on the step size in the search region;

looping through each region in the second intermediate image;

determining an absolute gray error between the region in the second intermediate image and a corresponding region in the second input image;

setting a displacement with a smallest absolute gray error as a best alignment of the region.

14. The method of claim 3, wherein said adjusting color of the second intermediate image comprises:

looping through each pixel in the second intermediate image;

determining a first absolute color error between the pixel and a first corresponding pixel in the second input image locally aligned with the pixel by said locally aligning;

determining if the first absolute color error is less than a first threshold;

if the first absolute color error is less than the first threshold;

determining a first weak-sorting color error between the pixel and the first corresponding pixel;

determining if a first combination of the first absolute color error and the first weak-sorting color error is less than a second threshold;

if the first combination of the first absolute color error and the first weak-sorting color error is less than the second threshold, setting the color of the pixel equal to the color of the first corresponding pixel.

15. The method of claim 14, further comprising:

if the first combination of the first absolute color error and the first weak-sorting color error is not less than the second threshold;

determining a second absolute color error between the pixel and a second corresponding pixel in the second input image globally aligned with the pixel by said globally aligning;

determining if the second absolute color error is less than the first threshold;

if the second absolute color error is less than the first threshold;

determining a second weak-sorting color error between the pixel and the second corresponding pixel;

determining if a second combination of the second absolute color error and the second weak-sorting color error is less than the second threshold;

if the second combination of the second absolute color error and the second weak-sorting color error is less than the second threshold, setting the color of

the pixel equal to the color of the second corresponding pixel.

16. The method of claim 15, further comprising:

if the first absolute color error is not less than the first threshold;

determining the second absolute color error between the pixel and the second corresponding pixel;

determining if the second absolute color error is less than the first threshold;

if the second absolute color error is less than the first threshold;

determining the second weak-sorting color error between the pixel and the second corresponding pixel;

determining if the second combination of the second absolute color error and the second weak-sorting color error is less than the second threshold;

if the second combination of the second absolute color error and the second weak-sorting color error is less than the second threshold, setting the color of the pixel equal to the color of the second corresponding pixel.

17. The method of claim 15, wherein the first and the second weak-sorting color errors are each defined as:

18. The method of claim 17, wherein the first and the second combinations are each defined as:

E_total=(3−

E2)*E1;

where E_total is a combination of the first or the second absolute error and the first or the second weak-sorting color error, E1 is the first or the second absolute error, and E2 is the first or the second weak-sorting color error.

19. The method of claim 3, wherein said transferring color from the third intermediate image to the first input image to generate the output image comprises:

converting the first input image and the third intermediate image from their original color space to 1α

β

color space;

looping through each size of a neighborhood-window;

looping through each pixel of a plurality of pixels in the first input image and the third intermediate image;

determining mean and variance values of each color channel from neighboring pixels in the neighborhood-window in the first input image and corresponding neighboring pixels in the neighborhood-window in the third intermediate image;

applying a color transfer based on the mean and the variance values to the pixel;

converting the first input image from the 1α

β

color space back to the original color space;

saving the first input image as the output image.

20. The method of claim 16, wherein the size of the neighborhood-window range from 7*7 to 5*5 to 3*3.

21. The method of claim 16, wherein said applying a color transfer comprises:

22. The method of claim 1, wherein said transferring color from the fifth image to the first image comprising:

converting the first image, the third image, and the fifth image from its original color space to 1α

β

color space;

looping through each pixel of a plurality of pixels in the first image;

determining mean and variance values of each color channel of neighboring pixels of a corresponding pixel in the third image and neighboring pixels of a corresponding pixel in the fifth image;

determining a mean difference between the pixel and its four closest neighboring pixels in the third image;

if the mean difference is less than a threshold, determining new mean and new variance values based on bilinear interpolation of mean and variance values of the four closest neighbors in the third and the fifth images;

if the mean difference is not less than the threshold, determining the new mean and new variance values based on 4-neighborhood interpolation of mean and variance values of the four closest neighbors in the third and the fifth images;

applying a color transfer based on the new mean and the new variance values to the pixel;

converting the first input image from the 1α

β

color space back to the original color space;

saving the first image as the final image.

23. The method of claim 22, wherein the mean difference is defined as:

Mean difference=|1_x+1,y+1_x,y+1+1_x+1,y+1−

3*1_x,y|+|α

_x+1,y+α

_x,y+1+α

a_x+1,y+1−

3*α

_x,y|+|β

_x+1,y+β

_x,y+1β

_x+1,y+1−

3*β

_x,y|,

where (1_x,y, α

_x,y, β

_x,y), (1_x+1,y, α

_x+1,y, β

_x+1,y), (1_x,y+1, α

_x,y+1, β

_x,y+1), and (1_x+1,y+1, α

_x+1,y+1, β

_x+1,y+1) are four mean values of four neighboring pixel windows centered about the closest four neighboring pixels.

24. The method of claim 23, wherein said applying a color transfer comprises: