Image Processing Method and Apparatus

• US 20080219581A1
• Filed: 09/18/2007
• Published: 09/11/2008
• Est. Priority Date: 03/05/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Vision Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Zamfir, Adrian, Florea, Corneliu, Drimbarean, Alexandru, Albu, Felix, Corcoran, Peter, Poenaru, Vlad
• Granted Patent: US 8,417,055 B2
• Time in Patent Office: Days
• US Class Current: 382/261
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/003 Deblurring; Sharpening vibration or motion blur correction for cameras comprising an electronic image sensor H04N23/682
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

A method and apparatus for providing image processing. For one embodiment of the invention, an image processing apparatus is arranged to process a first relatively underexposed and sharp image of a scene, and a second relatively well exposed and blurred image, nominally of the same scene, the first and second images being derived from respective image sources. The apparatus provides a portion of the relatively first underexposed image as an input signal to an adaptive filter; and a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter. The adaptive filter produces an output signal from the input signal and the desired signal; and an image generator constructs a first filtered image from the output signal, relatively less blurred than the second image.

Claims

1. An image processing method comprising:

obtaining a first relatively underexposed, sharp image of a scene;

obtaining a second relatively well exposed, blurred image, nominally of the scene, the first and second images being derived from respective image sources;

providing a portion of the first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

adaptively filtering the input signal to produce an output signal; and

constructing a first filtered image from the output signal, relatively less blurred than the second image.

2. The image processing method according to claim 1 wherein the first and second images are in RGB format and wherein the image portions comprise a respective color plane of the first and second images.

3. The image processing method according to claim 2 wherein the adaptively filtering includes producing a set of filter coefficients from a combination of the input signal and an error signal, the error signal being the difference between the desired signal and the output signal;

and further comprising;

constructing each color plane of the first filtered image from a combination of the filter coefficients and color plane information of the input signal.

4. The image processing method according to claim 1 wherein the first and second images are in YCC format and wherein the image portions comprise a respective Y plane of the first and second images.

5. The image processing method according to claim 4 wherein the constructing the first filtered image comprises using the output signal as a Y plane of the first filtered image and using Cb and Cr planes of the input image as the Cb and Cr planes of the first filtered image.

6. The image processing method according to claim 1 comprising:

providing a portion of the first filtered image as the input signal to an adaptive filter;

providing a corresponding portion of the second image as a desired signal to the adaptive filter;

further adaptively filtering the input signal to produce a further output signal;

constructing a further filtered image from the further output signal relatively less blurred than the first filtered image

7. The image processing method according to claim 6 wherein:

the first and second images are in RGB format and, for producing the first filtered image, the image portions comprise a respective color plane of the first and second images;

the providing a portion of the first filtered image comprises converting the first filtered image to YCC format;

the method further comprises converting the second image to YCC format; and

wherein the image portions for further adaptive filtering comprise a respective Y plane of the converted images.

8. The image processing method according to claim 6 wherein:

the first and second images are in YCC format and, for producing the first filtered image, the image portions comprise a respective Y plane of the first and second images;

the providing a portion of the first filtered image comprises converting the first filtered image to RGB format;

the method further comprises converting the second image to RGB format; and

wherein the image portions for further adaptive filtering comprise a respective color plane of the converted images.

9. The image processing method according to claim 1 wherein the image source for the second image is of a relatively higher resolution than the image source for the first image.

10. The image processing method according to claim 9 further comprising aligning and interpolating the first image source to match the alignment and resolution of the second image source.

11. The image processing method as claimed in claim 9 comprising:

prior to adaptive filtering, estimating a point spread function, PSF, for the second image; and

de-blurring the second image using the estimated PSF.

12. The image processing method according to claim 11 wherein the de-blurring is performed in response to the estimated PSF being less than a pre-determined threshold.

13. The image processing method according to claim 1 wherein the image source for the second image is of a lower resolution than the image source for the first image.

14. The image processing method according to claim 13 further comprising aligning and interpolating the second source to match the alignment and resolution of the first source.

15. The image processing method according to claim 14 comprising:

responsive to the first and second sources being misaligned by more than a pre-determined threshold, providing the desired signal from a linear combination of the first and second image sources.

16. The image processing method according to claim 14 comprising:

responsive to the first and second sources being misaligned by more than a pre-determined threshold, providing the desired signal from a combination of phase values from one of the first and second image sources and amplitude values for the other of the first and second image sources.

17. The image processing method according to claim 6 wherein the adaptively filtering is performed one of row or column wise on the input signal and wherein further adaptively filtering is performed on the other of row or column wise on the input signal.

18. The image processing method according to claim 9 comprising the steps of amplifying the luminance characteristics of the under exposed image prior to the adaptively filtering.

19. The image processing method according to claim 1 comprising noise filtering the first filtered image.

20. The image processing method according to claim 19 comprising applying color correction to one of the first filtered image or the noise filtered image.

21. The image processing method according to claim 9 wherein the first image is one of an image acquired soon before or after the second image.

22. The image processing method according to claim 1 comprising:

acquiring a first partially exposed image from an image sensor;

acquiring a second further exposed image from the image sensor; and

subsequently resetting the image sensor,

wherein the obtaining the first relatively underexposed and sharp image of a scene comprises subtracting the first partially exposed image from the second further exposed image.

23. The image processing method according to claim 22 wherein the second relatively well exposed and blurred image is obtained from the image sensor immediately prior to the resetting.

24. An image processing apparatus arranged to process a first relatively underexposed and sharp image of a scene, and a second relatively well exposed and blurred image, nominally of the same scene, the first and second images being derived from respective image sources, the apparatus comprising:

means for providing a portion of the relatively first underexposed image as an input signal to an adaptive filter;

means providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

an adaptive filter arranged to produce an output signal from the input signal and the desired signal; and

an image generator arranged to construct a first filtered image from the output signal, relatively less blurred than the second image.