Image processing method and apparatus

• US 8,417,055 B2
• Filed: 09/18/2007
• Issued: 04/09/2013
• Est. Priority Date: 03/05/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Albu, Felix, Steinberg, Eran, Drimbarean, Alexandru, Florea, Corneliu, Zamfir, Adrian, Corcoran, Peter, Poenaru, Vlad
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): Roz, Mark
• Application Number:
  US11/856,721
• Publication Number:
  US 20080219581A1
• Time in Patent Office: 2,030 Days
• Field of Search: 382/275, 382/255
• US Class Current: 382/275
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

A method and apparatus for providing image processing. For one embodiment of the invention, an image processing apparatus is arranged to process a first relatively underexposed and sharp image of a scene, and a second relatively well exposed and blurred image, nominally of the same scene, the first and second images being derived from respective image sources. The apparatus provides a portion of the relatively first underexposed image as an input signal to an adaptive filter; and a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter. The adaptive filter produces an output signal from the input signal and the desired signal; and an image generator constructs a first filtered image from the output signal, relatively less blurred than the second image.

Claims

1. An image processing method comprising:

obtaining a first image and a second image of approximately a same scene, the first image being a relatively underexposed, sharp image of the scene compared with the second image, the second image comprising a relatively well exposed, blurred image compared with the first image, the first and second images being derived from respective image sources;

providing a portion of the first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

adaptively filtering the input signal to produce an output signal, including multiplying the input signal by a coefficient that is calculated from values of the input signal and the desired signal; and

constructing a first filtered image from the output signal, relatively less blurred than the second image, and

wherein the first and second images are in RGB format and wherein the image portions comprise a respective color plane of the first and second images, and

wherein the adaptively filtering and constructing of the first filtered image comprise applying the color plane of the relatively well-exposed, blurred second image to the color plane of the relatively underexposed, sharp first image to enhance the color plane of the first image to construct the first filtered image.

2. The image processing method according to claim 1 wherein the adaptively filtering includes producing a set of filter coefficients from a combination of the input signal and an error signal, the error signal being the difference between the desired signal and the output signal;

and further comprising;

constructing each color plane of the first filtered image from a combination of the filter coefficients and color plane information of the input signal.

5. The image processing method according to claim 1 comprising:

providing a portion of the first filtered image as the input signal to an adaptive filter;

providing a corresponding portion of the second image as a desired signal to the adaptive filter;

further adaptively filtering the input signal to produce a further output signal;

constructing a further filtered image from the further output signal relatively less blurred than the first filtered image.

6. The image processing method according to claim 5 wherein:

the first and second images are in RGB format and, for producing the first filtered image, the image portions comprise a respective color plane of the first and second images;

the providing a portion of the first filtered image comprises converting the first filtered image to YCC format;

the method further comprises converting the second image to YCC format; and

wherein the image portions for further adaptive filtering comprise a respective Y plane of the converted images.

8. The image processing method according to claim 1 or claim 4, wherein the image source for the second image is of a relatively higher resolution than the image source for the first image.

9. The image processing method according to claim 8 further comprising aligning and interpolating the first image source to match the alignment and resolution of the second image source.

10. The image processing method according to claim 1 or claim 4, wherein the image source for the second image is of a lower resolution than the image source for the first image.

11. The image processing method according to claim 10 further comprising aligning and interpolating the second source to match the alignment and resolution of the first source.

12. The image processing method according to claim 11 comprising:

responsive to the first and second sources being misaligned by more than a pre-determined threshold, providing the desired signal from a linear combination of the first and second image sources.

13. The image processing method according to claim 11 comprising:

responsive to the first and second sources being misaligned by more than a pre-determined threshold, providing the desired signal from a combination of phase values from one of the first and second image sources and amplitude values for the other of the first and second image sources.

14. The image processing method according to claim 5 wherein the adaptively filtering is performed one of row or column wise on the input signal and wherein further adaptively filtering is performed on the other of row or column wise on the input signal.

15. The image processing method according to claim 8 further comprising amplifying the luminance characteristics of the under exposed image prior to the adaptively filtering.

16. The image processing method according to claim 1 or claim 4 further comprising noise filtering the first filtered image.

17. The image processing method according to claim 16 further comprising applying color correction to one of the first filtered image or the noise filtered image.

18. The image processing method according to claim 8 wherein the first image is one of an image acquired soon before or after the second image.

19. The image processing method according to claim 1 or claim 4 comprising:

acquiring a first partially exposed image from an image sensor;

acquiring a second further exposed image from the image sensor; and

subsequently resetting the image sensor,

wherein the obtaining the first relatively underexposed and sharp image of a scene comprises subtracting the first partially exposed image from the second further exposed image.

20. The image processing method according to claim 19 wherein the second relatively well exposed and blurred image is obtained from the image sensor immediately prior to the resetting.

3. An image processing method comprising:

obtaining a first image and a second image of approximately a same scene, the first image being a relatively underexposed, sharp image of the scene compared with the second image, the second image comprising relatively well exposed, blurred image compared with the first image, the first and second images being derived from respective image sources;

providing a portion of the first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

adaptively filtering the input signal to produce an output signal, including multiplying the input signal by a coefficient that is calculated from values of the input signal and the desired signal; and

constructing a first filtered image from the output signal, relatively less blurred than the second image, and

wherein the first and second images are in YCC format and wherein the image portions comprise a respective Y plane of the first and second images, and

wherein the adaptively filtering and constructing of the first filtered image comprise applying the Y plane of the relatively well-exposed, blurred second image to the color plane of the relatively underexposed, sharp first image to enhance the Y plane of the first image to construct the first filtered image.

4. The image processing method according to claim 3 wherein the constructing the first filtered image comprises using the output signal as a Y plane of the first filtered image and using Cb and Cr planes of the input image as the Cb and Cr planes of the first filtered image.

7. The image processing method according to claim 3, further comprising:

providing a portion of the first filtered image as the input signal to an adaptive filter;

providing a corresponding portion of the second image as a desired signal to the adaptive filter;

further adaptively filtering the input signal to produce a further output signal;

constructing a further filtered image from the further output signal relatively less blurred than the first filtered image, and

wherein the first and second images are in YCC format and, for producing the first filtered image, the image portions comprise a respective Y plane of the first and second images;

the providing a portion of the first filtered image comprises converting the first filtered image to RGB format;

the method further comprises converting the second image to RGB format; and

wherein the image portions for further adaptive filtering comprise a respective color plane of the converted images.

21. An image acquisition and processing apparatus including a lens and image sensor for acquiring digital images and a processor arranged to acquire and process a first image of a scene that is relatively underexposed and sharp compared with a second relatively well exposed and blurred image also acquired by the apparatus, of approximately the same scene, the first and second images being derived from respective image sources, the apparatus comprising:

means for providing a portion of the relatively first underexposed image as an input signal to an adaptive filter;

means for providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

an adaptive filter arranged to produce an output signal from the input signal and the desired signal, the adaptive filter being arranged to multiply the input signal by a coefficient that is calculated from values of the input signal and the desired signal; and

an image generator arranged to construct a first filtered image from the output signal, relatively less blurred than the second image, and

wherein the first and second images are in RGB format and wherein the image portions comprise a respective color plane of the first and second images, and

wherein the adaptive filter and image generator are configured to apply the color plane of the relatively well-exposed, blurred second image to the color plane of the relatively underexposed, sharp first image to enhance the color plane of the first image to construct the first filtered image.

22. One or more tangible, non-transitory processor readable media having code embodied therein for programming one or more processors to perform an image processing method, wherein the method comprises:

obtaining a first image and a second image of approximately a same scene, the first image being a relatively underexposed, sharp image of the scene compared with the second image, the second image comprising a relatively well exposed, blurred image compared with the first image, the first and second images being derived from respective image sources;

providing a portion of the first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

adaptively filtering the input signal to produce an output signal, including multiplying the input signal by a coefficient that is calculated from values of the input signal and the desired signal; and

constructing a first filtered image from the output signal, relatively less blurred than the second image, and

wherein the first and second images are in RGB format and wherein the image portions comprise a respective color plane of the first and second images, and

wherein the adaptively filtering and constructing of the first filtered image comprise applying the color plane of the relatively well-exposed, blurred second image to the color plane of the relatively underexposed, sharp first image to enhance the color plane of the first image to construct the first filtered image.

23. The one or more tangible, non-transitory processor readable media according to claim 22 wherein the adaptively filtering includes producing a set of filter coefficients from a combination of the input signal and an error signal, the error signal being the difference between the desired signal and the output signal;

and further comprising;

constructing each color plane of the first filtered image from a combination of the filter coefficients and color plane information of the input signal.

26. The one or more tangible, non-transitory processor readable media according to claim 22, wherein the method comprises:

providing a portion of the first filtered image as the input signal to an adaptive filter;

providing a corresponding portion of the second image as a desired signal to the adaptive filter;

further adaptively filtering the input signal to produce a further output signal;

constructing a further filtered image from the further output signal relatively less blurred than the first filtered image.

27. The one or more tangible, non-transitory processor readable media according to claim 26 wherein:

the first and second images are in RGB format and, for producing the first filtered image, the image portions comprise a respective color plane of the first and second images;

the providing a portion of the first filtered image comprises converting the first filtered image to YCC format;

the method further comprises converting the second image to YCC format; and

wherein the image portions for further adaptive filtering comprise a respective Y plane of the converted images.

28. The one or more tangible, non-transitory processor readable media according to claim 22 further comprising providing a portion of the first filtered image as the input signal to an adaptive filter;

providing a corresponding portion of the second image as a desired signal to the adaptive filter;

further adaptively filtering the input signal to produce a further output signal;

constructing a further filtered image from the further output signal relatively less blurred than the first filtered image, and

wherein the first and second images are in YCC format and, for producing the first filtered image, the image portions comprise a respective Y plane of the first and second images;

the providing a portion of the first filtered image comprises converting the first filtered image to RGB format;

the method further comprises converting the second image to RGB format; and

wherein the image portions for further adaptive filtering comprise a respective color plane of the converted images.

29. The one or more tangible, non-transitory processor readable media according to claim 22 or claim 24 wherein the image source for the second image is of a relatively higher resolution than the image source for the first image.

30. The one or more tangible, non-transitory processor readable media according to claim 29 wherein the method further comprises aligning and interpolating the first image source to match the alignment and resolution of the second image source.

31. The one or more tangible, non-transitory processor readable media according to claim 22 or claim 24 wherein the image source for the second image is of a lower resolution than the image source for the first image.

32. The one or more tangible, non-transitory processor readable media according to claim 31 wherein the method further comprises aligning and interpolating the second source to match the alignment and resolution of the first source.

33. The one or more tangible, non-transitory processor readable media according to claim 32 wherein the method further comprises:

responsive to the first and second sources being misaligned by more than a pre-determined threshold, providing the desired signal from a linear combination of the first and second image sources.

34. The one or more tangible, non-transitory processor readable media according to claim 32 wherein the method further comprises:

responsive to the first and second sources being misaligned by more than a pre-determined threshold, providing the desired signal from a combination of phase values from one of the first and second image sources and amplitude values for the other of the first and second image sources.

35. The one or more tangible, non-transitory processor readable media according to claim 22 or claim 24 wherein the adaptively filtering is performed one of row or column wise on the input signal and wherein further adaptively filtering is performed on the other of row or column wise on the input signal.

36. The one or more tangible, non-transitory processor readable media according to claim 28 wherein the method further comprises amplifying the luminance characteristics of the under exposed image prior to the adaptively filtering.

37. The one or more tangible, non-transitory processor readable media according to claim 22 or claim 24 wherein the method further comprises noise filtering the first filtered image.

38. The one or more tangible, non-transitory processor readable media according to claim 37 wherein the method further comprises applying color correction to one of the first filtered image or the noise filtered image.

39. The one or more tangible, non-transitory processor readable media according to claim 29 wherein the first image is one of an image acquired soon before or after the second image.

40. The one or more tangible, non-transitory processor readable media according to claim 22 or claim 24 wherein the method further comprises:

acquiring a first partially exposed image from an image sensor;

acquiring a second further exposed image from the image sensor; and

subsequently resetting the image sensor,

wherein the obtaining the first relatively underexposed and sharp image of a scene comprises subtracting the first partially exposed image from the second further exposed image.

41. The one or more tangible, non-transitory processor readable media according to claim 40 wherein the second relatively well exposed and blurred image is obtained from the image sensor immediately prior to the resetting.

24. One or more tangible, non-transitory processor readable media having code embodied therein for programming one or more processors to perform an image processing method, wherein the method comprises:

obtaining a first image and a second image of approximately a same scene, the first image being a relatively underexposed, sharp image of the scene compared with the second image, the second image comprising a relatively well exposed, blurred image compared with the first image, the first and second images being derived from respective image sources;

providing a portion of the first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

adaptively filtering the input signal to produce an output signal, including multiplying the input signal by a coefficient that is calculated from values of the input signal and the desired signal; and

constructing a first filtered image from the output signal, relatively less blurred than the second image, and

wherein the first and second images are in YCC format and wherein the image portions comprise a respective Y plane of the first and second images, and

wherein the adaptively filtering and constructing of the first filtered image comprise applying the Y plane of the relatively well-exposed, blurred second image to the color plane of the relatively underexposed, sharp first image to enhance the Y plane of the first image to construct the first filtered image.

25. The one or more tangible, non-transitory processor readable media according to claim 24 wherein the constructing the first filtered image comprises using the output signal as a Y plane of the first filtered image and using Cb and Cr planes of the input image as the Cb and Cr planes of the first filtered image.

42. A digital image acquisition device, comprising:

a lens and image sensor for acquiring digital images;

a processor;

a memory having processor readable code embodied therein for programming one or more processors to perform an image processing method, wherein the method comprises;

obtaining a first image and a second image of approximately a same scene, the first image being a relatively underexposed, sharp image of the scene compared with the second image, the second image comprising a;

relatively well exposed, blurred image compared with the first image, the first and second images being derived from respective image sources;

providing a portion of the first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

adaptively filtering the input signal to produce an output signal, including multiplying the input signal by a coefficient that is calculated from values of the input signal and the desired signal; and

constructing a first filtered image from the output signal, relatively less blurred than the second image, and

wherein the first and second images are in RGB format and wherein the image portions comprise a respective color plane of the first and second images, and

wherein the adaptively filtering and constructing of the first filtered image comprise applying the color plane of the relatively well-exposed, blurred second image to the color plane of the relatively underexposed, sharp first image to enhance the color plane of the first image to construct the first filtered image.

43. The device according to claim 42 wherein the adaptively filtering includes producing a set of filter coefficients from a combination of the input signal and an error signal, the error signal being the difference between the desired signal and the output signal;

and further comprising;

constructing each color plane of the first filtered image from a combination of the filter coefficients and color plane information of the input signal.

46. The device according to claim 42, wherein the method comprises:

providing a portion of the first filtered image as the input signal to an adaptive filter;

providing a corresponding portion of the second image as a desired signal to the adaptive filter;

further adaptively filtering the input signal to produce a further output signal;

constructing a further filtered image from the further output signal relatively less blurred than the first filtered image.

47. The device according to claim 46 wherein:

the first and second images are in RGB format and, for producing the first filtered image, the image portions comprise a respective color plane of the first and second images;

the providing a portion of the first filtered image comprises converting the first filtered image to YCC format;

the method further comprises converting the second image to YCC format; and

wherein the image portions for further adaptive filtering comprise a respective Y plane of the converted images.

49. The device according to claim 42 or claim 44 wherein the image source for the second image is of a relatively higher resolution than the image source for the first image.

50. The device according to claim 49 wherein the method further comprises aligning and interpolating the first image source to match the alignment and resolution of the second image source.

51. The device according to claim 42 or claim 44 wherein the image source for the second image is of a lower resolution than the image source for the first image.

52. The device according to claim 51 wherein the method further comprises aligning and interpolating the second source to match the alignment and resolution of the first source.

53. The device according to claim 52 wherein the method further comprises:

responsive to the first and second sources being misaligned by more than a pre-determined threshold, providing the desired signal from a linear combination of the first and second image sources.

54. The device according to claim 52 wherein the method further comprises:

responsive to the first and second sources being misaligned by more than a pre-determined threshold, providing the desired signal from a combination of phase values from one of the first and second image sources and amplitude values for the other of the first and second image sources.

55. The device according to claim 42 or claim 44 wherein the adaptively filtering is performed one of row or column wise on the input signal and wherein further adaptively filtering is performed on the other of row or column wise on the input signal.

57. The device according to claim 42 wherein the method further comprises noise filtering the first filtered image.

58. The device according to claim 57 wherein the method further comprises applying color correction to one of the first filtered image or the noise filtered image.

59. The device according to claim 42 or claim 44 wherein the first image is one of an image acquired soon before or after the second image.

60. The device according to claim 42 or claim 44, wherein the method further comprises:

acquiring a first partially exposed image from an image sensor;

acquiring a second further exposed image from the image sensor; and

subsequently resetting the image sensor,

wherein the obtaining the first relatively underexposed and sharp image of a scene comprises subtracting the first partially exposed image from the second further exposed image.

61. The device according to claim 60 wherein the second relatively well exposed and blurred image is obtained from the image sensor immediately prior to the resetting.

44. A digital image acquisition device, comprising:

a lens and image sensor for acquiring digital images;

a processor;

a memory having processor readable code embodied therein for programming one or more processors to perform an image processing method, wherein the method comprises;

obtaining a first image and a second image of approximately a same scene, the first image being a relatively underexposed, sharp image of the scene compared with the second image, the second image comprising a relatively well exposed, blurred image compared with the first image, the first and second images being derived from respective image sources;

providing a portion of the first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

adaptively filtering the input signal to produce an output signal, including multiplying the input signal by a coefficient that is calculated from values of the input signal and the desired signal; and

constructing a first filtered image from the output signal, relatively less blurred than the second image, and

wherein the first and second images are in YCC format and wherein the image portions comprise a respective Y plane of the first and second images, and

wherein the adaptively filtering and constructing of the first filtered image comprise applying the Y plane of the relatively well-exposed, blurred second image to the color plane of the relatively underexposed, sharp first image to enhance the Y plane of the first image to construct the first filtered image.

45. The device according to claim 44 wherein the constructing the first filtered image comprises using the output signal as a Y plane of the first filtered image and using Cb and Cr planes of the input image as the Cb and Cr planes of the first filtered image.

48. The device according to claim 44 wherein the method comprises:

providing a portion of the first filtered image as the input signal to an adaptive filter;

providing a corresponding portion of the second image as a desired signal to the adaptive filter;

further adaptively filtering the input signal to produce a further output signal; and

constructing a further filtered image from the further output signal relatively less blurred than the first filtered image, and

for producing the first filtered image, the image portions comprise a respective Y plane of the first and second images; and

the providing a portion of the first filtered image comprises converting the first filtered image to RGB format; and

the method further comprises converting the second image to RGB format; and

wherein the image portions for further adaptive filtering comprise a respective color plane of the converted images.

56. The device according to claim 48 wherein the method further comprises amplifying the luminance characteristics of the under exposed image prior to the adaptively filtering.

62. An image acquisition and processing apparatus including a lens and image sensor for acquiring digital images and a processor arranged to acquire and process a first image of a scene that is relatively underexposed and sharp compared with a second relatively well exposed and blurred image also acquired by the apparatus, of approximately the same scene, the first and second images being derived from respective image sources, the apparatus comprising:

means for providing a portion of the relatively first underexposed image as an input signal to an adaptive filter;

means providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

an adaptive filter arranged to produce an output signal from the input signal and the desired signal, the adaptive filter being arranged to multiply the input signal by a coefficient that is calculated from values of the input signal and the desired signal; and

an image generator arranged to construct a first filtered image from the output signal, relatively less blurred than the second image, and

wherein the first and second images are in YCC format and wherein the image portions comprise a respective Y plane of the first and second images, and

wherein the adaptive filter and image generator are configured to apply the Y plane of the relatively well-exposed, blurred second image to the Y plane of the relatively underexposed, sharp first image to enhance the Y plane of the first image to construct the first filtered image.

63. An image processing method comprising:

obtaining a first image and a second image of approximately a same scene, the first image being a relatively underexposed, sharp image of the scene compared with the second image, the second image comprising relatively well exposed, blurred image compared with the first image, the first and second images being derived from respective image sources;

providing a portion of the first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter;

adaptively filtering the input signal to produce an output signal, including multiplying the input signal by a coefficient that is calculated from values of the input signal and the desired signal; and

constructing a first filtered image from the output signal, relatively less blurred than the second image, and

wherein the first and second images are in YCC format and wherein the image portions comprise a respective Y plane of the first and second images, and

wherein the method further comprises;

providing a portion of the first filtered image as the input signal to an adaptive filter;

providing a corresponding portion of the second image as a desired signal to the adaptive filter;

further adaptively filtering the input signal to produce a further output signal;

constructing a further filtered image from the further output signal relatively less blurred than the first filtered image, without calculating any blur function to assist in deblurring the second relatively well-exposed, blurred image, and

wherein the adaptively filtering is performed one of row or column wise on the input signal and wherein further adaptively filtering is performed on the other of row or column wise on the input signal, and

wherein the adaptively filtering and constructing of the first filtered image comprise applying the Y plane of the relatively well-exposed, blurred second image to the Y plane of the relatively underexposed, sharp first image to enhance the Y plane of the first image to construct the first filtered image.

64. The image processing method according to claim 63 further comprising amplifying the luminance characteristics of the under exposed image prior to the adaptively filtering.

65. The one or more tangible, non-transitory processor readable media according to claim 63 wherein the method further comprises amplifying the luminance characteristics of the under exposed image prior to the adaptively filtering.