Image processing method and apparatus

• US 8,989,516 B2
• Filed: 12/16/2008
• Issued: 03/24/2015
• Est. Priority Date: 09/18/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Albu, Felix, Zamfir, Adrian
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Roz, Mark
• Application Number:
  US12/336,416
• Publication Number:
  US 20090179999A1
• Time in Patent Office: 2,289 Days
• Field of Search: 382/275, 382/255, 382/167
• US Class Current: 382/275
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

An image processing apparatus is arranged to process a first relatively underexposed and sharp image of a scene, and a second relatively well exposed and blurred image, nominally of the same scene, the first and second images being derived from respective image sources. The apparatus provides a portion of the relatively first underexposed image as an input signal to an adaptive filter; and a corresponding portion of the second relatively well exposed image as a desired signal to the adaptive filter. The adaptive filter produces an output signal from the input signal and the desired signal; and an image generator constructs a first filtered image from the output signal, relatively less blurred than the second image.

Claims

1. An image processing method to process a first relatively underexposed and sharp image of a scene, and a second relatively well exposed and blurred image nominally of the same scene, said method comprising:

providing a portion of said first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of said second relatively well exposed image as a desired signal to said adaptive filter;

producing an output signal from said input signal and said desired signal by multiplying the input signal by one or more coefficients that are calculated from values of the input signal and the desired signal;

constructing a first filtered image, from said output signal, that is relatively less blurred than said second image; and

displaying, transmitting, projecting or storing the first filtered image or a further processed version, or combinations thereof;

wherein the method is performed by one or more computing devices.

2. An image processing method according to claim 1 wherein said first and second images are in YCC format and has image portions that comprise a respective Y plane of said first and second images.

3. An image processing method according to claim 2 wherein said constructing said first filtered image comprises using said output signal as a Y plane of said first filtered image and using Cb and Cr planes of said input image as the Cb and Cr planes of said first filtered image.

4. An image processing method according to claim 1 wherein:

said first image is derived from a first image source;

said second image is derived from a second image source; and

said second image source for said second image is of a relatively higher resolution than said first image source for said first image.

5. An image processing method according to claim 4 further comprising aligning and interpolating said first image source to match the alignment and resolution of said second image source.

6. An image processing method according to claim 1 wherein said first image is one of an image acquired soon before or after said second image.

7. An image processing method according to claim 1 wherein:

said first image is derived from a first image source;

said second image is derived from a second image source; and

said second image source for said second image is of a relatively lower resolution than said first image source for said first image.

8. An image processing method according to claim 7 further comprising aligning and interpolating said second image source to match the alignment and resolution of said first image source.

9. An image processing method according to claim 8 comprising, responsive to said first and second image sources being misaligned by more than a pre-determined threshold, providing said desired signal from a linear combination of said first and second image sources.

10. An image processing method according to claim 8 comprising, responsive to said first and second image sources being misaligned by more than a pre-determined threshold, providing said desired signal from a combination of phase values from one of said first and second image sources and amplitude values for the other of said first and second image sources.

11. An image processing method according to claim 1 comprising noise filtering said first filtered image.

12. An image processing method according to claim 1 comprising applying color correction to the first filtered image or a noise filtered image or both.

13. An image processing method according to claim 1 comprising:

acquiring a first partially exposed image from an image sensor;

acquiring a second further exposed image from said image sensor; and

subsequently resetting said image sensor,

wherein said first relatively underexposed and sharp image of a scene is obtained by subtracting said first partially exposed image from said second further exposed image.

14. An image processing method according to claim 13 wherein said second relatively well exposed and blurred image is obtained from said image sensor just prior to said resetting.

15. An image processing method comprising:

obtaining a first relatively underexposed and sharp image of a scene;

obtaining a second relatively well exposed and blurred image, nominally of the same scene;

providing a portion of said first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of said second relatively well exposed image as a desired signal to said adaptive filter;

adaptively filtering said input signal to produce an output signal by multiplying the input signal by one or more coefficients that are calculated from values of the input signal and the desired signal;

constructing a first filtered image, from said output signal, relatively less blurred than said second image; and

displaying, transmitting, projecting or storing the first filtered image or a further processed version, or combinations thereof;

,

wherein the method is performed using one or more computing devices.

16. An image processing method according to claim 15 comprising:

providing a portion of said first filtered image as said input signal to an adaptive filter;

providing a corresponding portion of said second image as a desired signal to said adaptive filter;

further adaptively filtering said input signal to produce a further output signal;

constructing a further filtered image from said further output signal relatively less blurred than said first filtered image.

17. An image processing method according to claim 16, wherein said first and second images are in RGB format to produce said first filtered image having image portions;

wherein said image portions comprise a respective color plane of said first and second images;

wherein said providing a portion of said first filtered image comprises converting said first filtered image to YCC format;

wherein said method further comprises converting said second image to YCC format; and

wherein said image portions for further adaptive filtering comprise a respective Y plane of said converted image.

18. An image processing method according to claim 17,

wherein said first and second images are in YCC format and, for producing said first filtered image;

wherein said image portions comprise a respective Y plane of said first and second images;

wherein said providing a portion of said first filtered image comprises converting said first filtered image to RGB format;

wherein said method further comprises converting said second image to RGB format; and

wherein said image portions for further adaptive filtering comprise a respective color plane of said converted image.

19. An image processing method according to claim 15 wherein said adaptively filtering is performed one of row or column wise on said input signal and wherein further adaptively filtering is performed on the other of row or column wise on said input signal.

20. An image processing method according to claim 15 comprising noise filtering said first filtered image.

21. An image processing method according to claim 15 comprising applying color correction to one of said first filtered image or a noise filtered image.

22. An image processing method according to claim 17, wherein said first and second images are in RGB format and wherein said image portions comprise a respective color plane of said first and second images.

23. An image processing method according to claim 22,

wherein said adaptively filtering includes producing a set of filter coefficients from a combination of said input signal and an error signal being the difference between said desired signal and said output signal;

and

wherein the method further comprises constructing each color plane of said first filtered image from a combination of said filter coefficients and color plane information of said input signal.

24. An image processing method as claimed in claim 15 comprising:

prior to adaptive filtering, estimating a point spread function, PSF, for said second image; and

de-blurring said second image with said point spread function.

25. An image processing method according to claim 24 wherein said de-blurring is performed in response to said PSF being less than a pre-determined threshold.

26. An image processing method according to claim 15 comprising the steps of amplifying luminance characteristics of said under exposed image prior to said adaptively filtering.

27. An image processing apparatus arranged to process a first relatively underexposed and sharp image of a scene, and a second relatively well exposed and blurred image, nominally of the same scene, said apparatus comprising:

means for providing a portion of said first relatively underexposed image as an input signal to an adaptive filter;

means providing a corresponding portion of said second relatively well exposed image as a desired signal to said adaptive filter;

an adaptive filter arranged to produce an output signal from said input signal and said desired signal by multiplying the input signal by one or more coefficients that are calculated from values of the input signal and the desired signal; and

an image generator arranged to construct a first filtered image from said output signal, relatively less blurred than said second image.

28. One or more non-transitory processor-readable media having embedded therein processor-readable code for programming a processor to perform an image processing method to process a first relatively underexposed and sharp image of a scene, and a second relatively well exposed and blurred image nominally of the same scene, said method comprising:

providing a portion of said first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of said second relatively well exposed image as a desired signal to said adaptive filter;

producing an output signal from said input signal and said desired signal by multiplying the input signal by one or more coefficients that are calculated from values of the input signal and the desired signal;

constructing a first filtered image, from said output signal, that is relatively less blurred than said second image.

29. One or more processor-readable media according to claim 28 wherein said first and second images are in YCC format and has image portions that comprise a respective Y plane of said first and second images.

30. One or more processor-readable media according to claim 29 wherein said constructing said first filtered image comprises using said output signal as a Y plane of said first filtered image and using Cb and Cr planes of said input image as the Cb and Cr planes of said first filtered image.

31. One or more processor-readable media according to claim 28 wherein:

said first image is derived from a first image source;

said second image is derived from a second image source; and

said second image source for said second image is of a relatively higher resolution than said first image source for said first image.

32. One or more processor-readable media according to claim 31 wherein the method further comprises aligning and interpolating said first image source to match the alignment and resolution of said second image source.

33. One or more processor-readable media according to claim 28, wherein said first image is one of an image acquired soon before or after said second image.

34. One or more processor-readable media according to claim 31, wherein said second image source for said second image is of a relatively lower resolution than said first image source for said first image.

35. One or more processor-readable media according to claim 34, wherein the method further comprises aligning and interpolating said second image source to match the alignment and resolution of said first image source.

36. One or more processor-readable media according to claim 35, wherein the method further comprises, responsive to said first and second image sources being misaligned by more than a pre-determined threshold, providing said desired signal from a linear combination of said first and second image sources.

37. One or more processor-readable media according to claim 31, wherein the method further comprises, responsive to said first and second image sources being misaligned by more than a pre-determined threshold, providing said desired signal from a combination of phase values from one of said first and second image sources and amplitude values for the other of said first and second image sources.

38. One or more processor-readable media according to claim 28, wherein the method further comprises noise filtering said first filtered image.

39. One or more processor-readable media according to claim 28, wherein the method further comprises applying color correction to the first filtered image or a noise filtered image or both.

40. One or more processor-readable media according to claim 28, wherein the method further comprises:

acquiring a first partially exposed image from an image sensor;

acquiring a second further exposed image from said image sensor; and

subsequently resetting said image sensor,

wherein said first relatively underexposed and sharp image of a scene are obtained by subtracting said first partially exposed image from said second further exposed image.

41. One or more processor-readable media according to claim 40, wherein said second relatively well exposed and blurred image is obtained from said image sensor just prior to said resetting.

42. One or more non-transitory processor-readable media having embedded therein processor-readable code for programming a processor to perform an image processing method that comprises:

obtaining a first relatively underexposed and sharp image of a scene;

obtaining a second relatively well exposed and blurred image, nominally of the same scene

providing a portion of said first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of said second relatively well exposed image as a desired signal to said adaptive filter;

adaptively filtering said input signal to produce an output signal by multiplying the input signal by one or more coefficients that are calculated from values of the input signal and the desired signal; and

constructing a first filtered image, from said output signal, relatively less blurred than said second image.

43. One or more processor-readable media according to claim 42, wherein the method further comprises:

providing a portion of said first filtered image as said input signal to an adaptive filter;

providing a corresponding portion of said second image as a desired signal to said adaptive filter;

further adaptively filtering said input signal to produce a further output signal;

constructing a further filtered image from said further output signal relatively less blurred than said first filtered image.

44. One or more processor-readable media according to claim 43, wherein said first and second images are in RGB format to produce said first filtered image that has image portions that comprise a respective color plane of said first and second images;

wherein said providing a portion of said first filtered image comprises converting said first filtered image to YCC format;

wherein said method further comprises converting said second image to YCC format; and

wherein said image portions for further adaptive filtering comprise a respective Y plane of said converted image.

45. One or more processor-readable media according to claim 44, wherein said first and second images are in YCC format and, for producing said first filtered image;

wherein said image portions comprise a respective Y plane of said first and second images; and

wherein said providing a portion of said first filtered image comprises converting said first filtered image to RGB format;

wherein said method further comprises converting said second image to RGB format; and

wherein said image portions for further adaptive filtering comprise a respective color plane of said converted image.

46. One or more processor-readable media according to claim 42 wherein said adaptively filtering is performed one of row or column wise on said input signal and wherein further adaptively filtering is performed on the other of row or column wise on said input signal.

47. One or more processor-readable media according to claim 42, wherein the method further comprises noise filtering said first filtered image.

48. One or more processor-readable media according to claim 42, wherein the method further comprises applying color correction to one of said first filtered image or a noise filtered image.

49. One or more processor-readable media according to claim 44, wherein said first and second images are in RGB format and wherein said image portions comprise a respective color plane of said first and second images.

50. One or more processor-readable media according to claim 49, wherein said adaptively filtering includes producing a set of filter coefficients from a combination of said input signal and an error signal being the difference between said desired signal and said output signal;

and wherein the method further comprises constructing each color plane of said first filtered image from a combination of said filter coefficients and color plane information of said input signal.

51. One or more processor-readable media as claimed in claim 42, wherein the method further comprises:

prior to adaptive filtering, estimating a point spread function, PSF, for said second image; and

de-blurring said second image with said point spread function.

52. One or more processor-readable media according to claim 51 wherein said de-blurring is performed in response to said PSF being less than a pre- determined threshold.

53. One or more processor-readable media according to claim 42, wherein the method further comprises amplifying luminance characteristics of said under exposed image prior to said adaptively filtering.

54. A portable digital image acquisition device, comprising a lens, an image sensor for capturing a digital image, a processor, and one or more processor-readable media having processor-readable code embedded therein for programming the processor to process a first relatively underexposed and sharp image of a scene, and a second relatively well exposed and blurred image nominally of the same scene, and wherein the processor performs:

providing a portion of said first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of said second relatively well exposed image as a desired signal to said adaptive filter;

producing an output signal from said input signal and said desired signal by multiplying the input signal by one or more coefficients that are calculated from values of the input signal and the desired signal;

constructing a first filtered image, from said output signal, that is relatively less blurred than said second image; and

displaying, transmitting, projecting or storing the first filtered image or a further processed version, or combinations thereof.

55. A portable digital image acquisition device according to claim 54 wherein said first and second images are in YCC format and has image portions that comprise a respective Y plane of said first and second images.

56. A portable digital image acquisition device according to claim 55 wherein said constructing said first filtered image comprises using said output signal as a Y plane of said first filtered image and using Cb and Cr planes of said input image as the Cb and Cr planes of said first filtered image.

57. A portable digital image acquisition device according to claim 54 wherein;

said first image is derived from a first image source;

said second image is derived from a second image source; and

said second image source for said second image is of a relatively higher resolution than said first image source for said first image.

58. A portable digital image acquisition device according to claim 57 further comprising aligning and interpolating said first image source to match the alignment and resolution of said second image source.

59. A portable digital image acquisition device according to claim 54 wherein said first image is one of an image acquired soon before or after said second image.

60. A portable digital image acquisition device according to claim 57 wherein said second image source for said second image is of a relatively lower resolution than said first image source for said first image.

61. A portable digital image acquisition device according to claim 60, wherein the method further comprises aligning and interpolating said second image source to match the alignment and resolution of said first image source.

62. A portable digital image acquisition device according to claim 61, wherein the method further comprises, responsive to said first and second image sources being misaligned by more than a pre-determined threshold, providing said desired signal from a linear combination of said first and second image sources.

63. A portable digital image acquisition device according to claim 61, wherein the method further comprises, responsive to said first and second sources being misaligned by more than a pre-determined threshold, providing said desired signal from a combination of phase values from one of said first and second image sources and amplitude values for the other of said first and second image sources.

64. A portable digital image acquisition device according to claim 54, wherein the method further comprises noise filtering said first filtered image.

65. A portable digital image acquisition device according to claim 54, wherein the method further comprises applying color correction to said first filtered image or a noise filtered image or both.

66. A portable digital image acquisition device according to claim 54, wherein the method further comprises:

acquiring a first partially exposed image from an image sensor;

acquiring a second further exposed image from said image sensor; and

subsequently resetting said image sensor,

wherein said first relatively underexposed and sharp image of a scene are obtained by subtracting said first partially exposed image from said second further exposed image.

67. A portable digital image acquisition device according to claim 66, wherein said second relatively well exposed and blurred image is obtained from said image sensor just prior to said resetting.

68. A portable digital image acquisition device, comprising a lens, an image sensor for capturing a digital image, a processor, and one or more processor-readable media having processor-readable code embedded therein for programming the processor to perform:

obtaining a first relatively underexposed and sharp image of a scene;

obtaining a second relatively well exposed and blurred image, nominally of the same scene

providing a portion of said first relatively underexposed image as an input signal to an adaptive filter;

providing a corresponding portion of said second relatively well exposed image as a desired signal to said adaptive filter;

adaptively filtering said input signal to produce an output signal by multiplying the input signal by one or more coefficients that are calculated from values of the input signal and the desired signal; and

constructing a first filtered image,. from said output signal, relatively less blurred than said second image.

69. A portable digital image acquisition device according to claim 68, wherein the method further comprises:

providing a portion of said first filtered image as said input signal to an adaptive filter;

providing a corresponding portion of said second image as a desired signal to said adaptive filter;

further adaptively filtering said input signal to produce a further output signal;

constructing a further filtered image from said further output signal relatively less blurred than said first filtered image.

70. A portable digital image acquisition device according to claim 69, wherein said first and second images are in RGB format to produce said first filtered image and has image portions that comprise a respective color plane of said first and second images;

wherein said providing a portion of said first filtered image comprises converting said first filtered image to YCC format;

wherein said method further comprises converting said second image to YCC format; and

wherein said image portions for further adaptive filtering comprise a respective Y plane of said converted image.

71. A portable digital image acquisition device according to claim 70, wherein said first and second images are in YCC format and, for producing said first filtered image;

and

wherein said image portions comprise a respective Y plane of said first and second images;

wherein said providing a portion of said first filtered image comprises converting said first filtered image to RGB format;

wherein said method further comprises converting said second image to RGB format; and

wherein said image portions for further adaptive filtering comprise a respective color plane of said converted image.

72. A portable digital image acquisition device according to claim 68, wherein said adaptively filtering is performed one of row or column wise on said input signal and wherein further adaptively filtering is performed on the other of row or column wise on said input signal.

73. A portable digital image acquisition device according to claim 68, wherein the method further comprises noise filtering said first filtered image.

74. A portable digital image acquisition device according to claim 68, wherein the method further comprises applying color correction to one of said first filtered image or a noise filtered image.

75. A portable digital image acquisition device according to claim 70, wherein said first and second images are in RGB format and wherein said image portions comprise a respective color plane of said first and second images.

76. A portable digital image acquisition device according to claim 75, wherein said adaptively filtering includes producing a set of filter coefficients from a combination of said input signal and a error signal being the difference between said desired signal and said output signal;

and wherein the method further comprises constructing each color plane of said first filtered image from a combination of said filter coefficients and said input signal color plane information.

77. A portable digital image acquisition device as claimed in claim 68, wherein the method further comprises:

prior to adaptive filtering, estimating a point spread function, PSF, for said second image; and

de-blurring said second image with said point spread function.

78. A portable digital image acquisition device according to claim 77, wherein said de-blurring is performed in response to said PSF being less than a pre- determined threshold.

79. A portable digital image acquisition device according to claim 68, wherein the method further comprises amplifying luminance characteristics of said under exposed image prior to said adaptively filtering.