Tone mapping for low-light video frame enhancement

• US 8,890,983 B2
• Filed: 08/28/2013
• Issued: 11/18/2014
• Est. Priority Date: 03/05/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Vranceanu, Ruxandra
• Primary Examiner(s): MOE, AUNG SOE
• Application Number:
  US14/012,806
• Publication Number:
  US 20130342738A1
• Time in Patent Office: 447 Days
• Field of Search: 348/222.1, 348/239, 348/241, 348/242, 348/252, 348/263, 348/254, 348/255, 348/256, 348/208.4, 348/208.13, 382/254, 382/261, 382/275
• US Class Current: 348/253
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20008 Globally adaptive
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/86 for controlling the colour saturation of colour signals, e.g. automatic chroma control circuits
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
• CPC Class Codes: H04N 9/68 for controlling the amplitude of colour signals, e.g. automatic chroma control circuits H04N9/71, H04N9/73 take precedence; camera processing pipelines for controlling the colour saturation of colour signals H04N23/86

Abstract

A technique is provided for generating sharp, well-exposed, color images from low-light images. A series of under-exposed images is acquired. A mean image is computed and a sum image is generated each based on the series of under-exposed images. Chrominance variables of pixels of the mean image are mapped to chrominance variables of pixels of the sum image. Chrominance values of pixels within the series of under-exposed images are replaced with chrominance values of the sum image. A set of sharp, well-exposed, color images is generated based on the series of under-exposed images with replaced chrominance values.

Claims

1. An image processing method comprising;

obtaining a first underexposed and sharp image of a scene;

obtaining a second image relatively well exposed and blurred compared to said first image, nominally of the same scene, wherein said second image is acquired at a different exposure time than said first image;

scanning across said first and second images to;

provide a portion of said first image as an input signal to an adaptive filter;

provide a corresponding portion of said second image as a desired signal to said adaptive filter; and

calculate filter coefficients for said portion of said images at a given scanning location based on a combination of said input signal, said desired signal and existing filter coefficients for said scanning location; and

adaptively filter said input signal based on said filter coefficients to produce an output signal and constructing a first filtered image from said output signal, less blurred than said second image.

2. An image processing method according to claim 1, wherein said first and second images being in RGB format and wherein said image portions comprise a respective color plane of said first and second images and further comprising constructing each color plane of said first filtered image from a combination of said filter coefficients and said input signal color plane information.

3. An image processing method according to claim 1, wherein said first and second images being in YCC format, wherein said image portions comprise a respective Y plane of said first and second images, and wherein said constructing said first filtered image comprises using said output signal as a Y plane of said first filtered image and using Cb and Cr planes of said input image as the Cb and Cr planes of said first filtered image.

4. An image processing method according to claim 1, further comprising:

providing a portion of said first filtered image as said input signal to an adaptive filter;

providing a corresponding portion of said second image as a desired signal to said adaptive filter;

further adaptively filtering said input signal to produce a further output signal; and

constructing a further filtered image from said further output signal less blurred than said first filtered image.

5. An image processing method according to claim 4, wherein:

said first and second images being in RGB format and, for producing said first filtered image, said image portions comprise a respective color plane of said first and second images;

said providing a portion of said first filtered image comprises converting said first filtered image to YCC format;

said method further comprises converting said second image to YCC format; and

wherein said image portions for further adaptive filtering comprise a respective Y plane of said converted images.

6. An image processing method according to claim 4 wherein:

said first and second images being in YCC format and, for producing said first filtered image, said image portions comprise a respective Y plane of said first and second images;

said providing a portion of said first filtered image comprises converting said first filtered image to RGB format;

said method further comprises converting said second image to RGB format; and

wherein said image portions for further adaptive filtering comprise a respective color plane of said converted images.

7. An image processing method according to claim 4 wherein said adaptively filtering being performed row-wise or column-wise on said input signal and wherein further adaptive filtering is performed on the other of row-wise or column wise on said input signal.

8. An image processing method according to claim 1, wherein said image source for said second image comprises relatively higher resolution than said image source for said first image.

9. An image processing method as claimed in claim 8, further comprising:

prior to adaptive filtering, estimating a point spread function, PSF, for said second image; and

responsive to said PSF being less than a pre-determined threshold, de-blurring said second image with said point spread function.

10. An image processing method according to claim 8 further comprising amplifying the luminance characteristics of said under exposed image prior to said adaptively filtering.

11. An image processing method according to claim 8 wherein said first image comprising one of an image acquired soon before or after said second image.

12. An image processing method according to claim 1 wherein said image source for said second image comprises relatively lower resolution than said image source for said first image.

13. An image processing method according to claim 12 further comprising:

aligning and interpolating said second source to match the alignment and resolution of said first source and

responsive to said first and second sources being misaligned by more than a predetermined threshold, providing said desired signal from a linear combination of said first and second image sources.

14. An image processing method according to claim 12 further comprising:

aligning and interpolating said second source to match the alignment and resolution of said first source and

responsive to said first and second sources being misaligned by more than a pre-determined threshold, providing said desired signal from a combination of phase values from one of said first and second image sources and amplitude values for the other of said first and second image sources.

15. An image processing method according to claim 1, wherein said first and second images being derived from respective image sources.

16. An image processing method according to claim 1, wherein said second image being acquired at a different resolution than said first image.

17. A digital video acquisition and processing device, comprising:

a lens and an image sensor for capturing a stream of multiple digital video or still images;

a processor; and

a memory having processor-readable code embedded therein for programming the processor to perform a method of generating sharp, well-exposed, color images from under-exposed, low-light images, wherein the method comprises;

obtaining a first underexposed and sharp image of a scene;

obtaining a second image relatively well exposed and blurred compared to said first image, nominally of the same scene, wherein said second image is acquired at a different exposure time than said first image;

scanning across said first and second images to;

provide a portion of said first image as an input signal to an adaptive filter;

provide a corresponding portion of said second image as a desired signal to said adaptive filter; and

calculate filter coefficients for said portion of said images at a given scanning location based on a combination of said input signal, said desired signal and existing filter coefficients for said scanning location; and

adaptively filter said input signal based on said filter coefficients to produce an output signal and constructing a first filtered image from said output signal, less blurred than said second image.

18. An image processing device according to claim 17, wherein said first and second images being in RGB format and wherein said image portions comprise a respective color plane of said first and second images and further comprising constructing each color plane of said first filtered image from a combination of said filter coefficients and said input signal color plane information.

19. An image processing device according to claim 17, wherein said first and second images being in YCC format, wherein said image portions comprise a respective Y plane of said first and second images, and wherein said constructing said first filtered image comprises using said output signal as a Y plane of said first filtered image and using Cb and Cr planes of said input image as the Cb and Cr planes of said first filtered image.

20. An image processing device according to claim 17, wherein the method further comprises:

providing a portion of said first filtered image as said input signal to an adaptive filter;

providing a corresponding portion of said second image as a desired signal to said adaptive filter;

further adaptively filtering said input signal to produce a further output signal; and

constructing a further filtered image from said further output signal less blurred than said first filtered image.

21. An image processing device according to claim 20, wherein:

said first and second images being in RGB format and, for producing said first filtered image, said image portions comprise a respective color plane of said first and second images;

said providing a portion of said first filtered image comprises converting said first filtered image to YCC format;

said method further comprises converting said second image to YCC format; and

wherein said image portions for further adaptive filtering comprise a respective Y plane of said converted images.

22. An image processing device according to claim 20 wherein:

said first and second images being in YCC format and, for producing said first filtered image, said image portions comprise a respective Y plane of said first and second images;

said providing a portion of said first filtered image comprises converting said first filtered image to RGB format;

said method further comprises converting said second image to RGB format; and

wherein said image portions for further adaptive filtering comprise a respective color plane of said converted images.

23. An image processing device according to claim 20 wherein said adaptively filtering being performed row-wise or column-wise on said input signal and wherein further adaptive filtering is performed on the other of row-wise or column wise on said input signal.

24. An image processing device according to claim 17, wherein said image source for said second image comprises relatively higher resolution than said image source for said first image.

25. An image processing device as claimed in claim 24, wherein the method further comprises:

prior to adaptive filtering, estimating a point spread function, PSF, for said second image; and

responsive to said PSF being less than a pre-determined threshold, de-blurring said second image with said point spread function.

26. An image processing device according to claim 24 wherein the method further comprises amplifying the luminance characteristics of said under exposed image prior to said adaptively filtering.

27. An image processing device according to claim 24 wherein said first image comprising one of an image acquired soon before or after said second image.

28. An image processing device according to claim 17 wherein said image source for said second image comprises relatively lower resolution than said image source for said first image.

29. An image processing device according to claim 28 wherein the method further comprises:

aligning and interpolating said second source to match the alignment and resolution of said first source and

responsive to said first and second sources being misaligned by more than a predetermined threshold, providing said desired signal from a linear combination of said first and second image sources.

30. An image processing device according to claim 28 wherein the method further comprises:

aligning and interpolating said second source to match the alignment and resolution of said first source and

responsive to said first and second sources being misaligned by more than a pre-determined threshold, providing said desired signal from a combination of phase values from one of said first and second image sources and amplitude values for the other of said first and second image sources.

31. An image processing device according to claim 17, wherein said first and second images being derived from respective image sources.

32. An image processing device according to claim 17, wherein said second image being acquired at a different resolution than said first image.

33. One or more non-transitory processor-readable media having code embedded therein for programming one or more processors to perform a method of generating sharp, well-exposed, color images from under-exposed, low-light images, wherein the method comprises:

obtaining a first underexposed and sharp image of a scene;

obtaining a second image relatively well exposed and blurred compared to said first image, nominally of the same scene, wherein said second image is acquired at a different exposure time than said first image;

scanning across said first and second images to;

provide a portion of said first image as an input signal to an adaptive filter;

provide a corresponding portion of said second image as a desired signal to said adaptive filter; and

calculate filter coefficients for said portion of said images at a given scanning location based on a combination of said input signal, said desired signal and existing filter coefficients for said scanning location; and

adaptively filter said input signal based on said filter coefficients to produce an output signal and constructing a first filtered image from said output signal, less blurred than said second image.

34. One of more non-transitory processor-readable media according to claim 33, wherein said first and second images being in RGB format and wherein said image portions comprise a respective color plane of said first and second images and further comprising constructing each color plane of said first filtered image from a combination of said filter coefficients and said input signal color plane information.

35. One of more non-transitory processor-readable media according to claim 33, wherein said first and second images being in YCC format, wherein said image portions comprise a respective Y plane of said first and second images, and wherein said constructing said first filtered image comprises using said output signal as a Y plane of said first filtered image and using Cb and Cr planes of said input image as the Cb and Cr planes of said first filtered image.

36. One of more non-transitory processor-readable media according to claim 33, wherein the method further comprises:

providing a portion of said first filtered image as said input signal to an adaptive filter;

providing a corresponding portion of said second image as a desired signal to said adaptive filter;

further adaptively filtering said input signal to produce a further output signal; and

constructing a further filtered image from said further output signal less blurred than said first filtered image.

37. One of more non-transitory processor-readable media according to claim 36, wherein:

said first and second images being in RGB format and, for producing said first filtered image, said image portions comprise a respective color plane of said first and second images;

said providing a portion of said first filtered image comprises converting said first filtered image to YCC format;

said method further comprises converting said second image to YCC format; and

wherein said image portions for further adaptive filtering comprise a respective Y plane of said converted images.

38. One of more non-transitory processor-readable media according to claim 36 wherein:

said first and second images being in YCC format and, for producing said first filtered image, said image portions comprise a respective Y plane of said first and second images;

said providing a portion of said first filtered image comprises converting said first filtered image to RGB format;

said method further comprises converting said second image to RGB format; and

wherein said image portions for further adaptive filtering comprise a respective color plane of said converted images.

39. One of more non-transitory processor-readable media according to claim 36 wherein said adaptively filtering being performed row-wise or column-wise on said input signal and wherein further adaptive filtering is performed on the other of row-wise or column wise on said input signal.

40. One of more non-transitory processor-readable media according to claim 33, wherein said image source for said second image comprises relatively higher resolution than said image source for said first image.

41. One of more non-transitory processor-readable media as claimed in claim 40, wherein the method further comprises:

prior to adaptive filtering, estimating a point spread function, PSF, for said second image; and

responsive to said PSF being less than a pre-determined threshold, de-blurring said second image with said point spread function.

42. One of more non-transitory processor-readable media according to claim 40 wherein the method further comprises amplifying the luminance characteristics of said under exposed image prior to said adaptively filtering.

43. One of more non-transitory processor-readable media according to claim 40 wherein said first image comprising one of an image acquired soon before or after said second image.

44. One of more non-transitory processor-readable media according to claim 33 wherein said image source for said second image comprises relatively lower resolution than said image source for said first image.

45. One of more non-transitory processor-readable media according to claim 44 wherein the method further comprises:

aligning and interpolating said second source to match the alignment and resolution of said first source and

responsive to said first and second sources being misaligned by more than a predetermined threshold, providing said desired signal from a linear combination of said first and second image sources.

46. One of more non-transitory processor-readable media according to claim 44 wherein the method further comprises:

aligning and interpolating said second source to match the alignment and resolution of said first source and

responsive to said first and second sources being misaligned by more than a pre-determined threshold, providing said desired signal from a combination of phase values from one of said first and second image sources and amplitude values for the other of said first and second image sources.

47. One of more non-transitory processor-readable media according to claim 33, wherein said first and second images being derived from respective image sources.

48. One of more non-transitory processor-readable media according to claim 33, wherein said second image being acquired at a different resolution than said first image.