Image enhancement apparatus and method

• US 8,768,069 B2
• Filed: 02/17/2012
• Issued: 07/01/2014
• Est. Priority Date: 02/24/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Springer, Paul, Richter, Martin, Nishi, Toru, Brueggemann, Matthias
• Primary Examiner(s): Rahmjoo, Mike
• Application Number:
  US13/399,405
• Publication Number:
  US 20120219229A1
• Time in Patent Office: 865 Days
• Field of Search: 382/199, 382/236, 382/233, 382/232, 382/243, 382/248, 382/250, 382/190, 382/209, 382/239, 382/216, 382/217, 382/224
• US Class Current: 382/199
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

The present invention relates to an image enhancement apparatus for enhancing an input image of a sequence of input images. To provide the ability to increase the resolution of an input image and/or to temporally reduce artifacts and/or noise in an input image, the apparatus comprises a motion compensation unit, a weighted selection unit, a feature analysis unit, an image model unit configured to generate a modelled image by applying an image model on said input image and/or said weighted selection image, a spatio-temporal detail signal generation unit configured to generate a detail signal from said input image and said weighted selection image, and a combination unit configured to generate said enhanced output image from said input image, said detail signal and said modelled image.

Claims

1. An image enhancement apparatus for enhancing an input image of a sequence of input images and obtaining an enhanced output image, said apparatus comprising:

a motion compensation unit configured to generate at least one preceding motion compensated image by compensating motion in a preceding output image,

a weighted selection unit configured to generate a weighted selection image from said input image and said preceding motion compensated image, both being weighted by a respective selection weighting factor,

a feature analysis unit configured to analyse at least the input image and to generate feature information by segmenting the input image into two or more input image regions having different features,

an image model unit configured to generate a modelled image by applying an image model, in particular a Huber Markov image model, on at least one of said input image said weighted selection image,

a spatio-temporal detail signal generation unit configured to generate a detail signal from said input image and said weighted selection image, and

a combination unit configured to generate said enhanced output image from said input image, said detail signal and said modelled image,

wherein said weighted selection unit is configured to generate said weighted selection image by adding said input image after weighting it with a first selection weighting factor and said preceding motion compensated image after weighting it with a second selection weighting factor.

2. The image processing apparatus as claimed in claim 1,

further comprising a frame buffer configured to buffer one or more preceding output images for use by said motion compensation unit.

3. The image processing apparatus as claimed in claim 1,

wherein said weighted selection unit is configured to determine said respective selection weighting factors based on at least one of said feature information and the local summed absolute difference between said input image and said preceding motion compensated image.

4. The image processing apparatus as claimed in claim 1,

wherein said weighted selection unit is configured to generate said first and second selection weighting factors such that the sum of said first and second selection weighting factors is 1.

5. The image processing apparatus as claimed in claim 1,

wherein said feature analysis unit is configured to generate one or more feature maps as feature information, said one or more feature maps including one or more of a motion vector length map, a flat map, a noise map and an edge gain map.

6. The image processing apparatus as claimed in claim 5,

wherein said feature analysis unit is configured to generate said motion vector length map from motion vector information describing the motion between successive input images.

7. The image processing apparatus as claimed in claim 5,

wherein said feature analysis unit is configured to detect flat regions in said input image for generating said flat map.

8. The image processing apparatus as claimed in claim 5,

wherein said feature analysis unit is configured to determine a combined local standard deviation from said input image and said preceding motion compensated image for generating said noise map.

9. The image processing apparatus as claimed in claim 5,

wherein said feature analysis unit is configured to detect edges in said weighted selection image for generating said edge gain map.

10. The image processing apparatus as claimed in claim 1,

wherein said image model unit comprises

a low pass filter bank unit configured to low-pass filter said weighted selection image,

a high frequency signal generation unit configured to generate a Laplacian from said input image,

a control unit configured to detect edge regions and directions in said input image,

a selection unit configured to receive the output of said low pass filter bank unit, said high frequency signal generation unit and said control unit and to select output data of at least one of said low pass filter bank unit and said high frequency signal generation unit based on detected edge regions and directions, and

an image model combination unit for combining, in particular adding, the selected output data of said low pass filter bank unit and/or said high frequency signal generation unit.

11. The image processing apparatus as claimed in claim 10,

wherein said motion compensation unit is configured to generate said at least one preceding motion compensated image by compensating motion in the directly preceding output image converted from the input image directly preceding the present input image.

12. The image processing apparatus as claimed in claim 1,

wherein said spatio-temporal detail signal generation unit comprises a low-pass filter configured to low-pass filter said input image and to subtract said low-pass filtered input image from said weighted selection image.

13. The image processing apparatus as claimed in claim 12,

wherein said spatio-temporal detail signal generation unit further comprises

a local variance estimation unit configured to estimate the local variance of a filter kernel of said low-pass filter from said input image and

an adapted filter selection unit configured to select an optimal filter kernel of said low-pass filter based on said estimated local variance and a set of available filters.

14. The image processing apparatus as claimed in claim 1, further comprising a multiplication unit configured to multiply said detail signal with a predetermined gain factor,

wherein said combination unit is configured to generate said enhanced output image from said input image, said detail signal multiplied with said gain factor and said modelled image.

15. The image processing apparatus as claimed in claims 14,

further comprising a gain factor computation unit configured to generate said gain factor from said feature information.

16. The image processing apparatus as claimed in claim 15,

wherein said gain factor computation unit is configured to generate said gain factor by multiplying said flat map, said noise map, 1 minus the edge gain map and a global predetermined gain value.

17. The image processing apparatus as claimed in claim 1,

further comprising a summation unit configured to add said detail signal to said input image to generate a high detail image,

wherein said combination unit is configured to generate said enhanced output image by combining said high detail image and said modelled image.

18. The image processing apparatus as claimed in claim 1,

wherein said motion compensation unit is configured to use at least one of motion vectors and motion compensation information previously acquired and/or stored in a motion compensation information storage.

19. The image processing apparatus as claimed in claim 1,

wherein said motion compensation unit is configured to generate at least one of motion vectors and motion compensation information from said output images.

20. A display device comprising

an image enhancement apparatus as claimed in claim 1 for enhancing an input image of a sequence of input images and obtaining an enhanced output image and

a display for displaying said output image.

21. An image enhancement method for enhancing an input image of a sequence of input images and obtaining an enhanced output image, said method comprising:

generating, using a processor, at least one preceding motion compensated image by compensating motion in a preceding output image,

generating a weighted selection image from said input image and said preceding motion compensated image, both being weighted by a respective selection weighting factor,

analysing at least the input image and generating feature information by segmenting the input image into two or more input image regions having different features,

generating a modelled image by applying an image model, in particular a Huber Markov image model, on at least one of said input image and said weighted selection image,

generating a detail signal from said input image and said weighted selection image, and

generating said enhanced output image from said input image, said detail signal and said modelled image,

wherein said weighted selection image is generated by adding said input image after weighting it with a first selection weighting factor and said preceding motion compensated image after weighting it with a second selection weighting factor.

22. Computer readable non-transitory medium having instructions stored thereon which, when carried out on a computer, cause the computer to perform the steps of the method as claimed in claim 21.

23. An image enhancement apparatus for enhancing an input image of a sequence of input images and obtaining an enhanced output image, said apparatus comprising:

a motion compensation means for generating at least one preceding motion compensated image by compensating motion in a preceding output image,

a weighted selection means for generating a weighted selection image from said input image and said preceding motion compensated image, both being weighted by a respective selection weighting factor,

a feature analysis means configured for analysing at least the input image and generating feature information by segmenting the input image into two or more input image regions having different features,

an image model means for generating a modelled image by applying an image model, in particular a Huber Markov image model, on at least one of said input image and said weighted selection image,

a spatio-temporal detail signal generation means for generating a detail signal from said input image and said weighted selection image, and

a combination means for generating said enhanced output image from said input image, said detail signal and said modelled image,

wherein said weighted selection means is configured for generating said weighted selection image by adding said input image after weighting it with a first selection weighting factor and said preceding motion compensated image after weighting it with a second selection weighting factor.