IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD AND IMAGE PROCESSING PROGRAM

• US 20110274370A1
• Filed: 05/03/2011
• Published: 11/10/2011
• Est. Priority Date: 05/10/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: NAGUMO, Takefumi, LUO, Jun, KONDO, Yuhi
• Granted Patent: US 8,620,109 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

Disclosed herein is an image processing apparatus including an up-sampling section configured to carry out up-sampling processing in order to generate an up-sampled image, a motion-compensated image generation section configured to generate a motion-compensated image as a result of correction processing to adjust a referenced image having the second resolution to a photographing-object position on the up-sampled image by making use of information on a difference between the up-sampled image and the referenced image, a blending processing section configured to generate a blended image as a result of blending processing to blend the up-sampled image with the referenced image, and an output-image generation section configured to receive and process the blended image as well as the up-sampled image in order to generate an output blended image obtained by blending a super-resolution processing-result image with a noise-reduction processing-result image.

Claims

1. An image processing apparatus comprising:

an up-sampling section configured to carry out up-sampling processing to increase the number of pixels composing an input image having a first resolution to the number of pixels composing an image having a second resolution higher than said first resolution in order to generate an up-sampled image;

a motion-compensated image generation section configured to generate a motion-compensated image as a result of correction processing to adjust a referenced image having said second resolution to a photographing-object position on said up-sampled image by making use of information on a difference between said up-sampled image and said referenced image;

a blending processing section configured to generate an intermediate blended image as a result of blending processing to blend said up-sampled image with said referenced image by comparing said up-sampled image with said motion-compensated image for every image area unit and by setting a blending coefficient of said referenced image at a such a value that, the higher the degree of pixel-value matching for an image area unit, the higher the value at which said blending coefficient is set for said image area unit; and

an output-image generation section configured to receive and process said intermediate blended image as well as said up-sampled image in order to generate an output blended image obtained by blending a super-resolution processing-result image obtained as a result of super-resolution processing carried out as processing to increase the resolution of said input image with a noise-reduction processing-result image obtained as a result of noise reduction processing carried out to reduce noises of said input image.

3. The image processing apparatus according to claim 1 wherein said output-image generation section comprises:

an integration filter configured to carry out down-sampling processing to reduce the number of pixels composing said intermediate blended image or said motion-compensated image to a pixel count corresponding to said first resolution and carry out up-sampling processing to increase the number of pixels composing said blended image or said motion-compensated image to a pixel count corresponding to said second resolution; and

a calculation processing section configured to carry out calculation processing on received images including said blended image or said motion-compensated image, an image output by said integration filter and said up-sampled image output by said up-sampling section in order to generate said output blended image obtained as a result of blending said super-resolution processing-result image and said noise-reduction processing-result image.

5. The image processing apparatus according to claim 3 wherein said integration filter is a poly-phase filter for convolving a value varying from phase to phase.

7. The image processing apparatus according to claim 1 wherein said output-image generation section is configured to set a blending ratio of said super-resolution processing-result image and said noise-reduction processing-result image in accordance with a parameter supplied to said calculation processing section.

9. The image processing apparatus according to claim 7 wherein said output-image generation section carries out processing to adjust said blending ratio of said super-resolution processing-result image and said noise-reduction processing-result image in accordance with an ISO sensitivity computed for every image area unit of said input image.

10. The image processing apparatus according to claim 7 wherein said output-image generation section carries out processing to adjust said blending ratio of said super-resolution processing-result image and said noise-reduction processing-result image in accordance with an image magnification factor adopted by said up-sampling section for said input image.

11. The image processing apparatus according to claim 7 wherein said output-image generation section carries out processing to adjust said blending ratio of said super-resolution processing-result image and said noise-reduction processing-result image in accordance with a variance of pixel values for each image area unit of said input image.

2. An image processing apparatus comprising:

an up-sampling section configured to carry out up-sampling processing to increase the number of pixels composing an input image having a first resolution to the number of pixels composing an image having a second resolution higher than said first resolution in order to generate an up-sampled image;

a motion-compensated image generation section configured to generate a motion-compensated image as a result of correction processing to adjust a referenced image having said second resolution to a photographing-object position on said up-sampled image by making use of information on a difference between said up-sampled image and said referenced image; and

an output-image generation section configured to receive and process said motion-compensated image as well as said up-sampled image in order to generate an output blended image obtained by blending a super-resolution processing-result image obtained as a result of super-resolution processing carried out as processing to increase the resolution of said input image with a noise-reduction processing-result image obtained as a result of noise reduction processing carried out to reduce noises of said input image.

4. The image processing apparatus according to claim 2 wherein said output-image generation section comprises:

an integration filter configured to carry out down-sampling processing to reduce the number of pixels composing said intermediate blended image or said motion-compensated image to a pixel count corresponding to said first resolution and carry out up-sampling processing to increase the number of pixels composing said blended image or said motion-compensated image to a pixel count corresponding to said second resolution; and

a calculation processing section configured to carry out calculation processing on received images including said blended image or said motion-compensated image, an image output by said integration filter and said up-sampled image output by said up-sampling section in order to generate said output blended image obtained as a result of blending said super-resolution processing-result image and said noise-reduction processing-result image.

6. The image processing apparatus according to claim 4 wherein said integration filter is a poly-phase filter for convolving a value varying from phase to phase.

8. The image processing apparatus according to claim 2 wherein said output-image generation section is configured to set a blending ratio of said super-resolution processing-result image and said noise-reduction processing-result image in accordance with a parameter supplied to said calculation processing section.

12. The image processing apparatus according to claim 8 wherein said output-image generation section carries out processing to adjust said blending ratio of said super-resolution processing-result image and said noise-reduction processing-result image in accordance with an ISO sensitivity computed for every image area unit of said input image.

13. The image processing apparatus according to claim 8 wherein said output-image generation section carries out processing to adjust said blending ratio of said super-resolution processing-result image and said noise-reduction processing-result image in accordance with an image magnification factor adopted by said up-sampling section for said input image.

14. The image processing apparatus according to claim 8 wherein said output-image generation section carries out processing to adjust said blending ratio of said super-resolution processing-result image and said noise-reduction processing-result image in accordance with a variance of pixel values for each image area unit of said input image.

15. An image processing method to be adopted by an image processing apparatus employing an up-sampling section, a motion-compensated image generation section, a blending processing section and an output-image generation section, said image processing method comprising:

an up-sampling processing step of driving said up-sampling section to carry out up-sampling processing to increase the number of pixels composing an input image having a first resolution to the number of pixels composing an image having a second resolution higher than said first resolution in order to generate an up-sampled image;

a motion-compensated image generation step of driving said motion-compensated image generation section to generate a motion-compensated image as a result of correction processing to adjust a referenced image having said second resolution to a photographing-object position on said up-sampled image by making use of information on a difference between said up-sampled image and said referenced image;

a blending processing step of driving said blending processing section to generate an intermediate blended image as a result of blending processing to blend said up-sampled image with said referenced image by comparing said up-sampled image with said motion-compensated image for every image area unit and by setting a blending coefficient of said referenced image at a such a value that, the higher the degree of pixel-value matching for an image area unit, the higher the value at which said blending coefficient is set for said image area unit; and

an output-image generation step of driving said output-image generation section to receive and process said intermediate blended image as well as said up-sampled image in order to generate an output blended image obtained by blending a super-resolution processing-result image obtained as a result of super-resolution processing carried out as processing to increase the resolution of said input image with a noise-reduction processing-result image obtained as a result of noise reduction processing carried out to reduce noises of said input image.

16. An image processing method to be adopted by an image processing apparatus employing an up-sampling section, a motion-compensated image generation section and an output-image generation section, said image processing method comprising:

an up-sampling processing step of driving said up-sampling section to carry out up-sampling processing to increase the number of pixels composing an input image having a first resolution to the number of pixels composing an image having a second resolution higher than said first resolution in order to generate an up-sampled image;

a motion-compensated image generation step of driving said motion-compensated image generation section to generate a motion-compensated image as a result of correction processing to adjust a referenced image having said second resolution to a photographing-object position on said up-sampled image by making use of information on a difference between said up-sampled image and said referenced image; and

an output-image generation step of driving said output-image generation section to receive and process said motion-compensated image as well as said up-sampled image in order to generate an output blended image obtained by blending a super-resolution processing-result image obtained as a result of super-resolution processing carried out as processing to increase the resolution of said input image with a noise-reduction processing-result image obtained as a result of noise reduction processing carried out to reduce noises of said input image.

17. An image processing program to be executed by an image processing apparatus, which employs an up-sampling section, a motion-compensated image generation section, a blending processing section and an output-image generation section, in order to carry out:

an up-sampling processing step of driving said up-sampling section to carry out up-sampling processing to increase the number of pixels composing an input image having a first resolution to the number of pixels composing an image having a second resolution higher than said first resolution in order to generate an up-sampled image;

a motion-compensated image generation step of driving said motion-compensated image generation section to generate a motion-compensated image as a result of correction processing to adjust a referenced image having said second resolution to a photographing-object position on said up-sampled image by making use of information on a difference between said up-sampled image and said referenced image;

a blending processing step of driving said blending processing section to generate an intermediate blended image as a result of blending processing to blend said up-sampled image with said referenced image by comparing said up-sampled image with said motion-compensated image for every image area unit and by setting a blending coefficient of said referenced image at a such a value that, the higher the degree of pixel-value matching for an image area unit, the higher the value at which said blending coefficient is set for said image area unit; and

an output-image generation step of driving said output-image generation section to receive and process said intermediate blended image as well as said up-sampled image in order to generate an output blended image obtained by blending a super-resolution processing-result image obtained as a result of super-resolution processing carried out as processing to increase the resolution of said input image with a noise-reduction processing-result image obtained as a result of noise reduction processing carried out to reduce noises of said input image.

18. An image processing program to be executed by an image processing apparatus, which employs an up-sampling section, a motion-compensated image generation section and an output-image generation section, in order to carry out:

an up-sampling processing step of driving said up-sampling section to carry out up-sampling processing to increase the number of pixels composing an input image having a first resolution to the number of pixels composing an image having a second resolution higher than said first resolution in order to generate an up-sampled image;

a motion-compensated image generation step of driving said motion-compensated image generation section to generate a motion-compensated image as a result of correction processing to adjust a referenced image having said second resolution to a photographing-object position on said up-sampled image by making use of information on a difference between said up-sampled image and said referenced image; and

an output-image generation step of driving said output-image generation section to receive and process said motion-compensated image as well as said up-sampled image in order to generate an output blended image obtained by blending a super-resolution processing-result image obtained as a result of super-resolution processing carried out as processing to increase the resolution of said input image with a noise-reduction processing-result image obtained as a result of noise reduction processing carried out to reduce noises of said input image.