IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM

• US 20100157073A1
• Filed: 12/22/2009
• Published: 06/24/2010
• Est. Priority Date: 12/22/2008
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Luo, Jun, Kondo, Yuhi, Nagumo, Takefumi
• Granted Patent: US 8,169,490 B2
• Time in Patent Office: Days
• US Class Current: 348/208.400
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

An image processing apparatus includes a motion prediction processor configured to detect a motion vector that indicates inter-image motion between a current image and a reference image; a motion compensation processor configured to perform a motion compensation process for the reference image by using the motion vector and generate a motion compensated image; an addition processor configured to generate a noise reduced image in which noise of the current image is reduced by adding the current image and the motion compensated image; an addition determination unit configured to compute an addition weight in units of pixels of the motion compensated image; a down-sampling processor configured to perform a process for reducing the current image and the motion compensated image; and an up-sampling processor configured to perform a process for expanding an addition coefficient map that is an output of the addition determination unit.

Claims

1. An image processing apparatus comprising:

a motion prediction processor configured to detect a motion vector that indicates inter-image motion between a current image and a reference image;

a motion compensation processor configured to perform a motion compensation process for the reference image by using the motion vector and generate a motion compensated image;

an addition processor configured to generate a noise reduced image in which noise of the current image is reduced by performing a process for adding the current image and the motion compensated image;

an addition determination unit configured to compute an addition weight in units of pixels of the motion compensated image that is used for a process for adding the current image and the motion compensated image, the addition process being performed by the addition processor;

a down-sampling processor configured to perform a process for reducing the current image and the motion compensated image that are input to the addition determination unit; and

an up-sampling processor configured to perform a process for expanding an addition coefficient map that is an output of the addition determination unit.

2. The image processing apparatus according to claim 1, wherein, by using reduced images of the current image and the motion compensated image generated by the down-sampling processor, the addition determination unit computes an addition weight corresponding to constituent pixels of the reduced image.

3. An image processing apparatus comprising:

a motion prediction processor configured to detect a motion vector that indicates inter-image motion between a current image and a reference image;

a motion compensation processor configured to perform a motion compensation process for the reference image by using the motion vector and generate a motion compensated image;

an addition processor configured to generate a noise reduced image in which noise of the current image is reduced by performing a process for adding the current image and the motion compensated image; and

an addition determination unit configured to compute an addition weight in units of pixels of the motion compensated image that is used for a process for adding the current image and the motion compensated image, the addition process being performed by the addition processor,

wherein the addition determination unit includes

an area determination unit configured to perform analysis of the current image and the motion compensated image and determine an area attribute as to which one of (a) motion estimation success area, (b) motion estimation failure area, and (c) detailed confirmation necessity area that is neither one of (a) and (b) each of small areas in which the motion compensated image is divided corresponds to, and

an addition weight setting unit configured to perform a process for computing an addition weight different according to the area attribute that is a determination result of the area determination unit, the addition weight belonging to each area.

4. The image processing apparatus according to claim 3,

wherein when the small area is a motion estimation success area, the addition weight setting unit performs a process for setting an addition weight corresponding to a pixel contained in the small area to 1, and

when the small area is a motion estimation failure area, the addition weight setting unit performs a process for setting an addition weight corresponding to a pixel contained in the small area to 0.

5. The image processing apparatus according to any one of claims 3 and 4, wherein, when the small area is neither one of the motion estimation success area and the motion estimation failure area, the addition weight setting unit computes an addition weight corresponding to each pixel by analyzing information containing difference information on corresponding pixels of the current image and the motion compensated image in units of pixels contained in the area.

6. The image processing apparatus according to claim 3, wherein the area determination unit computes at least one of a variance value, an average value, a median value, a mode value, a maximum value, and a minimum value corresponding to difference values of corresponding pixels of the current image and the motion compensated image, and performs an area determination process by comparing the computed value with a preset threshold value.

7. The image processing apparatus according to claim 3, wherein the area determination unit is configured to compute a normalization correlation between the current image and the motion compensated image and performs an area determination process by comparing the normalization correlation with a preset threshold value.

8. The image processing apparatus according to claim 3,

wherein the addition determination unit further includes a noise determination table generator configured to generate a noise determination table in which estimated amounts of noise corresponding to pixel values are changed in accordance with the area attribute that is a determination result of the area determination unit, and

wherein the addition weight setting unit performs a process of setting an addition weight different according to the area attribute, the addition weight corresponding to a pixel belonging to each area, by using the amount of noise estimated using the noise determination table.

9. The image processing apparatus according to claim 3, wherein the addition determination unit further includes

an addition weight correction value computation unit configured to compute an addition weight correction value in accordance with the area attribute that is a determination result of the area determination unit,

a computation unit configured to perform a computation process for the value of the addition weight corresponding to the pixel, which is computed by the addition weight setting unit, and the addition weight correction value computed by the addition weight correction value computation unit, and

an addition weight adjustment unit configured to adjust an output of the computation unit to a range of 0 to 1.

10. The image processing apparatus according to any one of claims 3 to 9, wherein the addition determination unit computes an addition weight corresponding to constituent pixels of reduced images of the current image and the motion compensated image generated by a down-sampling processor.

11. An image processing method for use with an image processing apparatus configured to generate a noise reduced image, the image processing method comprising the steps of:

detecting, by using a motion prediction processor, a motion vector that indicates inter-image motion between a current image and a reference image;

performing, by using a motion compensation processor, a motion compensation process for the reference image by using the motion vector and generating a motion compensated image;

generating, by using an addition processor, a noise reduced image in which noise of the current image is reduced by performing a process for adding the current image and the motion compensated image;

computing, by using an addition determination unit, an addition weight in units of pixels of the motion compensated image that is used for a process for adding the current image and the motion compensated image, the addition process being performed by the addition processor;

performing, by using a down-sampling processor, a process for reducing the current image and the motion compensated image that are input to the addition determination unit; and

performing, by using an up-sampling processor, a process for expanding an addition coefficient map that is an output of the addition determination unit.

12. An image processing method for use with an image processing apparatus configured to generate a noise reduced image, the image processing method comprising the steps of:

detecting, by using a motion prediction processor, a motion vector that indicates inter-image motion between a current image and a reference image;

performing, by using a motion compensation processor, a motion compensation process for the reference image by using the motion vector and generating a motion compensated image;

generating, by using an addition processor, a noise reduced image in which noise of the current image is reduced by performing a process for adding the current image and the motion compensated image; and

computing, by using an addition determination unit, an addition weight in units of pixels of the motion compensated image that is used for a process for adding the current image and the motion compensated image, the addition process being performed by the addition processor,

wherein the addition determination step includes the steps of

performing analysis of the current image and the motion compensated image and determining an area attribute as to which one of (a) motion estimation success area, (b) motion estimation failure area, and (c) detailed confirmation necessity area that is neither one of (a) and (b) each of small areas in which the motion compensated image is divided corresponds to, and

performing a process for computing an addition weight belonging to each area, a different process being used in accordance with an area attribute that is a determination result in the area determination step.

13. A program for causing an image processing apparatus to execute image processing comprising the steps of:

causing a motion prediction processor to detect a motion vector that indicates inter-image motion between a current image and a reference image;

causing a motion compensation processor to perform a motion compensation process for the reference image by using the motion vector and generate a motion compensated image;

causing an addition processor to generate a noise reduced image in which noise of the current image is reduced by performing a process for adding the current image and the motion compensated image;

causing an addition determination unit to compute an addition weight in units of pixels of the motion compensated image that is used for a process for adding the current image and the motion compensated image, the addition process being performed by the addition processor;

causing a down-sampling processor to perform a process for reducing the current image and the motion compensated image that are input to the addition determination unit; and

causing an up-sampling processor to perform a process for expanding an addition coefficient map that is an output of the addition determination unit.

14. A program for causing an image processing apparatus to execute image processing comprising the steps of:

causing a motion prediction processor to detect a motion vector that indicates inter-image motion between a current image and a reference image;

causing a motion compensation processor to perform a motion compensation process for the reference image by using the motion vector and generate a motion compensated image;

causing an addition processor to generate a noise reduced image in which noise of the current image is reduced by performing a process for adding the current image and the motion compensated image; and

causing an addition determination unit to compute an addition weight in units of pixels of the motion compensated image that is used for a process for adding the current image and the motion compensated image, the addition process being performed by the addition processor,

wherein the addition determination step includes the steps of

performing analysis of the current image and the motion compensated image and determining an area attribute as to which one of (a) motion estimation success area, (b) motion estimation failure area, and (c) detailed confirmation necessity area that is neither one of (a) and (b) each of small areas in which the motion compensated image is divided corresponds to, and

performing a process for computing an addition weight belonging to each area, a different process being used in accordance with an area attribute that is a determination result in the area determination step.