IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM

• US 20120274798A1
• Filed: 07/07/2010
• Published: 11/01/2012
• Est. Priority Date: 07/07/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Takahashi, Hiroaki, Mitsunaga, Tomoo
• Granted Patent: US 8,659,675 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

The present invention relates to an image processing apparatus, an image processing method, and a program that are capable of removing an influence due to illumination components from a plurality of images in which illumination environments are different and generating an image having high color reproducibility.

A demosaic processing unit 221 performs a demosaic process on a reference image held in a reference mosaic image holding unit 111. A demosaic processing unit 222 performs a demosaic process on a processing target image that is held in a processing target mosaic image holding unit 112. A white balance calculation unit 230 calculates a white-balance value for each channel with respect to the reference image. A channel gain calculation unit 250 calculates a gain for each channel for converting illumination components on the basis of RGB values of the reference image and the processing target image, and the white balance of the reference image. At the time of gain calculation, blackout condition, saturation, and the like are considered. A channel gain application unit 260 applies the gain calculated by the channel gain calculation unit 250 to each channel of the processing target image.

Claims

1. An image processing apparatus comprising:

a channel gain calculation unit that calculates a gain for converting illumination components for each channel at each pixel position on the basis of a reference image and a processing target image that are captured under different illumination environments; and

a channel gain application unit that applies the gain to the processing target image.

2. The image processing apparatus according to claim 1, wherein the channel gain calculation unit calculates a gain in such a manner that a gain of a specific channel of the processing target image is fixed and a color balance of a processing target pixel is made to match that of the reference image.

3. The image processing apparatus according to claim 1, wherein the channel gain calculation unit fixes a luminance value of the processing target image and calculates the gain so that a color balance of the processing target pixel matches that of the reference image.

4. The image processing apparatus according to claim 1, wherein the channel gain calculation unit sets a predetermined upper limit value as the gain in a case where a value exceeding the upper limit value is calculated as the gain, or sets a predetermined lower limit value as the gain in a case where a value that falls below the lower limit value is calculated as the gain.

5. The image processing apparatus according to claim 1, wherein the channel gain calculation unit includes

a saturation degree calculation unit that calculates a saturation degree of pixels on the basis of pixel values of the reference image, and

a saturation compensation unit that performs a process for compensating for the gain in accordance with the saturation degree.

6. The image processing apparatus according to claim 1, wherein the channel gain calculation unit includes

a blackout condition degree calculation unit that calculates a blackout condition degree of pixels on the basis of pixel values of the reference image;

and

a blackout condition compensation unit that performs the process for compensating for the gain in accordance with the blackout condition degree.

7. The image processing apparatus according to claim 1, wherein the channel gain calculation unit includes

a blackout condition degree calculation unit that calculates a blackout condition degree of pixels on the basis of pixel values of the reference image and an inter-channel ratio of the reference image to the processing target image, and

a blackout condition compensation unit that performs the process for compensating for the gain in accordance with the blackout condition degree.

8. The image processing apparatus according to claim 1, further comprising an exposure compensation unit that adjusts an intensity so that exposure intensities become equal in order to compensate for a difference between image-capturing conditions of the reference image and the processing target image.

9. The image processing apparatus according to claim 1, wherein the channel gain application unit includes

a saturation reliability calculation unit that calculates a saturation reliability of pixels on the basis of pixel values of the reference image, and

a smoothing processing unit that performs spatial smoothing on the gain on the basis of the saturation reliability and then applies the gain to the processing target pixel.

10. The image processing apparatus according to claim 1, wherein the channel gain application unit includes

a blackout condition reliability calculation unit that calculates a blackout condition reliability of pixels on the basis of pixel values of the reference image, and

a smoothing processing unit that performs spatial smoothing on the gain on the basis of the blackout condition reliability and then applies the gain to the processing target pixel.

11. The image processing apparatus according to claim 1, wherein the channel gain application unit includes

a blackout condition reliability calculation unit that calculates a blackout condition reliability of pixels on the basis of pixel values of the reference image and an inter-channel ratio of the reference image to the processing target image, and

a smoothing processing unit that performs spatial smoothing on the gain on the basis of the blackout condition reliability and that applies the gain to the processing target pixel.

12. The image processing apparatus according to claim 1, wherein the channel gain application unit includes

a moving subject reliability calculation unit that determines whether or not the pixel is a subject that has moved in space on the basis of a change in characteristic values in corresponding pixels of the reference image and the processing target image and that calculates a moving subject reliability;

and

a smoothing processing unit that performs spatial smoothing on the gain on the basis of the moving subject reliability and then applies the gain to the processing target pixel.

13. The image processing apparatus according to claim 12, wherein the moving subject reliability calculation unit includes

a ratio reverse degree calculation unit that calculates the moving subject reliability in accordance with whether or not a change of signal intensities of pixels of the processing target image is in reverse to the direction of a change of illumination components of the entire reference image and processing target image.

14. The image processing apparatus according to claim 12, wherein the moving subject reliability calculation unit includes

a spectral reflectance change amount prediction unit that calculates the moving subject reliability in accordance with a change in the spectral reflectances of corresponding pixels of the reference image and the processing target image.

15. The image processing apparatus according to claim 1, further comprising a resolution conversion unit that converts a resolution of the reference image or the processing target image and then supplies the image to the channel gain calculation unit.

16. The image processing apparatus according to claim 15, wherein the resolution conversion unit performs the resolution conversion by reduction based on thinning out of pixels.

17. The image processing apparatus according to claim 15, wherein the resolution conversion unit performs the resolution conversion by reduction based on an average of pixels in a block.

18. The image processing apparatus according to claim 15, wherein the resolution conversion unit performs the resolution conversion by reduction based on a central value of pixels in a block.

19. The image processing apparatus according to claim 15, wherein the resolution conversion unit performs the resolution conversion by an edge preservation type smoothing filter process.

20. The image processing apparatus according to claim 15, wherein the resolution conversion unit performs the resolution conversion so that sizes of the reference image and the processing target image are made to match each other.

21. The image processing apparatus according to claim 1, further comprising:

a frame memory for storing images that are captured continuously;

an addition unit that adds the images that are captured continuously while performing weighting thereon and that generates an input image; and

a control parameter determination unit that determines a control parameter used for the continuous image capturing so as to perform image-capture repetition control.

22. The image processing apparatus according to claim 1, wherein the reference image is an image captured without flash emission, and the processing target image is an image captured with flash emission.

23. The image processing apparatus according to claim 1, wherein the channel gain application unit includes

a luminance calculation unit that calculates a luminance on the basis of pixel values of the processing target image, and

a smoothing processing unit that performs spatial smoothing on the gain on the basis of the difference in the luminances between pixels and then applies the gain to the processing target pixel.

24. The image processing apparatus according to claim 23, wherein the luminance calculation unit calculates the luminance as a linear sum using a weight that is set in advance to the pixel value of the processing target image.

25. The image processing apparatus according to claim 23, wherein the luminance calculation unit calculates the luminance on the basis of multiple regression analysis in which an intensity ratio of corresponding pixels of the reference image to the processing target image is an object variate, and the pixel values of the processing target image is an explanatory variate.

26. An image processing apparatus comprising:

a block histogram calculation unit that calculates a frequent value of pixels of a processing target image as a block histogram with regard to each of blocks such that an area is divided into a plurality of portions in a spatial axis direction and in a luminance axis direction;

a block integration value calculation unit that calculates an integration value of characteristic values belonging to each of the blocks;

a weighted product-sum unit that calculates a global gain value at the target pixel position on the basis of the block histogram, the block integration value, and the luminance value at the target pixel position; and

a gain application unit that applies the global gain value to the processing target image.

27. The image processing apparatus according to claim 26, wherein the integration value of the characteristic value for each of the blocks is a grand total, for each channel, of gains for converting illumination components for each channel of pixels belonging to each of the blocks.

28. The image processing apparatus according to claim 26, wherein the integration value of the characteristic values for each block is such that a central value of the gains, for each channel, for converting illumination components of pixels belonging to each of the blocks is multiplied by a frequent value of pixels of the processing target image in the block.

29. The image processing apparatus according to claim 26, wherein the weighted product-sum unit includes

a first interpolation unit that performs an interpolation of the block histogram to the target pixel position by using a weight function in a spatial axis direction, which is defined in advance;

a second interpolation unit that performs an interpolation of a characteristic value for each block to the target pixel position by using a weight function in a spatial axis direction, which is defined in advance;

a first product-sum unit that calculates a load sum of the interpolated block histogram by using a weight function in a luminance axis direction, which is defined in advance;

a second product-sum unit that calculates a load sum of interpolated characteristic values by using a weight function in a luminance axis direction, which is defined in advance; and

a division unit that divides an output of the second product-sum unit by an output of the first product-sum unit.

30. The image processing apparatus according to claim 26, wherein the weighted product-sum unit includes

a first division unit that divides the characteristic value for each block by the value of the block histogram so as to calculate an average characteristic value at each block position;

a comparison unit that compares the average characteristic value with a characteristic value regarding the target pixel position so as to calculate a weight;

a first multiplication unit that multiplies the weight calculated in the comparison unit by the characteristic value for each block at a corresponding block position;

a second multiplication unit that multiplies the weight calculated in the comparison unit by the value of the block histogram at a corresponding block position;

a first interpolation unit that performs an interpolation of a weighted block histogram that is an output of the first multiplication unit to the target pixel position in accordance with a weight function in a spatial axis direction, which is defined in advance;

a second interpolation unit that performs an interpolation of a weighted characteristic value for each block, which is an output of the second multiplication unit, to the target pixel position by using a weight function in a spatial axis direction, which is defined in advance;

a first product-sum unit that calculates a load sum of the interpolated block histograms by using a weight function in a luminance axis direction, which is defined in advance;

a second product-sum unit that calculates a load sum of the interpolated characteristic values by using a weight function in the luminance axis direction, which is defined in advance; and

a second division unit that divides an output of the second product-sum unit by an output of the first product-sum unit.

31. An image processing method comprising:

a channel gain calculation procedure of calculating a gain for converting illumination components for each channel at each pixel position on the basis of a reference image and a processing target image that are captured under different illumination environments; and

a channel gain application procedure of applying the gain to the processing target image.

32. An image processing method comprising:

a block histogram calculation procedure of calculating, as a block histogram, a frequent value of pixels of a processing target image regarding each block in which an area is divided into a plurality of portions in a spatial axis direction and in a luminance axis direction;

a block integration value calculation procedure of calculating an integration value of characteristic values belonging to each of the blocks;

a weighted product-sum procedure of calculating a global gain value at the target pixel position on the basis of the block histogram, the block integration value, and the luminance value at the target pixel position; and

a gain application procedure of applying the global gain value to the processing target image.

33. A program for causing a computer to perform:

a channel gain calculation procedure of calculating a gain for converting illumination components for each channel at each pixel position on the basis of a reference image and a processing target image that are captured under different illumination environments; and

a channel gain application procedure of applying the gain to the processing target image.

34. A program for causing a computer to perform:

a block histogram calculation procedure of calculating, as a block histogram, a frequent value of pixels of a processing target image regarding each block in which an area is divided into a plurality of portions in a spatial axis direction and in a luminance axis direction;

a block integration value calculation procedure of calculating an integration value of characteristic values belonging to each of the blocks;

a weighted product-sum procedure of calculating a global gain value at the target pixel position on the basis of the target block integration value of the block histogram and the luminance value at the pixel position; and

a gain application procedure of applying the global gain value to the processing target image.