Image pickup apparatus, image processing method, and computer program capable of obtaining high-quality image data by controlling imbalance among sensitivities of light-receiving devices

• US 7,839,437 B2
• Filed: 05/10/2007
• Issued: 11/23/2010
• Est. Priority Date: 05/15/2006
• Status: Active Grant

• Current Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kasai, Masanori, Oike, Yusuke
• Primary Examiner(s): VU, NGOC YEN T
• Application Number:
  US11/747,129
• Publication Number:
  US 20080012969A1
• Time in Patent Office: 1,293 Days
• Field of Search: 348/236, 348/238, 348/273, 348/276, 348/370
• US Class Current: 348/238
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements

Abstract

An image pickup apparatus includes an image pickup device that has high-sensitivity pixel devices receiving a relatively large amount of light and low-sensitivity pixel devices receiving a relatively small amount of light, an exposure control unit independently controlling exposure periods of the high-sensitivity pixel devices and the low-sensitivity pixel devices, and an image generation unit performing image generation on the basis of an output signal of the image pickup device. The image generation unit compares a high-sensitivity pixel evaluation image generated using data output from the high-sensitivity pixel devices with a low-sensitivity pixel evaluation image generated using data output from the low-sensitivity pixel devices by obtaining a difference or ratio between pixel values of corresponding pixels in the two evaluation images, and performs different types of image processing for a region composed of pixels each having a small difference and a region composed of pixels each having a large difference.

Claims

1. An image pickup apparatus comprising:

an image pickup device having a plurality of high-sensitivity pixel devices that receive a relatively large amount of light and a plurality of low-sensitivity pixel devices that receive a relatively small amount of light, the high-sensitivity pixel devices and the low-sensitivity pixel devices being arranged in an array;

an exposure control unit configured to independently control an exposure period of the high-sensitivity pixel devices and an exposure period of the low-sensitivity pixel devices, the high-sensitivity pixel devices and the low-sensitivity pixel devices being included in the image pickup device;

an image generation unit configured to perform image generation processing on the basis of a signal output from the image pickup device, and

wherein the image generation unit performs comparison processing of a high-sensitivity pixel evaluation image generated using data output from the high-sensitivity pixel devices and a low-sensitivity pixel evaluation image generated using data output from the low-sensitivity pixel devices by obtaining a difference or ratio between a pixel value of each pixel included in the high-sensitivity pixel evaluation image and a pixel value of a corresponding pixel included in the low-sensitivity pixel evaluation image, discriminates a pixel region composed of pixels each having a small difference between corresponding pixels in the two evaluation images from a pixel region composed of pixels each having a large difference between corresponding pixels in the two evaluation images, and performs different types of image processing for the pixel regions.

2. The image pickup apparatus according to claim 1, wherein the image generation unit generates an image for the pixel region composed of pixels each having a small difference between corresponding pixels in the two evaluation images by performing image processing using both of the data output from the high-sensitivity pixel devices and the data output from the low-sensitivity pixel devices, and generates an image for the pixel region composed of pixels each having a large difference between corresponding pixels in the two evaluation images by performing image processing using only one of the data output from the high-sensitivity pixel devices and the data output from the low-sensitivity pixel devices.

3. The image pickup apparatus according to claim 1, wherein each of the low-sensitivity pixel devices included in the image pickup device is configured to receive light in an R, G, or B specific wavelength range, and each of the high-sensitivity pixel devices included in the image pickup device is configured to receive light in a visible light range, and

wherein, when it is assumed that an evaluation image generated using data output from the low-sensitivity pixel devices is a low-sensitivity pixel evaluation image WL and an evaluation image generated using data output from the high-sensitivity pixel devices is a high-sensitivity pixel evaluation image WH, the image generation unit generates the low-sensitivity pixel evaluation image WL on the basis of equation WL=aR+bG+cB, where coefficients a, b, and c function as parameters and are determined in accordance with characteristics of the image pickup device.

4. The image pickup apparatus according to claim 3, wherein the image generation unit determines values of the parameters in accordance with a light source.

5. The image pickup apparatus according to claim 1, wherein the image generation unit compares an absolute value |E| of a difference between pixel values of corresponding pixels in the high-sensitivity pixel evaluation image and the low-sensitivity pixel evaluation image with a predetermined threshold value, and performs different types of image processing for a pixel region composed of pixels each having a pixel value difference larger than the threshold value and a pixel region composed of pixels each having a pixel value difference equal to or smaller than the threshold value.

6. The image pickup apparatus according to claim 1, wherein the image generation unit compares an absolute value |E| of a difference between pixel values of corresponding pixels in the high-sensitivity pixel evaluation image and the low-sensitivity pixel evaluation image with a plurality of predetermined threshold values, and performs different types of image processing in accordance with degrees of difference among a pixel value difference and the threshold values.

7. The image pickup apparatus according to claim 6, wherein the image generation unit performs different types of image processing in accordance with degrees of difference among a pixel value difference and the threshold values by using the data output from the high-sensitivity pixel devices and the data output from the low-sensitivity pixel devices at different ratios.

8. The image pickup apparatus according to claim 1, wherein the image generation unit removes high-frequency components from the high-sensitivity pixel evaluation image and the low-sensitivity pixel evaluation image prior to the comparison processing of them.

9. The image pickup apparatus according to claim 1, wherein the image generation unit generates an image for a pixel region composed of a pixel having a large difference between pixel values of corresponding pixels in the two evaluation images and pixels around the pixel by performing image processing using only one of the data output from the high-sensitivity pixel devices and the data output from the low-sensitivity pixel devices.

10. The image pickup apparatus according to claim 1, wherein the image generation unit performs comparison processing of the high-sensitivity pixel evaluation image generated using data that has yet to be subjected to interpolation processing and the low-sensitivity pixel evaluation image generated using data that has yet to be subjected to interpolation processing.

11. The image pickup apparatus according to any one of claims 1 to 10, wherein the high-sensitivity pixel devices included in the image pickup device receive visible light and invisible light such as infrared light.

12. An image processing method of performing image processing in an image pickup apparatus, comprising the steps of:

using an image pickup device having a plurality of high-sensitivity pixel devices that receive a relatively large amount of light and a plurality of low-sensitivity pixel devices that receive a relatively small amount of light, independently controlling an exposure period of the high-sensitivity pixel devices and an exposure period of the low-sensitivity pixel devices at the time of image capturing, and inputting an image capturing signal output from the image pickup device into an image generation unit;

causing the image generation unit to perform comparison processing of a high-sensitivity pixel evaluation image generated using data output from the high-sensitivity pixel devices and a low-sensitivity pixel evaluation image generated using data output from the low-sensitivity pixel devices by obtaining a difference or ratio between a pixel value of each pixel included in the high-sensitivity pixel evaluation image and a pixel value of a corresponding pixel included in the low-sensitivity pixel evaluation image; and

causing the image generation unit to discriminate a pixel region composed of pixels each having a small difference between corresponding pixels in the two evaluation images from a pixel region composed of pixels each having a large difference between corresponding pixels in the two evaluation images, and perform different types of image processing for the pixel regions.

13. The image processing method according to claim 12, wherein image processing is performed for the pixel region composed of pixels each having a small difference between corresponding pixels in the two evaluation images using both of the data output from the high-sensitivity pixel devices and the data output from the low-sensitivity pixel devices, and image processing is performed for the pixel region composed of pixels each having a large difference between corresponding pixels in the two evaluation images using only one of the data output from the high-sensitivity pixel devices and the data output from the low-sensitivity pixel devices.

14. The image processing method according to claim 12, wherein each of the low-sensitivity pixel devices included in the image pickup device is configured to receive light in an R, G, or B specific wavelength range, and each of the high-sensitivity pixel devices included in the image pickup device is configured to receive light in a visible light range, and

wherein, when it is assumed that an evaluation image generated using data output from the low-sensitivity pixel devices is a low-sensitivity pixel evaluation image WL and an evaluation image generated using data output from the high-sensitivity pixel devices is a high-sensitivity pixel evaluation image WH, the image generation unit generates the low-sensitivity pixel evaluation image WL on the basis of equation WL=aR+bG+cB, where coefficients a, b, and c function as parameters and are determined in accordance with characteristics of the image pickup device.

15. The image processing method according to claim 14, wherein the image generation unit determines values of the parameters in accordance with a light source so as to generate the low-sensitivity pixel evaluation image WL.

16. The image processing method according to claim 12, wherein an absolute value |E| of a difference between pixel values of corresponding pixels in the high-sensitivity pixel evaluation image and the low-sensitivity pixel evaluation image is compared with a predetermined threshold value, and different types of image processing is performed for a pixel region composed of pixels each having a pixel value difference larger than the threshold value and a pixel region composed of pixels each having a pixel value difference equal to or smaller than the threshold value.

17. The image processing method according to claim 12, wherein an absolute value |E| of a difference between pixel values of corresponding pixels in the high-sensitivity pixel evaluation image and the low-sensitivity pixel evaluation image is compared with a plurality of predetermined threshold values, and different types of image processing is performed in accordance with degrees of difference among a pixel value difference and the threshold values.

18. The image processing method according to claim 17, wherein different types of image processing is performed in accordance with degrees of difference among a pixel value difference and the threshold values by using the data output from the high-sensitivity pixel devices and the data output from the low-sensitivity pixel devices at different ratios.

19. The image processing method according to claim 12, wherein high-frequency components are removed from the high-sensitivity pixel evaluation image and the low-sensitivity pixel evaluation image prior to the comparison processing of them.

20. The image processing method according to claim 12, wherein image generation processing is performed for a pixel region composed of a pixel having a large difference between pixel values of corresponding pixels in the two evaluation images and pixels around the pixel by performing image processing using only one of the data output from the high-sensitivity pixel devices and the data output from the low-sensitivity pixel devices.

21. The image processing method according to claim 12, wherein comparison processing of the high-sensitivity pixel evaluation image generated using data that has yet to be subjected to interpolation processing and the low-sensitivity pixel evaluation image generated using data that has yet to be subjected to interpolation processing is performed.

22. The image processing method according to any one of claims 12 to 21, wherein the high-sensitivity pixel devices included in the image pickup device receive visible light and invisible light such as infrared light.

23. A non-transitory computer readable medium comprising a computer program causing an image pickup apparatus to perform image processing comprising the steps of:

under the control of a control unit, using an image pickup device having a plurality of high-sensitivity pixel devices that receive a relatively large amount of light and a plurality of low-sensitivity pixel devices that receive a relatively small amount of light, independently controlling an exposure period of the high-sensitivity pixel devices and an exposure period of the low-sensitivity pixel devices at the time of image capturing, and inputting an image capturing signal output from the image pickup device into an image generation unit;

causing the image generation unit to perform comparison processing of a high-sensitivity pixel evaluation image generated using data output from the high-sensitivity pixel devices and a low-sensitivity pixel evaluation image generated using data output from the low-sensitivity pixel devices by obtaining a difference or ratio between a pixel value of each pixel included in the high-sensitivity pixel evaluation image and a pixel value of a corresponding pixel included in the low-sensitivity pixel evaluation image; and

causing the image generation unit to discriminate a pixel region composed of pixels each having a small difference between corresponding pixels in the two evaluation images from a pixel region composed of pixels each having a large difference between corresponding pixels in the two evaluation images, and perform different types of image processing for the pixel regions.