Imaging apparatus and image signal processing device

• US 7,956,325 B2
• Filed: 11/06/2007
• Issued: 06/07/2011
• Est. Priority Date: 11/10/2006
• Status: Active Grant

• Current Assignee: Semiconductor Components Industries LLC (ON Semiconductor Corporation)
• Original Assignee: Sanyo Semiconductor Company Limited (ON Semiconductor Corporation), Samsung Electronics Co. Ltd.
• Inventors: Tanimoto, Takishi
• Primary Examiner(s): Porta; David P
• Assistant Examiner(s): Bryant; Casey
• Application Number:
  US11/979,615
• Publication Number:
  US 20080111894A1
• Time in Patent Office: 1,309 Days
• Field of Search: 250330-33905
• US Class Current: 250/332
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 25/20 for transforming only infrared radiation into image signals
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Infrared light emitted by a fluorescent material is weak. When an infrared image obtained by this fluorescent material is superposed and displayed on a visible-light image, the infrared image is made clear. A signal <IR> of an IR pixel having selective sensitivity to infrared light is subtracted from RGB pixel signals <R>, <B>, and <G> of a solid-state image sensor by adders (20), and signals R₀, G₀, and B₀ that correspond to the R, G, and B components of incident light within <R>, <G>, and <B> are separated and extracted. <IR> is multiplied by each of the gains κ_R, κ_G, and κ_B to enhance the IR component by multipliers (22), and each is synthesized with R₀, G₀, and B₀ by adders (24). κ_R, κ_G, and κ_B are set so that the infrared image displayed on the visible-light image by the synthesized IR component is enhanced and displayed in a hue that can be distinguished from the color white or another visible-light image.

Claims

1. An imaging apparatus comprising:

a solid-state image sensor having, in a two dimensional array, infrared light-receiving pixels having selective sensitivity to infrared light and configured to output an infrared pixel signal, and a plurality of types of color light-receiving pixels having sensitivity to the infrared light and to visible light of mutually different specific colors and configured to output a plurality of types of color pixel signals each of which corresponds to each type of the color light-receiving pixels; and

an image signal processing circuit configured to generate a processed image signal for displaying an infrared composite image in which an enhanced infrared image is synthesized with and displayed on a visible image on the basis of a raw image signal composed of an infrared pixel signal that corresponds to the infrared light-receiving pixels and a plurality of types of color pixel signals that corresponds to each of the color light-receiving pixels, wherein

the image signal processing circuit includes;

a component image signal generator configured to generate an infrared component image signal that corresponds to an infrared light component, and to generate a plurality of types of color component image signals that corresponds to visible-light components of the specific colors on the basis of the infrared pixel signal and the color pixel signals;

an infrared composite synthesizer configured to synthesize, for each of the specific colors, the color component image signal and a signal obtained by multiplying the infrared component image signal by a gain of the specific color in accordance with the manner in which the infrared image is enhanced, and to generate a converted color component image signal of each of the specific colors; and

a processed image signal generator configured to generate the processed image signal on the basis of the plurality of converted color component image signals.

2. The image signal processing device according to claim 1, wherein

the component image signal generator has a first adder for each of the specific colors and is configured to output the infrared pixel signal as the infrared component image signal, the first adder being configured to subtract a signal that is obtained by using the infrared pixel signal and is equivalent to a signal generated by the infrared light incident on the color light-receiving pixel of a specific color corresponding to the first adder from the color pixel signal of the specific color corresponding to the first adder, and being configured to output a color component image signal corresponding to a visible-light component of the specific color;

and

the infrared composite synthesizer has a multiplier and a second adder for each of the specific colors, the multiplier being configured to multiply the infrared component image signal by the gain of a specific color corresponding to the multiplier, and the second adder being configured to add a signal outputting from the multiplier and the color component image signal of the specific color, and to output the converted color component image signal of the specific color.

3. An imaging apparatus comprising:

a solid-state image sensor having, in a two dimensional array, infrared light-receiving pixels having selective sensitivity to infrared light and configured to output an infrared pixel signal, and a plurality of types of color light-receiving pixels having sensitivity to the infrared light and to visible light of mutually different specific colors and configured to output a plurality of types of color pixel signals each of which corresponds to each type of the color light-receiving pixels; and

an image signal processing circuit configured to generate a processed image signal for displaying an infrared composite image in which an infrared image is synthesized with and displayed on a visible image on the basis of a raw image signal composed of an infrared pixel signal that corresponds to the infrared light-receiving pixels and a plurality of types of color pixel signals that corresponds to each of the color light-receiving pixels;

wherein

the image signal processing circuit includes;

a component image signal generator configured to generate an infrared component image signal that corresponds to an infrared light component, and to generate a plurality of types of color component image signals that corresponds to visible-light components of the specific colors on the basis of the infrared pixel signal and the color pixel signals;

an intermediate image generator configured to generate the intermediate image signal that corresponds to the visible-light image on the basis of the plurality of color component image signals; and

a processed image signal generator configured to generate the processed image signal by varying luminance or hue of each pixel that expresses the intermediate image signal in accordance with signal level of the infrared component image signal.

4. An image signal processing device configured to generate a processed image signal for displaying an infrared composite image in which an enhanced infrared image is synthesized with and displayed on a visible image on the basis of a raw image signal that is obtained by an image sensor having, in a two dimensional array, infrared light-receiving pixels having selective sensitivity to infrared light and configured to output an infrared pixel signal, and a plurality of types of color light-receiving pixels having sensitivity to the infrared light and to visible light of mutually different specific colors and configured to output a plurality of types of color pixel signals each of which corresponds to each type of the color light-receiving pixels, and that is composed of an infrared pixel signal and a plurality of types of color pixel signals;

the image signal processing device includes;

a component image signal generator configured to generate an infrared component image signal that corresponds to an infrared light component, and to generate a plurality of types of color component image signals that corresponds to visible-light components of the specific colors on the basis of the infrared pixel signal and the color pixel signals;

an infrared composite synthesizer configured to synthesize, for each of the specific colors, the color component image signal and a signal obtained by multiplying the infrared component image signal by a gain of the specific color in accordance with the manner in which the infrared image is enhanced, and to generate a converted color component image signal of each of the specific colors; and

a processed image signal generator configured to generate the processed image signal on the basis of the plurality of converted color component image signals.

5. The image signal processing device according to claim 4, wherein the gains are set to vary luminance of the infrared composite image in accordance with the infrared component image signal.

6. The image signal processing device according to claim 4, wherein the gains are set to vary hue of each pixel on the infrared composite image in accordance with the signal level of the infrared component image signal.

7. The image signal processing device according to claim 6, wherein, the processed image signal generator is further configured to perform an enhanced hue setting process for setting the hue to a predetermined enhanced hue when the signal level of the infrared component image signal is at a predetermined threshold or above.

8. The image signal processing device according to claim 7, wherein the processed image signal generator increases saturation of the enhanced hue in conjunction with the signal level of the infrared component image signal at or above the threshold.

9. The image signal processing device according to claim 4, wherein the processed image signal generator further includes:

a visible image suppression processor configured to stop the input of the color component image signals into the infrared composite synthesizer; and

a composition control processor configured to control the visible image suppression processor and allowing switching to be performed between generation of the processed image signal for displaying the infrared composite image, and generation of the processed image signal for displaying an image composed of the infrared image alone.

10. The image signal processing device according to claim 4, further comprising:

a switching circuit configured to stop the input of the color component image signals into the second adders; and

a control circuit configured to turn the switching circuit on and off, whereby an image output from the processed image signal generator is switched between the infrared composite image and an image composed of the infrared image alone.

11. An image signal processing device configured to generate a processed image signal for displaying an infrared composite image in which an infrared image is synthesized with and displayed on a visible image on the basis of a raw image signal that is obtained by an image sensor having, in a two dimensional array, infrared light-receiving pixels having selective sensitivity to infrared light and being configured to output an infrared pixel signal, and a plurality of types of color light-receiving pixels having sensitivity to the infrared light and to visible light of mutually different specific colors and being configured to output a plurality of types of color pixel signals each of which corresponds to each type of the color light-receiving pixels, and that is composed of an infrared pixel signal and a plurality of types of color pixel signals;

the image signal processing device comprising;

a component image signal generator configured to generate an infrared component image signal that corresponds to an infrared light component, and to generate a plurality of types of color component image signals that corresponds to visible-light components of the specific colors on the basis of the infrared pixel signal and the color pixel signals;

an intermediate image generator configured to generate the intermediate image signal that corresponds to the visible-light image on the basis of the plurality of color component image signals; and

a processed image signal generator configured to generate the processed image signal by varying luminance or hue of the pixels for displaying the intermediate image signal, in accordance with signal level of the infrared component image signal.

12. The image signal processing device according to claim 11, wherein processed image signal generator further includes an achromatic color setting processor configured to change the hue expressed by the intermediate image signal to an achromatic color for pixels in which the signal level of the infrared component image signal is less than a predetermined threshold.

13. The image signal processing device according to claim 11, wherein

the component image signal generator has an adder for each of the specific colors and is configured to output the infrared pixel signal as the infrared component image signal, the adder being configured to subtract a signal that is obtained by using the infrared pixel signal and is equivalent to a signal generated by the infrared light incident on the color light-receiving pixel of a specific color corresponding to the adder from the color pixel signal of the specific color corresponding to the adder, and being configured to output a color component image signal corresponding to a visible-light component of the specific color;

and

the intermediate image generator has a matrix arithmetic circuit, the matrix arithmetic circuit being configured to perform a matrix calculation for converting a plurality of types of the color component image signals into a luminance signal and color difference signals which correspond to the intermediate image signal in a situation without synthesizing the infrared image;

the processed image signal generator has a control circuit, the control circuit being configured to manipulate conversion coefficients of the matrix calculation for each pixel in the intermediate image signal in accordance with signal level of the infrared pixel signal when the infrared image is synthesized with the visible image, and the matrix arithmetic circuit being configured to generate the processed image of which luminance or hue changed of each pixel is changed from that of the intermediate image.