Combined RGB and IR Imaging Sensor

• US 20100289885A1
• Filed: 10/03/2008
• Published: 11/18/2010
• Est. Priority Date: 10/04/2007
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Harris, James Daniel, Higgins-Luthman, Michael J., Lu, Yuesheng, Livengood, William R.
• Granted Patent: US 8,446,470 B2
• Time in Patent Office: Days
• US Class Current: 348/61
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

An imaging system suitable for use in a vehicle includes an imaging sensor having a two-dimensional array of photosensing elements, an image processor for processing image data captured by the imaging sensor, and a display for displaying images responsive to the image processor. The array has a plurality of sub-arrays, with each of the sub-arrays having a first pixel sensing primarily a first color, a second pixel sensing primarily a second color, a third pixel sensing primarily a third color and a fourth pixel sensing primarily infrared radiation. The image processor processes the output of the fourth pixel to determine an infrared component of the imaged scene and subtracts the infrared component from the outputs of the first, second and third pixels to obtain a more accurate or true color response for the pixels and to limit or avoid infrared color wash-out.

Claims

1. An imaging system suitable for use in a vehicle, said imaging system comprising:

an imaging sensor comprising a two-dimensional array of photosensing elements, said two-dimensional array comprising a plurality of sub-arrays, each of said sub-arrays having a first pixel sensing primarily a first color, a second pixel sensing primarily a second color, a third pixel sensing primarily a third color and a fourth pixel sensing primarily infrared radiation;

an image processor for processing image data captured by said imaging sensor, wherein said image processor processes outputs of said pixels, said image processor processing said output of said fourth pixel to determine an infrared component of the imaged scene and subtracting the infrared component from said outputs of said first, second and third pixels to obtain a more accurate or true color response for the pixels and to limit or avoid infrared color wash-out.

2. The imaging system of claim 1, wherein said image processor processes image data via a de-mosaicing technique.

3. The imaging system of claim 2, wherein said image processor processes image data to determine the color values at each pixel location of said sub-array via the following equations:

R(r0c0)=R1−

A_R*IR1, G(r0c0)=G1−

A_G*IR1, B(r0c0)=B1−

A_B*IR1, and IR(r0c0)=IR1;

R(r0c1)=R1−

A_R*IR1, G(r0c1)=G1−

A_G*IR1, B(r0c1)=B1−

A_B*IR1, and IR(r0c1)=IR1;

R(r1c0)=R1−

A_R*IR1, G(r1c0)=G1−

A_G*IR1, B(r1c0)=B1−

A_B*IR1, and IR(r1c0)=IR1;

R(r1c1)=R1−

A_R*IR1, G(r1c1)=G1−

A_G*IR1, B(r1c1)=B1−

A_B*IR1, and IR(r1c1)=IR1,

where A_R, A_G, and A_B are the selected factors for said first pixels, said second pixels and said third pixels, respectively, and where R(rx,cx) is the first color value for the respective pixel location, G(rx,cx) is the second color value for the respective pixel location, B(rx,cx) is the third color value for the respective pixel location, and IR(rx,cx) is the infrared value for the respective pixel location, and where R1 is an output value of said first pixel, G1 is an output value of said second pixel, B1 is an output value of said third pixel and IR1 is an output value of said fourth pixel.

4. The imaging system of claim 2, wherein said image processor processes image data from multiple sub-arrays to determine the color values at each pixel location of said sub-arrays via the following equations:

R(r0c0)=R1−

A_R*IR1, G(r0c0)=G1−

A_G*IR1, B(r0c0)=B1−

A_B*IR1, and IR(r0c0)=IR1;

R(r0c1)=R2−

A_R*IR1, G(r0c1)=G1−

A_G*IR1, B(r0c1)=B2−

A_B*IR1, and IR(r0c1)=IR1;

R(r1c0)=R3−

A_R*IR1, G(r1c0)=G3−

A_G*IR1, B(r1c0)=B1−

A_B*IR1, and IR(r1c0)=IR1;

R(r1c1)=R4−

A_R*IR1, G(r1c1)=G3−

A_G*IR1, B(r1c1)=B2−

A_B*IR1, and IR(r1c1)=IR1,

where A_R, A_G, and A_B are the selected factors for said first pixels, said second pixels and said third pixels, respectively, and where R(rx,cx) is the first color value for the respective pixel location, G(rx,cx) is the second color value for the respective pixel location, B(rx,cx) is the third color value for the respective pixel location, and IR(rx,cx) is the infrared value for the respective pixel location, and where Rx is an output value of said first pixel of a respective sub-array, Gx is an output value of said second pixel of a respective sub-array, Bx is an output value of said third pixel of a respective sub-array and IRx is an output value of said fourth pixel of a respective sub-array.

5. The imaging system of claim 2, wherein said image processor processes image data from multiple sub-arrays to determine the color values at each pixel location of said sub-arrays via the following equations:

R(r1c1)=(R1+R2+R3+R4)/4−

A_R*IR1, G(r1c1)=(G1+G3)/2−

A_G*IR1, B(r1c1)=(B1+B2)/2−

A_B*IR1, and IR(r0c0)=IR1;

R(r1c2)=(R2+R4)/2−

A_R*(IR1+IR2)/2, G(r1c2)=(G1+G2+G3+G4)/4−

A_G*(IR1+IR2)/2, B(r0c1)=B2−

A_B*(IR1+IR2)/2, and IR(r0c1)=(IR1+IR2)/2;

R(r2c1)=(R3+R4)/2−

A_R*(IR1+IR3)/2, G(r2c1)=G3−

A_G*(IR1+IR3)/2, B(r2c1)=(B1+B2+B3+B4)/4−

A_B*(IR1+IR3)/2, and IR(r2c1)=(IR1+IR3)/2;

R(r2c2)=R4−

A_R*(IR1+IR2+IR3+IR4)/4, G(r1c1)=(G3+G4)/2−

A_G*(IR1+IR2+IR3+IR4)/4, B(r2c2)=(B2+B4)/2−

A_B*(IR1+IR2+IR3+IR4)/4, and IR(r2c2)=(IR1+IR2+IR3+IR4)/4,

where A_R, A_G, and A_B are the selected factors for said first pixels, said second pixels and said third pixels, respectively, and where R(rx,cx) is the first color value for the respective pixel location, G(rx,cx) is the second color value for the respective pixel location, B(rx,cx) is the third color value for the respective pixel location, and IR(rx,cx) is the infrared value for the respective pixel location, and where Rx is an output value of said first pixel of a respective sub-array, Gx is an output value of said second pixel of a respective sub-array, Bx is an output value of said third pixel of a respective sub-array and IRx is an output value of said fourth pixel of a respective sub-array.

6. The imaging system of claim 1, wherein said first pixel primarily senses color in a red spectral band, said second pixel primarily senses color in a green spectral band, and said third pixel primarily senses color in a blue spectral band.

7. The imaging system of claim 1, wherein said processor is operable to generate an output indicative of either a color image or a monochromatic image.

8. The imaging system of claim 7, wherein said processor generates said output indicative of a color image in response to an ambient light level being greater than a threshold level, and wherein said processor generates said output indicative of a monochromatic image in response to an ambient light level being lower than a threshold level.

9. The imaging system of claim 1, wherein said imaging system is associated with at least one of an automatic headlamp control system, a lane departure warning system, an object detection system, a night vision system, a rain sensing system, and a reverse aid system.

10. The imaging system of claim 1 further comprising a near infrared illumination source for emitting near infrared energy at the scene imaged by said imaging device.

11. The imaging system of claim 1, wherein said imaging system provides infrared information to enhance identification accuracy of traffic signs.

12. The imaging system of claim 11, wherein said imaging system is for an intelligent headlamp control system.

13. The imaging system of claim 12, wherein said imaging system provides a pulse width modulated infrared headlamp output to enhance identification accuracy of traffic signs for said intelligent headlamp control system.

14. The imaging system of claim 1, wherein said fourth pixel senses primarily near infrared radiation.

15. The imaging system of claim 1 further comprising a display for displaying images responsive to said image processor.