Combined RGB and IR imaging sensor

• US 8,446,470 B2
• Filed: 10/03/2008
• Issued: 05/21/2013
• Est. Priority Date: 10/04/2007
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Lu, Yuesheng, Higgins-Luthman, Michael J., Livengood, William R., Harris, James Daniel
• Primary Examiner(s): NGUYEN, PHUOC H
• Application Number:
  US12/681,204
• Publication Number:
  US 20100289885A1
• Time in Patent Office: 1,691 Days
• Field of Search: 348/61, 348/148, 348/345
• US Class Current: 348/148
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

An imaging system suitable for use in a vehicle includes an imaging sensor having a two-dimensional array of photosensing elements, an image processor for processing image data captured by the imaging sensor, and a display for displaying images responsive to the image processor. The array has a plurality of sub-arrays, with each of the sub-arrays having a first pixel sensing primarily a first color, a second pixel sensing primarily a second color, a third pixel sensing primarily a third color and a fourth pixel sensing primarily infrared radiation. The image processor processes the output of the fourth pixel to determine an infrared component of the imaged scene and subtracts the infrared component from the outputs of the first, second and third pixels to obtain a more accurate or true color response for the pixels and to limit or avoid infrared color wash-out.

Claims

1. An imaging system suitable for use in a vehicle, said imaging system comprising:

an imaging sensor comprising a two-dimensional array of photosensing elements, said two-dimensional array comprising a plurality of sub-arrays, wherein each sub-array of said sub-arrays comprises four photosensing pixels, and wherein a first pixel of said sub-array primarily senses infrared radiation and a second pixel of said sub-array primarily senses red visible light;

wherein said imaging sensor is configured to be disposed at a vehicle equipped with said imaging system and wherein, when said imaging sensor is disposed at the equipped vehicle, said imaging sensor has a field of view forwardly of the equipped vehicle through a windshield of the equipped vehicle;

an image processor for processing image data captured by said imaging sensor, wherein said image processor processes outputs of said four photosensing pixels of said sub-array, said image processor processing an output of said first pixel of said sub-array to determine an infrared component of the imaged scene and said image processor processing an output of said second pixel of said sub-array to determine a red visible light component of the imaged scene;

wherein, responsive at least in part to determination of a red visible light component of the imaged scene via said processing of the output of said second pixel of said sub-array and with said imaging sensor disposed at the equipped vehicle and viewing forwardly through the windshield, said imaging system is operable to detect at least one of (i) a headlight of a vehicle ahead of the equipped vehicle (ii) a taillight of a vehicle ahead of the equipped vehicle and (iii) a traffic sign;

wherein a near infrared illumination source is disposed at the equipped vehicle and is operable to emit near infrared illumination to illuminate forward of the equipped vehicle to enhance nighttime viewing by said imaging sensor, and wherein said near infrared illumination source, when activated, illuminates at least a portion of the forward field of view of said imaging sensor with near infrared illumination; and

wherein, responsive at least in part to determination of an infrared component of the imaged scene via said processing of the output of said first pixel of said sub-array and with said imaging sensor disposed at the equipped vehicle and viewing forwardly through the windshield, said imaging system is operable to detect reflections by objects located forward of the equipped vehicle of near infrared illumination emitted by said near infrared illumination source of the equipped vehicle.

2. The imaging system of claim 1, wherein said image processor processes image data via a de-mosaicing technique.

3. The imaging system of claim 2, wherein a third pixel of said sub-array primarily senses a third color and a fourth pixel of said sub-array primarily senses a fourth color, and wherein said image processor processes image data to determine the color values at each pixel location of said sub-array via the following equations:

R(r0c0)=R1−

A_R*IR1,G(r0c0)=G1−

A_G*IR1,B(r0c0)=B1−

A_B*IR1, and IR(r0c0)=IR1;

R(r0c1)=R1−

A_R*IR1,G(r0c1)=G1−

A_G*IR1,B(r0c1)=B1−

A_B*IR1, and IR(r0c1)=IR1;

R(r1c0)=R1−

A_R*IR1,G(r1c0)=G1−

A_G*IR1,B(r1c0)=B1−

A_B*IR1, and IR(r1c0)=IR1;

R(r1c1)=R1−

A_R*IR1,G(r1c1)=G1−

A_G*IR1,B(r1c1)=B1−

A_B*IR1, and IR(r1c1)=IR1,

where A_R, A_G, and A_B are the selected factors for said second pixel, said third pixel and said fourth pixel, respectively, and where R(rx,cx) is the red color value for the respective pixel location, G(rx,cx) is the third color value for the respective pixel location, B(rx,cx) is the fourth color value for the respective pixel location, and IR(rx,cx) is the infrared value for the respective pixel location, and where R1 is an output value of said second pixel, G1 is an output value of said third pixel, B1 is an output value of said fourth pixel and IR1 is an output value of said fourth first pixel.

4. The imaging system of claim 2, wherein a third pixel of said sub-array senses primarily a third color and a fourth pixel of said sub-array senses primarily a fourth color, and wherein said image processor processes image data from multiple sub-arrays to determine the color values at each pixel location of said sub-arrays via the following equations:

R(r0c0)=R1−

A_R*IR1,G(r0c0)=G1−

A_G*IR1,B(r0c0)=B1−

A_B*IR1, and IR(r0c0)=IR1;

R(r0c1)=R2−

A_R*IR1,G(r0c1)=G1−

A_G*IR1,B(r0c1)=B2−

A_B*IR1, and IR(r0c1)=IR1;

R(r1c0)=R3−

A_R*IR1,G(r1c0)=G3−

A_G*IR1,B(r1c0)=B1−

A_B*IR1, and IR(r1c0)=IR1;

R(r1c1)=R4−

A_R*IR1,G(r1c1)=G3−

A_G*IR1,B(r1c1)=B2−

A_B*IR1, and IR(r1c1)=IR1,

where A_R, A_G, and A_B are the selected factors for said second pixel, said third pixel and said fourth pixel, respectively, and where R(rx,cx) is the red color value for the respective pixel location, G(rx,cx) is the third color value for the respective pixel location, B(rx,cx) is the fourth color value for the respective pixel location, and IR(rx,cx) is the infrared value for the respective pixel location, and where Rx is an output value of said second pixel of a respective sub-array, Gx is an output value of said third pixel of a respective sub-array, Bx is an output value of said fourth pixel of a respective sub-array and IRx is an output value of said first pixel of a respective sub-array.

5. The imaging system of claim 2, wherein a third pixel of said sub-array senses primarily a third color and a fourth pixel of said sub-array senses primarily a fourth color, and wherein said image processor processes image data from multiple sub-arrays to determine the color values at each pixel location of said sub-arrays via the following equations:

R(r1c1)=(R1+R2+R3+R4)/4−

A_R*IR1,G(r1c1)=(G1+G3)/2−

A_G*IR1,B(r1c1)=(B1+B2)/2−

A_B*IR1, and IR(r0c0)=IR1;

R(r1c2)=(R2+R4)/2−

A_R*(IR1+IR2)/2,G(r1c2)=(G1+G2+G3+G4)/4−

A_G*(IR1+IR2)/2,B(r0c1)=B2−

A_B*(IR1+IR2)/2, and IR(r0c1)=(IR1+IR2)/2;

R(r2c1)=(R3+R4)/2−

A_R*(IR1+IR3)/2,G(r2c1)=G3−

A_G*(IR1+IR3)/2,B(r2c1)=(B1+B2+B3+B4)/4−

A_B*(IR1+IR3)/2, and IR(r2c1)=(IR1+IR3)/2;

R(r2c2)=R4−

A_R*(IR1+IR2+IR3+IR4)/4,G(r1c1)=(G3+G4)/2−

A_G*(IR1+IR2+IR3+IR4)/4,B(r2c2)=(B2+B4)/2−

A_B*(IR1+IR2+IR3+IR4)/4, and IR(r2c2)=(IR1+IR2+IR3+IR4)/4,

where A_R, A_G, and A_B are the selected factors for said second pixel, said third pixel and said fourth pixel, respectively, and where R(rx,cx) is the red color value for the respective pixel location, G(rx,cx) is the third color value for the respective pixel location, B(rx,cx) is the fourth color value for the respective pixel location, and IR(rx,cx) is the infrared value for the respective pixel location, and where Rx is an output value of said second pixel of a respective sub-array, Gx is an output value of said third pixel of a respective sub-array, Bx is an output value of said fourth pixel of a respective sub-array and IRx is an output value of said first pixel of a respective sub-array.

6. The imaging system of claim 1, wherein a third pixel of said sub-array primarily senses color in a green spectral band, and a fourth pixel of said sub-array primarily senses color in a blue spectral band.

7. The imaging system of claim 1, wherein said image processor is operable to generate an output indicative of either a color image or a monochromatic image.

8. The imaging system of claim 7, wherein said image processor generates said output indicative of a color image in response to an ambient light level being greater than a threshold level, and wherein said image processor generates said output indicative of a monochromatic image in response to an ambient light level being lower than a threshold level.

9. The imaging system of claim 1, wherein said imaging system is associated with at least one of an automatic headlamp control system, a lane departure warning system, an object detection system, a night vision system, a rain sensing system, and a reverse aid system.

10. The imaging system of claim 1, wherein said near infrared illumination source comprises at least one of (i) a near infrared light-emitting light emitting diode and (ii) a near infrared light-emitting laser.

11. The imaging system of claim 1, wherein, responsive at least in part to determination of a red visible light component of the imaged scene via said processing of the output of said second pixel of said sub-array and with said imaging sensor disposed at the equipped vehicle and viewing forwardly through the windshield, said imaging system is operable to detect traffic signs.

12. The imaging system of claim 1, wherein said imaging system is associated with an intelligent headlamp control system.

13. The imaging system of claim 12, wherein said imaging system utilizes a pulse width modulated infrared output to enhance identification accuracy of objects.

14. The imaging system of claim 1, wherein said third of said sub-array primarily senses blue visible light and said fourth pixel primarily senses green visible light.

15. The imaging system of claim 1 further comprising a display for displaying images, and wherein the displayed images are at least in part derived from image data output by said image processor.

16. The imaging system of claim 1, wherein said image processor subtracts the infrared component from said outputs of the other pixels of said sub-array to obtain a more accurate or true color response for the pixels and to limit or avoid infrared color wash-out.