Dynamic demosaicing of camera pixels

• US 10,567,724 B2
• Filed: 04/10/2018
• Issued: 02/18/2020
• Est. Priority Date: 04/10/2018
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Patel, Shiv G., Murad, Mohannad, Machak, Joseph G., Huntzicker, Fred W.
• Primary Examiner(s): An, Shawn S
• Application Number:
  US15/949,261
• Publication Number:
  US 20190313073A1
• Time in Patent Office: 679 Days
• CPC Class Codes: B60R 21/01538 for image processing, e.g. cameras or sensor arrays
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 9/78 for separating the brightness signal or the chrominance signal from the colour television signal, e.g. using comb filter

Abstract

A method and system to perform dynamic demosaicing for a camera in a vehicle involve an array of image sensors of the camera to obtain light intensity values. Each image sensor of the array of image sensors represents a pixel. An array of filters of the camera overlays the array of image sensors and restricts a wavelength range for which the one image sensor of the array of image sensors obtains the light intensity value. The array of filters includes at least two different types of filters corresponding with two different wavelength ranges. A processor estimates a current state associated with the vehicle and selects a demosaicing algorithm based on the current state such that, for each pixel, the demosaicing algorithm facilitates an estimate of the light intensity value at a different wavelength range than the wavelength range for which the corresponding image sensor obtained the light intensity value.

Claims

1. A system to perform dynamic demosaicing for a camera in a vehicle, the system comprising:

an array of image sensors of the camera configured to obtain light intensity values, wherein each image sensor of the array of image sensors represents a pixel;

an array of filters of the camera configured to overlay the array of image sensors such that each filter of the array of filters corresponds with one image sensor of the array of image sensors and restricts a wavelength range for which the one image sensor of the array of image sensors obtains the light intensity value, wherein the array of filters includes at least two different types of filters corresponding with two different wavelength ranges; and

a processor configured to estimate a current state associated with the vehicle and to select a demosaicing algorithm based on the current state associated with the vehicle such that, for each pixel, the demosaicing algorithm facilitates an estimate of the light intensity value at a different wavelength range than the wavelength range for which the corresponding image sensor obtained the light intensity value, wherein the processor is configured to select the demosaicing algorithm according to a rule-based algorithm that matches the current state associated with the vehicle with the demosaicing algorithm, the at least two different types of filters include a red filter, a green filter, and a blue filter corresponding, respectively, with wavelength ranges of red light, green light, and blue light, and a monochrome filter corresponding with a larger wavelength range than the wavelength ranges of the red light, the green light, and the blue light, and the demosaicing algorithm, based on the current state being ambient light below a threshold level, adds a weighted sum of one or more light intensity values measured by one or more image sensors of the array of image sensors with the monochrome filter to an estimate of the red light, the green light, and the blue light at every pixel.

2. The system according to claim 1, wherein the demosaicing algorithm, based on the current state being an approach to a traffic light, uses the light intensity level measured by one or more image sensors of the array of image sensors with the monochrome filter as an estimate of the blue light at every pixel.

3. The system according to claim 1, wherein the demosaicing algorithm, based on the current state being a lane sensing state, uses the light intensity level measured by one or more image sensors of the array of image sensors with the monochrome filter as an estimate of the red light, the green light, and the blue light at every pixel.

4. The system according to claim 1, wherein the demosaicing algorithm, based on the current state being a forward braking event, uses the light intensity level measured by one or more image sensors of the array of image sensors with the monochrome filter as an estimate of the green light and the blue light at every pixel.

5. The system according to claim 1, wherein the processor is configured to estimate the current state using a radar system, a lidar system, or an ambient light detector.

6. A method of performing dynamic demosaicing for a camera in a vehicle, the method comprising:

obtaining light intensity values from an array of image sensors of the camera, each image sensor of the array of image sensors representing a pixel and each image sensor of the array of image sensors having a filter of an array of filters overlaid such that the filter of the array of filters restricts a wavelength range for which the image sensor of the array of image sensors obtains the light intensity value, wherein the array of filters includes at least two different types of filters corresponding with two different wavelength ranges;

estimating, using a processor, a current state associated with the vehicle; and

selecting, using the processor, a demosaicing algorithm based on the current state associated with the vehicle such that, for each pixel, the demosaicing algorithm facilitates an estimate of the light intensity value at a different wavelength range than the wavelength range for which the corresponding image sensor obtained the light intensity value, wherein the selecting the demosaicing algorithm is according to a rule-based algorithm that matches the current state associated with the vehicle with the demosaicing algorithm, the obtaining the light intensity values includes obtaining red light, green light, blue light, and monochromatic light based on the at least two different types of filters including a red filter, a green filter, and a blue filter corresponding, respectively, with wavelength ranges of the red light, the green light, and the blue light, and a monochrome filter corresponding with the monochromatic light with a larger wavelength range than the wavelength ranges of the red light, the green light, and the blue light, and, based on the estimating the current state indicating an ambient light level below a threshold level, the selecting the demosaicing algorithm includes selecting the demosaicing algorithm that adds a weighted sum of one or more light intensity values measured by one or more image sensors of the array of image sensors with the monochrome filter to an estimate of the red color, the green color, and the blue color at every pixel.

7. The method according to claim 6, wherein based on the estimating the current state indicating an approach to a traffic light, the selecting the demosaicing algorithm includes selecting the demosaicing algorithm that uses the light intensity level measured by one or more image sensors of the array of image sensors with the monochrome filter as an estimate of the blue light at every pixel.

8. The method according to claim 6, wherein based on the estimating the current state indicating a lane sensing state, the selecting the demosaicing algorithm includes selecting the demosaicing algorithm that uses the light intensity level measured by one or more image sensors of the array of image sensors with the monochrome filter as an estimate of the red light, the green light, and the blue light at every pixel.

9. The method according to claim 6, wherein based on the estimating the current state indicating a forward braking event, the selecting the demosaicing algorithm includes selecting the demosaicing algorithm that uses the light intensity level measured by one or more image sensors of the array of image sensors with the monochrome filter as an estimate of the green light and the blue light at every pixel.

10. The method according to claim 6, wherein the estimating the current state is based on information from a radar system, a lidar system, or an ambient light detector.

11. A non-transitory computer readable medium storing instructions that, when processed by processing circuitry, cause the processing circuitry to implement a method of performing dynamic demosaicing for a camera in a vehicle, the method comprising:

obtaining light intensity values from an array of image sensors of the camera, each image sensor of the array of image sensors representing a pixel and each image sensor of the array of image sensors having a filter of an array of filters overlaid such that the filter of the array of filters restricts a wavelength range for which the image sensor of the array of image sensors obtains the light intensity value, wherein the array of filters includes at least two different types of filters corresponding with two different wavelength ranges;

estimating a current state associated with the vehicle; and

selecting a demosaicing algorithm based on the current state associated with the vehicle such that, for each pixel, the demosaicing algorithm facilitates an estimate of the light intensity value at a different wavelength range than the wavelength range for which the corresponding image sensor obtained the light intensity value, wherein the selecting the demosaicing algorithm is according to a rule-based algorithm that matches the current state associated with the vehicle with the demosaicing algorithm, the obtaining the light intensity values includes obtaining red light, green light, blue light, and monochromatic light based on the at least two different types of filters including a red filter, a green filter, and a blue filter corresponding, respectively, with wavelength ranges of the red light, the green light, and the blue light, and a monochrome filter corresponding with the monochromatic light with a larger wavelength range than the wavelength ranges of the red light, the green light, and the blue light, and, based on the estimating the current state indicating an ambient light level below a threshold level, the selecting the demosaicing algorithm includes selecting the demosaicing algorithm that adds a weighted sum of one or more light intensity values measured by one or more image sensors of the array of image sensors with the monochrome filter to an estimate of the red color, the green color, and the blue color at every pixel.

12. The non-transitory computer readable medium according to claim 11, wherein the estimating the current state is based on information from a radar system, a lidar system, or an ambient light detector.