Vehicle vision system with reduction of temporal noise in images

• US 10,326,969 B2
• Filed: 08/11/2014
• Issued: 06/18/2019
• Est. Priority Date: 08/12/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Wierich, Thomas
• Primary Examiner(s): Beasley, Deirdre L
• Application Number:
  US14/456,162
• Publication Number:
  US 20150042806A1
• Time in Patent Office: 1,772 Days
• Field of Search: 348148
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/677 for reducing the column or line fixed pattern noise
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

A vision system of a vehicle includes a camera disposed at a vehicle and having a field of view exterior of the vehicle. The camera includes an imaging array having rows and columns of photosensing elements that may include red light sensing elements, green light sensing elements and blue light sensing elements. An image processor is operable to process image data captured by the camera. A display is disposed in the vehicle and viewable by a driver of the vehicle and is operable to display images derived from captured image data. Responsive to image processing of captured image data by the image processor, temporal noise in images derived from captured image data is reduced by determining a change in luminance of photosensing elements from a first frame of captured image data to a second frame of captured image data.

Claims

1. A vision system of a vehicle, said vision system comprising:

a camera disposed at a vehicle and having a field of view exterior of the vehicle;

wherein said camera comprises an imaging array having rows and columns of photosensing elements;

an image processor operable to process image data captured by said camera;

a display disposed in the vehicle and viewable by a driver of the vehicle, wherein said display is operable to display images derived from captured image data;

wherein, responsive to image processing of captured image data by said image processor, temporal noise in displayed images derived from captured image data is reduced by determining a change in luminance values of photosensing elements from a first frame of captured image data to a second frame of captured image data and adjusting luminance values of photosensing elements for the displayed derived image responsive, at least in part, to the determined change in luminance of photosensing elements being above or below a threshold change;

wherein, responsive to a determined change in a luminance value of a photosensing element being less than the threshold change from the first frame of captured image data to the second frame of captured image data, the change in luminance value for that photosensing element is dampened so as to not fully adopt the luminance value of that photosensing element of the second frame of captured image data for the displayed images; and

wherein the threshold change is adjusted responsive to a determined ambient light level at the vehicle.

2. The vision system of claim 1, wherein said imaging array comprises red photosensing elements, green photosensing elements and blue photosensing elements, and wherein a change in luminance values is determined for groups of red, green and blue photosensing elements from a first frame of captured image data to a second frame of captured image data.

3. The vision system of claim 1, wherein, responsive to the determined change in luminance values of photosensing elements being greater than the threshold change, the change in luminance values is dampened via a first weighting ratio and, responsive to the determined change in luminance values of photosensing elements being less than the threshold change, the change in luminance values is dampened via a second weighting ratio, and wherein said first weighting ratio has a greater weighting of luminance values of said second frame of captured image data than said second weighting ratio.

4. The vision system of claim 3, wherein said first weighting ratio is about 3:

7 and said second weighting ratio is about 9;

1.

5. The vision system of claim 3, wherein said first and second weighting ratios are selected to ensure that a substantial change in luminance of photosensing elements is represented in the displayed derived image and a small change in luminance of photosensing elements is not substantially represented in the displayed derived image.

6. The vision system of claim 1, wherein said vision system addresses temporal noise present randomly in captured image data.

7. The vision system of claim 1, wherein an algorithm addresses fix pattern noise present statically in captured image data.

8. The vision system of claim 7, wherein said algorithm (i) compares common color values of one column of photosensing elements with diagonal neighboring photosensing elements of the current frame by subtracting from another in pairs, and (ii) sums and divides by the total number of lines to determine an average luminance difference, and wherein the average luminance difference is used to determine and reduce fix pattern noise.

9. The vision system of claim 8, wherein the fix pattern noise is reduced by subtracting the average luminance difference from individual photosensing elements in said column of photosensing elements.

10. The vision system of claim 1, wherein a parameter set of said vision system adaptively changes in different light conditions.

11. The vision system of claim 1, wherein a parameter set of said vision system adaptively changes in different signal to noise ratio conditions of said imaging array.

12. The vision system of claim 11, wherein said parameter set of said vision system comprises a first weighted ratio applied responsive to the determined change in luminance values of photosensing elements being greater than the threshold change and a second weighted ratio applied responsive to the determined change in luminance of photosensing elements being less than the threshold change.

13. The vision system of claim 1, wherein said image processor operates to reduce temporal noise during lower lighting conditions and does not operate to reduce temporal noise during higher lighting conditions.

14. The vision system of claim 1, wherein said image processor operates to reduce temporal noise in a portion of the derived image while not reducing temporal noise in other portions of the derived image.

15. The vision system of claim 14, wherein said portion of the derived image comprises a lower light level region and said other portions of the derived image comprise higher light level regions.

16. A vision system of a vehicle, said vision system comprising:

a camera disposed at a vehicle and having a field of view exterior of the vehicle;

wherein said camera comprises an imaging array having rows and columns of photosensing elements, and wherein said imaging array comprises red photosensing elements, green photosensing elements and blue photosensing elements;

an image processor operable to process image data captured by said camera;

a display disposed in the vehicle and viewable by a driver of the vehicle, wherein said display is operable to display images derived from captured image data;

wherein, responsive to image processing of captured image data by said image processor, temporal noise in displayed images derived from captured image data is reduced by determining a change in luminance of individual red, green and blue photosensing elements from a first frame of captured image data to a second frame of captured image data and adjusting luminance values of red, green and blue photosensing elements for the displayed derived image responsive, at least in part, to the determined change in luminance values of individual red, green and blue photosensing elements being above or below a threshold change;

wherein, responsive to a determined change in a luminance values of individual photosensing elements being less than the threshold change from the first frame of captured image data to the second frame of captured image data, the change in luminance values for those photosensing elements is dampened via a first dampening ratio so as to not fully adopt the luminance value of the those photosensing elements of the second frame of captured image data for the displayed images;

wherein, responsive to the determined change in luminance values of individual photosensing elements being greater than the threshold change, the change in luminance values for those photosensing elements is dampened via a second dampening ratio;

wherein the first dampening ratio provides a greater dampening of luminance values of said second frame of captured image data than the second dampening ratio;

wherein the first and second dampening ratios are selected to ensure that a substantial change in luminance values of individual red, green and blue photosensing elements is represented in the displayed derived image and a small change in luminance of individual red, green and blue photosensing elements is not substantially represented in the displayed derived image; and

wherein the threshold change is adjusted responsive to a determined ambient light level at the vehicle.

17. The vision system of claim 16, wherein the first and second dampening ratios are adjusted responsive to at least one of (i) changes in lighting conditions in the field of view of said camera and (ii) changes in signal to noise ratio conditions of said imaging array.

18. The vision system of claim 16, wherein said image processor operates to reduce temporal noise in a portion of the derived image while not reducing temporal noise in other portions of the derived image, and wherein said portion of the derived image comprises a lower light level region and said other portions of the derived image comprise higher light level regions.

19. A vision system of a vehicle, said vision system comprising:

a camera disposed at a vehicle and having a field of view exterior of the vehicle;

wherein said camera comprises an imaging array having rows and columns of photosensing elements and wherein said imaging array comprises red photosensing elements, green photosensing elements and blue photosensing elements;

an image processor operable to process image data captured by said camera;

a display disposed in the vehicle and viewable by a driver of the vehicle, wherein said display is operable to display images derived from captured image data;

wherein, responsive to image processing of captured image data by said image processor, temporal noise in displayed images derived from captured image data is reduced by determining a change in luminance values of photosensing elements from a first frame of captured image data to a second frame of captured image data and adjusting luminance values of photosensing elements for the displayed derived image responsive, at least in part, to the determined change in luminance values of photosensing elements being above or below a threshold change;

wherein luminance values of photosensing elements for the displayed derived image are adjusted by adjusting a dampening ratio of luminance values in said first frame of captured image data to luminance values in said second frame of captured image data;

wherein, responsive to a determined change in a luminance values of individual photosensing elements being less than the threshold change from the first frame of captured image data to the second frame of captured image data, the change in luminance values for those photosensing elements is dampened via a first dampening ratio so as to not fully adopt the luminance value of the those photosensing elements of the second frame of captured image data for the displayed images;

wherein, responsive to the determined change in luminance values of individual photosensing elements being greater than the threshold change, the change in luminance values for those photosensing elements is dampened via a second dampening ratio, and wherein the first dampening ratio provides a greater dampening of luminance values of said second frame of captured image data than the second dampening ratio;

wherein an algorithm addresses fix pattern noise present statically in captured image data;

wherein said algorithm (i) compares luminance values of green photosensing elements in one column of photosensing elements with diagonal neighboring green photosensing elements in another column of photosensing elements of the current frame by subtracting from another in pairs, and (ii) sums and divides by the total number of lines to determine an average change in luminance, and wherein the average change in luminance is used to determine and reduce fix pattern noise; and

wherein the threshold change is adjusted responsive to a determined ambient light level at the vehicle.

20. The vision system of claim 19, wherein the fix pattern noise is reduced by subtracting the average luminance difference from photosensing elements in said column of photosensing elements.