Detection, identification, and mitigation of lens contamination for vehicle mounted camera systems

• US 10,013,616 B2
• Filed: 05/27/2015
• Issued: 07/03/2018
• Est. Priority Date: 05/27/2014
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH, Robert Bosch LLC (Robert Bosch GmbH)
• Original Assignee: Robert Bosch GmbH
• Inventors: Gehrke, Mark
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/311,237
• Publication Number:
  US 20170109590A1
• Time in Patent Office: 1,133 Days
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Systems and methods are presented for operating a vehicle camera system to detect, identify, and mitigate camera lens contamination. An image is received from a camera mounted on the vehicle and one or more metrics is calculated based on the received image. The system determines whether a lens of the camera is contaminated based on the one or more calculated metrics and, if so, determines a type of contamination. A specific mitigation routine is selected form a plurality of mitigation routines based on the determined type of contamination and is applied to the received image to create an enhanced image. The enhanced image is analyzed to determine whether the contamination is acceptably mitigated after application of the selected mitigation routine and a fault condition signal is output when the contamination is not acceptably mitigated.

Claims

1. A method for operating a vehicle camera system, the method comprising:

receiving an image from a camera mounted on a vehicle;

calculating one or more metrics based on the received image;

determining whether a lens of the camera is contaminated based on the one or more calculated metrics;

determining a type of contamination;

selecting one of a plurality of mitigation routines to enhance the image based on the determined type of contamination;

applying the selected mitigation routine to create an enhanced image;

analyzing the enhanced image to determine whether the contamination is acceptably mitigated after application of the selected mitigation routine; and

outputting a fault condition signal when the contamination is not acceptably mitigated after application of the selected mitigation routine.

2. The method of claim 1, further comprising dividing the received image into a plurality of sub-regions, and wherein determining whether the lens of the camera is contaminated and determining a type of contamination is performed for each of the plurality of sub-regions.

3. The method of claim 2, wherein outputting a fault condition signal includes outputting a fault condition signal identifying one or more specific sub-regions in which the contamination is not acceptably mitigated after application of the selected mitigation routine.

4. The method of claim 1, further comprising:

displaying the enhanced image on a display mounted in an interior of the vehicle when the fault condition signal is not output; and

displaying a notification on the display indicating that the image received from the camera is degraded when the fault condition signal is output.

5. The method of claim 4, further comprising preventing the display from showing the received image and the enhanced image when the fault condition signal is output.

6. The method of claim 1, further comprising providing the received image or the enhanced image to a vehicle assistance system, wherein the vehicle assistance system operates based on a camera image and wherein the vehicle assistance system is at least partially disabled when the fault condition signal is output.

7. The method of claim 6, further comprising displaying a notification on the display indicating that the vehicle assistance system is at least partially disabled when the fault condition signal is output.

8. The method of claim 6, wherein the vehicle assistance system includes at least one selected from a group consisting of an automated parking assist system and a lane detection system.

9. The method of claim 6, wherein some functionality of the vehicle assistance system is limited and other functionality of the vehicle assistance system is made available when the fault condition signal is output depending on a degree of image degradation indicated by one or more metrics calculated based on the enhanced image.

10. The method of claim 6, further comprising dividing the received image into a plurality of sub-regions,

wherein analyzing the enhanced image to determine whether the contamination is acceptably mitigated after application of the selected mitigation routine includes identifying any individual sub-regions where the contamination is not acceptably mitigated,

wherein outputting the fault condition signal includes transmitting a message to the vehicle assistance system identifying any individual sub-regions in which the contamination is not acceptably mitigated,

wherein the vehicle assistance system is configured to operate based on image data from only a sub-set of the plurality of sub-regions, and

wherein the vehicle assistance system is disabled in response to the output fault condition signal only when the output fault condition signal identifies one or more of the sub-regions from the sub-set of sub-regions as being contaminated and not acceptably mitigated.

11. The method of claim 1,

wherein calculating one or more metrics based on the received image includes:

comparing the received image to at least one previously received image from the camera, and

calculating pixel variation in a plurality of individual pixels from the received images, and

wherein determining the type of contamination includes;

determining that the contamination is condensation when the pixel variation for one or more of the plurality of individual pixels is greater than zero and less than a first threshold.

12. The method of claim 11, wherein determining the type of contamination includes determining that the contamination is physical debris on the lens of the camera when the pixel variation of one or more of the plurality of individual pixels is approximately zero.

13. The method of claim 11, wherein, when the contamination type is determined to be condensation, applying the selected mitigation routine includes:

determining a mask pattern indicative of image distortion based on a plurality of receiving camera images, and

subtracting the mask patent from the received image from the camera.

14. The method of claim 11, wherein, when the contamination type is determined to be condensation, analyzing the enhanced image includes:

comparing the enhanced image to at least one previously enhanced image from the camera,

calculating pixel variation in a plurality of individual pixels from the enhanced images, and

determining that the contamination is not acceptably mitigated after application of the selected mitigation routine when the calculated pixel variation from the enhanced images is less than a second threshold.

15. The method of claim 14, wherein the first threshold is equal to the second threshold.

16. The method of claim 1, further comprising:

determining a date and a time when the enhanced image is determined to be not acceptably mitigated; and

storing the date and the time to a memory.