Vehicle sensor cleaning

• US 10,829,091 B2
• Filed: 09/20/2018
• Issued: 11/10/2020
• Est. Priority Date: 09/20/2018
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Herman, David Michael, Patil, Sunil, Krishnan, Venkatesh
• Primary Examiner(s): Marc-Coleman, Marthe Y
• Application Number:
  US16/136,629
• Publication Number:
  US 20200094784A1
• Time in Patent Office: 782 Days
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B60S 1/0844 including a camera
• CPC Class Codes: B60S 1/0848 Cleaning devices for cameras on vehicle
• CPC Class Codes: B60S 1/56 specially adapted for cleaning other parts or devices than front windows or windscreens rear-view mirror arrangements mounted on vehicle exterior including cleaning devices B60R1/0602
• CPC Class Codes: G02B 27/0006 with means to keep optical surfaces clean, e.g. by preventing or removing dirt, stains, contamination, condensation G02B1/18 takes precedence; cleaning in general B08B
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 23/811 by dust removal, e.g. from surfaces of the image sensor or processing of the image signal output by the electronic image sensor

Abstract

A system includes a processor and a memory. The memory stores instructions executable by the processor to detect an occlusion on a surface in a vehicle sensor optical path based on a segmentation of sensor image data, and to select a cleaning plan for the surface based on the detected occlusion, map data, and vehicle route data.

Claims

1. A system, comprising a processor and a memory, the memory storing instructions executable by the processor to:

detect an occlusion at one or more points on a surface in a vehicle sensor optical path based on a segmentation of sensor image data that classifies objects in the vehicle sensor optical path to one of a plurality of segmentation classes, each of the plurality of segmentation classes classifying a type of object in the vehicle sensor optical path;

determine a priority of the one or more points based on the detected occlusion, the segmentation classes identified in the sensor image data, map data including geo-coordinates of a location of a vehicle, and vehicle route data of a route on which the vehicle is navigating, the priority being an importance of data collected at the one or more points in the vehicle sensor optical path to operate the vehicle; and

select a cleaning plan for the surface based on the segmentation class of the detected occlusion and the priority of the one or more points, the cleaning plan including a specified actuation of each of a wiper and a fluid sprayer and a time duration of the specified actuations.

2. The system of claim 1, wherein the instructions further include instructions to perform the segmentation of the sensor image data based on output of a neural network trained to detect a plurality of feature classes including at least an occlusion class.

3. The system of claim 2, wherein the plurality of feature classes further includes at least one of a road, a pedestrian, a car, a building, a vegetation, and a sky class.

4. The system of claim 1, wherein the surface includes at least one of a lens, a transparent cover, and a part of a vehicle windshield.

5. The system of claim 1, wherein the instructions further include instructions to

determine the segmentation class of a location including one or more points on an exterior of the surface based on the sensor field of view, the vehicle route data, and the map data.

6. The system of claim 1, wherein the selected cleaning plan further includes a time of actuation of a vehicle cleaning actuator and a predicted duration of cleaning.

7. The system of claim 6, wherein the instructions further include instructions to:

determine a location of the occlusion on an exterior of the surface;

predict a change of the priority of the location on the exterior surface based on the vehicle route; and

plan the time of cleaning based on the predicted change of the priority and a vehicle speed.

8. The system of claim 7, wherein the instructions further include instructions to select a priority threshold for cleaning based on the vehicle speed;

and

upon determining that the priority of the location on an exterior of the sensor exceeds the selected priority threshold, actuate the vehicle cleaning actuator to clean the exterior of the surface.

9. The system of claim 1, wherein the instructions further include instructions to:

determine a cleaning score based on a priority of a location of the occlusion, a vehicle speed, a number of occluded pixels in the image data, and a predicted duration of a cleaning; and

select the cleaning plan based on the determined score.

10. The system of claim 1, wherein the instructions further include instructions to perform the selected cleaning plan.

11. The system of claim 10, wherein the selected cleaning plan further includes at least one of a time of actuating a cleaning actuator, a duration of actuation, and a mode of operating the cleaning actuator.

12. The system of claim 11, wherein the instructions to perform the selected cleaning plan further include instructions to activate the cleaning actuator at the time of actuating the cleaning actuator for the duration of actuation.

13. The system of claim 11, wherein the instructions further include instructions to:

determine normalized scores from start to end of a vehicle route;

determine a time of maximum normalized score; and

determine the time of actuating the cleaning actuator based on the time of a maximum normalized score.

14. A system, comprising:

a vehicle camera sensor having an optical path; and

a processor programmed to;

detect an occlusion at one or more points on a surface in the optical path of the vehicle camera sensor based on a segmentation of sensor image data that classifies objects in the vehicle sensor optical path to one of a plurality of segmentation classes, each of the plurality of segmentation classes classifying a type of object in the vehicle sensor optical path;

determine a priority of the one or more points based on the detected occlusion, the segmentation classes identified in the sensor image data, map data including geo-coordinates of a location of a vehicle, and vehicle route data of a route on which the vehicle is navigating, the priority being an importance of data collected at the one or more points in the vehicle sensor optical path to operate the vehicle; and

select a cleaning plan for the surface based on the segmentation class of the detected occlusion and the priority of the one or more points, the cleaning plan including a specified actuation of each of a wiper and a fluid sprayer and a time duration of the specified actuations.

15. The system of claim 14, wherein the processor is further programmed to perform the segmentation of the sensor image data based on output of a neural network trained to detect a plurality of feature classes including at least an occlusion class.

16. The system of claim 14, wherein the processor is further programmed to

determine a predicted segmentation class of a location including one or more points on an exterior of the surface based on the sensor field of view, the vehicle route data, and the map data.

17. A method, comprising:

detecting an occlusion at one or more points on a surface in the optical path of a vehicle camera sensor based on a segmentation of sensor image data that classifies objects in the vehicle sensor optical path to one of a plurality of segmentation classes, each of the plurality of segmentation classes classifying a type of object in the vehicle sensor optical path;

determining a priority of the one or more points based on the detected occlusion, the segmentation classes identified in the sensor image data, map data including geo-coordinates of a location of a vehicle, and vehicle route data of a route on which the vehicle is navigating, the priority being an importance of data collected at the one or more points in the vehicle sensor optical path to operate the vehicle; and

selecting a cleaning plan for the surface based on the segmentation class of the detected occlusion and the priority of the one or more points, the cleaning plan including a specified actuation of each of a wiper and a fluid sprayer and a time duration of the specified actuations.

18. The method of claim 17, further comprising performing the segmentation of the sensor image data based on output of a neural network trained to detect a plurality of feature classes including at least an occlusion class.

19. The method of claim 17, further comprising:

determining the segmentation class of a location including one or more points on an exterior of the surface based on the sensor field of view, the vehicle route data, and the map data.

20. The method of claim 17, further comprising performing the selected cleaning plan.