Sensor surface object detection methods and systems

• US 10,430,833 B2
• Filed: 12/29/2016
• Issued: 10/01/2019
• Est. Priority Date: 11/21/2016
• Status: Active Grant

• Current Assignee: NIO Technology Company Limited (Shanghai Anbin Technology Co., Ltd.)
• Original Assignee: Nio Usa Inc.
• Inventors: Newman, Austin L.
• Primary Examiner(s): Worden, Thomas E
• Application Number:
  US15/394,425
• Publication Number:
  US 20180143298A1
• Time in Patent Office: 1,006 Days
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60S 1/56 specially adapted for cleaning other parts or devices than front windows or windscreens rear-view mirror arrangements mounted on vehicle exterior including cleaning devices B60R1/0602
• CPC Class Codes: B60S 1/62 Other vehicle fittings for cleaning
• CPC Class Codes: B60W 10/00 Conjoint control of vehicle sub-units of different type or different function for propulsion of purely electrically-propelled vehicles with power supplied within the vehicle B60L50/00 - B60L58/00
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/06 including control of combustion engines
• CPC Class Codes: B60W 10/08 including control of electric propulsion units, e.g. motors or generators
• CPC Class Codes: B60W 10/10 including control of change-speed gearings
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2040/0809 Driver authorisation; Driver identity check
• CPC Class Codes: B60W 2050/0004 In digital systems, e.g. discrete-time systems involving sampling
• CPC Class Codes: B60W 2050/0014 Adaptive controllers
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2300/34 Compact city vehicles, e.g., microcars or kei cars
• CPC Class Codes: B60W 2420/403 : Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 : Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2420/54 : Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2510/08 : Electric propulsion units
• CPC Class Codes: B60W 2510/18 : Braking system
• CPC Class Codes: B60W 2520/04 : Vehicle stop
• CPC Class Codes: B60W 2520/105 : Longitudinal acceleration
• CPC Class Codes: B60W 2540/043 : Identity of occupants
• CPC Class Codes: B60W 2540/18 : Steering angle
• CPC Class Codes: B60W 2540/22 : Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/30 : Driving style
• CPC Class Codes: B60W 2552/40 : Coefficient of friction
• CPC Class Codes: B60W 2552/50 : Barriers
• CPC Class Codes: B60W 2552/53 : Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 2554/00 : Input parameters relating to objects
• CPC Class Codes: B60W 2554/4026 : Cycles
• CPC Class Codes: B60W 2554/4029 : Pedestrians
• CPC Class Codes: B60W 2554/4041 : Position
• CPC Class Codes: B60W 2554/4046 : Behavior, e.g. aggressive or erratic
• CPC Class Codes: B60W 2554/80 : Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2554/802 : Longitudinal distance
• CPC Class Codes: B60W 2556/10 : Historical data
• CPC Class Codes: B60W 2556/45 : External transmission of data to or from the vehicle
• CPC Class Codes: B60W 2556/55 : using telemetry
• CPC Class Codes: B60W 2556/65 : Data transmitted between vehicles
• CPC Class Codes: B60W 2710/18 : Braking system
• CPC Class Codes: B60W 2710/20 : Steering systems
• CPC Class Codes: B60W 2756/10 : Involving external transmission of data to or from the vehicle
• CPC Class Codes: B60W 2900/00 : Indexing codes relating to the purpose of, or problem solved of road vehicle drive control systems not otherwise provided for in groups B60W30/00
• CPC Class Codes: B60W 30/08 : Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: B60W 30/085 : Taking automatic action to adjust vehicle attitude in preparation for collision, e.g. braking for nose dropping
• CPC Class Codes: B60W 30/09 : Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/095 : Predicting travel path or likelihood of collision
• CPC Class Codes: B60W 30/0953 : the prediction being responsive to vehicle dynamic parameters
• CPC Class Codes: B60W 30/0956 : the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 40/04 : Traffic conditions
• CPC Class Codes: B60W 40/08 : related to drivers or passengers
• CPC Class Codes: B60W 40/09 : Driving style or behaviour
• CPC Class Codes: B60W 40/105 : Speed
• CPC Class Codes: B60W 50/0097 : Predicting future conditions
• CPC Class Codes: B60W 50/0098 : Details of control systems ensuring comfort, safety or stability not otherwise provided for
• CPC Class Codes: B60W 50/08 : Interaction between the driver and the control system
• CPC Class Codes: B60W 50/082 : Selecting or switching between different modes of propelling
• CPC Class Codes: B60W 60/0027 : using trajectory prediction for other traffic participants
• CPC Class Codes: B62D 15/00 : Steering not otherwise provided for
• CPC Class Codes: B62D 15/0265 : Automatic obstacle avoidance by steering
• CPC Class Codes: G01C 21/3407 : specially adapted for specific applications
• CPC Class Codes: G01C 21/3461 : Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types, segments such as motorways, toll roads, ferries
• CPC Class Codes: G01C 21/3469 : Fuel consumption; Energy use; Emission aspects
• CPC Class Codes: G01C 21/3484 : Personalized, e.g. from learned user behaviour or user-defined profiles
• CPC Class Codes: G01C 21/3492 : employing speed data or traffic data, e.g. real-time or historical traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968
• CPC Class Codes: G01C 21/3682 : output of POI information on a road map G01C21/3614, G01C21/3685 take precedence
• CPC Class Codes: G01C 21/3691 : Retrieval, searching and output of information related to real-time traffic, weather, or environmental conditions arrangements for giving variable traffic instructions G08G1/09
• CPC Class Codes: G01C 21/3697 : Output of additional, non-guidance related information, e.g. low fuel level G01C21/3679 takes precedence
• CPC Class Codes: G01S 13/862 : Combination of radar systems with sonar systems
• CPC Class Codes: G01S 13/865 : Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 13/87 : Combinations of radar systems, e.g. primary radar and secondary radar
• CPC Class Codes: G01S 15/02 : using reflection of acoustic waves G01S15/66 takes precedence
• CPC Class Codes: G01S 17/89 : for mapping or imaging
• CPC Class Codes: G01S 19/13 : Receivers
• CPC Class Codes: G01S 2007/4977 : including means to prevent or remove the obstruction
• CPC Class Codes: G01S 2013/9316 : combined with communication equipment with other vehicles or with base stations
• CPC Class Codes: G01S 2013/9318 : Controlling the steering
• CPC Class Codes: G01S 2013/93185 : Controlling the brakes
• CPC Class Codes: G01S 2013/9319 : Controlling the accelerator
• CPC Class Codes: G01S 2013/932 : using own vehicle data, e.g. ground speed, steering wheel direction
• CPC Class Codes: G01S 2013/9322 : using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 2013/9325 : for inter-vehicle distance regulation, e.g. navigating in platoons
• CPC Class Codes: G01S 2013/93271 : in the front of the vehicles
• CPC Class Codes: G01S 2013/93272 : in the back of the vehicles
• CPC Class Codes: G01S 2013/93273 : on the top of the vehicles
• CPC Class Codes: G01S 2013/93274 : on the side of the vehicles
• CPC Class Codes: G01S 2013/93275 : in the bumper area
• CPC Class Codes: G01S 7/4021 : of receivers
• CPC Class Codes: G01S 7/4043 : including means to prevent or remove the obstruction
• CPC Class Codes: G01S 7/497 : Means for monitoring or calibrating
• CPC Class Codes: G02B 27/0006 : with means to keep optical surfaces clean, e.g. by preventing or removing dirt, stains, contamination, condensation G02B1/18 takes precedence; cleaning in general B08B
• CPC Class Codes: G05D 1/0212 : with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0276 : using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G05D 1/228 : Command input arrangements located on-board unmanned vehicles
• CPC Class Codes: G05D 1/247 : using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
• CPC Class Codes: G05D 1/617 : Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80
• CPC Class Codes: G05D 1/646 : Following a predefined trajectory, e.g. a line marked on the floor or a flight path
• CPC Class Codes: G05D 1/81 : Handing over between on-board automatic and on-board manual control
• CPC Class Codes: G06F 16/29 : Geographical information databases
• CPC Class Codes: G06F 16/95 : Retrieval from the web
• CPC Class Codes: G06Q 30/0266 : based on the position of the vehicle
• CPC Class Codes: G06Q 30/0269 : based on user profile or attribute
• CPC Class Codes: G07C 5/0808 : Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005
• CPC Class Codes: G08G 1/161 : Decentralised systems, e.g. inter-vehicle communication
• CPC Class Codes: G08G 1/163 : involving continuous checking
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/165 : for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

Methods, devices, and systems of a sensor surface object detection system are provided. Output from sensors of a vehicle may be used to describe an environment around the vehicle. In the event that a sensor is obstructed by dirt, debris, or detritus the sensor may not sufficiently describe the environment for autonomous control operations. The sensor surface object detection system may receive output from the sensors of the vehicle to determine whether any of the sensors are obstructed. The determination may be made by comparing the output of one sensor to another, determining whether the output of a sensor is within a predetermined threshold, or comparing characteristics of multiple sensor outputs to one another. When a sensor is determined to be obstructed, the system may send a command to a cleaning system to automatically remove the obstruction.

Claims

1. A method, comprising:

receiving, via a processor, output from sensors of a vehicle monitoring an environment around a portion of the vehicle;

determining, via the processor, a first output over time for a first sensor of the sensors of the vehicle;

determining, via the processor, a second output over time for a second sensor of the sensors of the vehicle;

identifying, via the processor and based on sensor information in the first output over time and the second output over time, an obstructed sensor from at least one of the first sensor and the second sensor; and

sending, via the processor, a message including information about the obstructed sensor to one or more devices of the vehicle,

wherein the sensor information in the first output over time includes a time of flight defining a time for a detection signal emitted by the first sensor to be detected by the first sensor,

wherein the first sensor is identified as the obstructed sensor when the time of flight in the first output over time is less than a time of flight minimum threshold, the time of flight minimum threshold being defined as a minimum time of flight for maintaining a desired distance between the first sensor and possible obstructions,

wherein the message instructs a sensor cleaning system of the vehicle to clean the obstructed sensor,

wherein the sensor cleaning system includes;

an actuator attached to the obstructed sensor that moves the obstructed sensor between a first position and a second position;

a support structure around the obstructed sensor and set back from a wall of the vehicle; and

at least one flexible cleaning member that covers the obstructed sensor in the second position and that bends over the support structure at a location between the obstructed sensor and the support structure,

wherein, as the actuator moves the obstructed sensor between the first position and the second position and vice versa, the at least one flexible member cleans the surface of the obstructed sensor.

2. The method of claim 1, wherein identifying the obstructed sensor further comprises:

determining, via the processor, that the first sensor and the second sensor are a same type of sensor;

comparing, via the processor, the first output over time to the second output over time; and

determining, via the processor, at least one difference between the first output over time and the second output over time.

3. The method of claim 2, further comprising:

determining, via the processor, a third output over time for a third sensor of the sensors of the vehicle, wherein the third sensor is a different type of sensor from the first and second sensors;

comparing, via the processor, detection characteristics of the third output over time to detection characteristics for each of the first output over time and the second output over time; and

determining, via the processor, the obstructed sensor is at least one of the first sensor and the second sensor when the detection characteristics of the third output over time fail to match the detection characteristics for the first output over time and/or the second output over time.

4. The method of claim 1, wherein the first sensor is a first imaging sensor and the first output over time includes first computer-generated images of the environment around the portion of the vehicle over time, and wherein the second sensor is a second imaging sensor and the second output over time includes second computer-generated images of the environment around the portion of the vehicle over time.

5. The method of claim 4, wherein identifying the obstructed sensor further comprises:

determining, via the processor, a position and size for targets in the first and second computer-generated images over time, wherein the targets represent objects in the environment around the portion of the vehicle.

6. The method of claim 5, wherein identifying the obstructed sensor further comprises:

determining, via the processor, the obstructed sensor is one of the first sensor or the second sensor when at least one of the targets in one of the first and second computer-generated images over time is not present in the other of the first and second computer-generated images over time.

7. The method of claim 5, wherein identifying the obstructed sensor further comprises:

determining, via the processor and based on the position and size for the targets, a predicted position and size for the targets at a subsequent time.

8. The method of claim 7, wherein identifying the obstructed sensor further comprises:

determining, via the processor, a subsequent position and size for the targets in subsequent first and second computer-generated images over time; and

determining, via the processor, the obstructed sensor is at least one of the first sensor and the second sensor when the subsequent position and size for the targets in the subsequent first and second computer-generated images over time fail to match the predicted position and size for the targets at the subsequent time.

9. A vehicle, comprising:

at least two sensors;

a sensor cleaning system;

a microprocessor coupled to the at least two sensors; and

a computer readable medium coupled to the microprocessor and comprising instructions stored thereon that cause the microprocessor to;

receive output from the at least two sensors monitoring an environment around a portion of the vehicle;

determine a first output over time for a first sensor of the at least two sensors;

determine a second output over time for a second sensor of the at least two sensors;

identify, based on sensor information in the first output over time and the second output over time, an obstructed sensor from at least one of the first sensor and the second sensor; and

send a message including information about the obstructed sensor to one or more devices of the vehicle,

wherein the sensor information in the first output over time includes a time of flight defining a time for a detection signal emitted by the first sensor to be detected by the first sensor,

wherein the first sensor is identified as the obstructed sensor when the time of flight in the first output over time is less than a time of flight minimum threshold, the time of flight minimum threshold being defined as a minimum time of flight for maintaining a desired distance between the first sensor and possible obstructions,

wherein the message instructs the sensor cleaning system to clean the obstructed sensor,

wherein the sensor cleaning system includes;

an actuator attached to the obstructed sensor that moves the obstructed sensor between a first position and a second position;

a support structure around the obstructed sensor and set back from a wall of the vehicle; and

at least one flexible cleaning member that covers the obstructed sensor in the second position and that bends over the support structure at a location between the obstructed sensor and the support structure,

wherein, as the actuator moves the obstructed sensor between the first position and the second position and vice versa, the at least one flexible member cleans the surface of the obstructed sensor.

10. The vehicle of claim 9, wherein the first sensor is a first imaging sensor and the first output over time includes first computer-generated images of the environment around the portion of the vehicle over time, and wherein the second sensor is a second imaging sensor and the second output over time includes second computer-generated images of the environment around the portion of the vehicle over time.

11. The vehicle of claim 10, wherein in identifying the obstructed sensor the instructions further cause the microprocessor to:

determine a position and size for targets in the first and second computer-generated images over time, wherein the targets represent objects in the environment around the portion of the vehicle; and

determine the obstructed sensor is one of the first sensor or the second sensor when at least one of the targets in one of the first and second computer-generated images over time is not present in the other of the first and second computer-generated images over time.

12. A device, comprising:

a microprocessor; and

a computer readable medium coupled to the microprocessor and comprising instructions stored thereon that cause the microprocessor to;

receive output from sensors of a vehicle monitoring an environment around a portion of the vehicle;

determine a first output over time for a first sensor of the sensors of the vehicle;

determine a second output over time for a second sensor of the sensors of the vehicle;

identify, based on sensor information in the first output over time and the second output over time, an obstructed sensor from at least one of the first sensor and the second sensor; and

send a message including information about the obstructed sensor to one or more devices of the vehicle,

wherein the sensor information in the first output over time includes a time of flight defining a time for a detection signal emitted by the first sensor to be detected by the first sensor,

wherein the first sensor is identified as the obstructed sensor when the time of flight in the first output over time is less than a time of flight minimum threshold, the time of flight minimum threshold being defined as a minimum time of flight for maintaining a desired distance between the first sensor and possible obstructions,

wherein the message instructs a sensor cleaning system of the vehicle to clean the obstructed sensor,

wherein the sensor cleaning system includes;

an actuator attached to the obstructed sensor that moves the obstructed sensor between a first position and a second position;

a support structure around the obstructed sensor and set back from a wall of the vehicle; and

at least one flexible cleaning member that covers the obstructed sensor in the second position and that bends over the support structure at a location between the obstructed sensor and the support structure,

wherein, as the actuator moves the obstructed sensor between the first position and the second position and vice versa, the at least one flexible member cleans the surface of the obstructed sensor.

13. The device of claim 12, wherein the instructions cause the microprocessor to render at least a portion of the information about the obstructed sensor to a display device of the vehicle.