Road surface condition detection with multi-scale fusion

• US 9,950,700 B2
• Filed: 03/30/2016
• Issued: 04/24/2018
• Est. Priority Date: 03/30/2016
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Tong, Wei, Zhao, Qingrong, Litkouhi, Bakhtiar B., Zeng, Shuqing
• Primary Examiner(s): BUTLER, RODNEY ALLEN
• Application Number:
  US15/084,648
• Publication Number:
  US 20170282869A1
• Time in Patent Office: 755 Days
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60T 2201/022 Collision avoidance systems
• CPC Class Codes: B60T 2210/10 Detection or estimation of road conditions
• CPC Class Codes: B60T 2210/12 Friction
• CPC Class Codes: B60T 7/12 for automatic initiation; for initiation not subject to will of driver or passenger limiting speed of vehicles other than rail vehicles B60K31/00
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60T 8/171 Detecting parameters used in the regulation; Measuring values used in the regulation
• CPC Class Codes: B60T 8/175 Brake regulation specially adapted to prevent excessive wheel spin during vehicle acceleration, e.g. for traction control safety devices for propulsion unit control responsive to, or preventing, skidding of wheels B60K28/16

Abstract

A method of determining a surface condition of a path of travel. A plurality of images is captured of a surface of the path of travel by an image capture device. The image capture device captures images at varying scales. A feature extraction technique is applied by a feature extraction module to each of the scaled images. A fusion technique is applied, by the processor, to the extracted features for identifying the surface condition of the path of travel. A road surface condition signal provide to a control device. The control device applies the road surface condition signal to mitigate the wet road surface condition.

Claims

1. A method of determining a surface condition of a path of travel comprising the steps of:

obtaining data of a surface of the path of travel by sensor-based devices, the sensor-based devices capturing data at different sized scales;

applying a feature extraction technique, by a feature extraction module, to each of the scaled data;

wherein each feature extracted by the feature extraction module is independently input to a classifier, wherein the classifier outputs a respective classified result for each extracted feature input to the processor;

wherein the classifier provides the classified results for each extracted feature to a decision level fusion module of the processor, the decision level fusion module determining the surface condition of the path of travel based on fused classified results;

wherein the fused classified results are determined based on the following formula;

C=Σ

_{i=1, . . . , k}w_iC_i

wherein C_i is the classified result of each extracted feature by the classifier, and w_i is a weighting function;

applying a fusion technique, by the processor, to the extracted features for identifying the surface condition of the path of travel; and

providing a road surface condition signal to a control device, the control device applying the road surface condition signal to mitigate the wet road surface condition.

2. The method of claim 1 wherein each of the extracted features from each scale data are concatenated in a feature level fusion module of the processor.

3. The method of claim 2 wherein an output from the feature level fusion module is input to a classifier for identifying the surface condition of the road.

4. The method of claim 1 wherein the weighting function is determined by a classification confidence of the classified result.

5. The method of claim 4 wherein the weighting function is determined by prior knowledge of a scaling size of the captured image.

6. The method of claim 1 wherein each of the extracted features are input to a feature level fusion module of the processor to generate a feature level fusion result and a decision level fusion module of the processor to generate a decision level fusion result.

7. The method of claim 6 wherein the feature level fusion result is determined as a function of extracting features from each respective scaled data, wherein each extracted feature is input to the feature level fusion module for generating the feature level fusion result.

8. The method of claim 7 wherein the feature level fusion result is input to a classifier for identifying the surface condition of the road as a function of the extracted features.

9. The method of claim 8 wherein the decision level fusion result is determined as a function of extracting features from each respective scaled data by the decision level fusion module, wherein each feature extracted by the feature extraction module is independently input to the classifier, wherein the classifier outputs a respective classified result for each extracted feature, and wherein the classifier provides the classified results to the decision level fusion module for generating a road surface condition based on decision level fusion result.

10. The method of claim 9 wherein the classified feature level fusion result and the decision level fusion result are input to a hybrid fusion module of the processor to generate a hybrid fusion result identifying the surface condition of the path of travel.

11. The method of claim 10 wherein fusion module determines the hybrid fusion result based on the following formula:

C_h=w_ƒ

C_ƒ

+w_dC_d

where C_ƒ

is the feature level fusion result, w_ƒ

is a classification confidence weight, Ca is decision level fusion result, and w_d is a classification confidence result.

12. The method of claim 1 wherein fusion results are determined as a function of extracting features from each respective scaled data, wherein each feature extracted by the feature extraction module for determining the fusion results is independently determined using a deep learning technique.

13. The method of claim 1 wherein the road surface condition signal is provided to a vehicle controller, the controller using the road surface condition signal to determine an autonomous actuating vehicle braking strategy.

14. The method of claim 1 wherein the road surface condition signal is provided to a vehicle controller, the controller using the road surface condition signal to determine a traction control strategy for autonomously actuating a traction control system.

15. The method of claim 1 wherein the identified road surface condition signal is provided to a wireless communication system for alerting other vehicles of the identified surface condition on the path of travel.

16. The method of claim 1 wherein the road surface condition signal alerts a driver of a potential reduced traction between vehicle tires and the road surface.

17. The method of claim 1 wherein the identified road surface condition signal alerts a driver of the vehicle against a use of automated features.

18. The method of claim 1 wherein the identified road surface condition signal is provided to a vehicle controller, the controller autonomously modifying a control setting of an automated control feature in response to the road surface condition signal.

19. The method of claim 1 wherein the identified road surface condition signal is provided to a vehicle controller, the controller autonomously disabling cruise control in response to the road surface condition signal.

20. The method of claim 1 wherein the identified road surface condition signal alerts a driver to reduce a vehicle speed.

21. The method of claim 1 the data is obtained by active capture devices.