Multi-sensor precipitation-classification apparatus and method

• US 10,427,645 B2
• Filed: 10/06/2016
• Issued: 10/01/2019
• Est. Priority Date: 10/06/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Karandikar, Ramchandra Ganesh, Nagraj Rao, Nikhil, Myers, Scott Vincent, Charette, Francois
• Primary Examiner(s): Heinle, Courtney D
• Assistant Examiner(s): Martinez Borrero, Luis A
• Application Number:
  US15/287,305
• Publication Number:
  US 20180099646A1
• Time in Patent Office: 1,090 Days
• Field of Search: 701 36, 701 70, 701117, 701 44, 701 74
• CPC Class Codes: B60S 1/0844 including a camera
• CPC Class Codes: B60S 1/0859 Other types of detection of rain, e.g. by measuring friction or rain drop impact
• CPC Class Codes: B60W 2050/0086 Recalibrating datum positions, e.g. by using check cycles
• CPC Class Codes: B60W 2422/00 Indexing codes relating to the special location or mounting of sensors
• CPC Class Codes: B60W 2552/40 Coefficient of friction
• CPC Class Codes: B60W 2756/10 Involving external transmission of data to or from the vehicle
• CPC Class Codes: B60W 30/18172 Preventing, or responsive to skidding of wheels
• CPC Class Codes: B60W 40/02 related to ambient conditions
• CPC Class Codes: B60W 40/068 Road friction coefficient
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A vehicle is disclosed that uses data from different types of on-board sensors to determine whether meteorological precipitation is failing near the vehicle. The vehicle may include an on-board camera capturing image data and an on-board accelerometer capturing accelerometer data. The image data may characterize an area in front of or behind the vehicle. The accelerometer data may characterize vibrations of a windshield of the vehicle. An artificial neural network may run on computer hardware carried on-board the vehicle. The artificial neural network may be trained to classify meteorological precipitation in an environment of the vehicle using the image data and the accelerometer data as inputs. The classifications of the artificial neural network may be used to control one or more functions of the vehicle such as windshield-wiper speed, traction-control settings, or the like.

Claims

1. A vehicle comprising:

a camera outputting image data;

a windshield;

an accelerometer outputting accelerometer data characterizing vibration of the windshield;

an artificial neural network trained to classify meteorological precipitation in an environment of the vehicle using the image data and the accelerometer data as inputs; and

at least one actuator controlling at least one function of the vehicle based on classifications made by the artificial neural network;

wherein the artificial neural network includes a plurality of classifiers, each classifier of the plurality of classifiers being trained to detect a type of precipitation corresponding to the each classifier, each classifier being configured to take as inputs the accelerometer data and the image data and configured to output a decision indicating whether the type of precipitation corresponding to the each classifier is present; and

wherein each classifier is further configured to apply weights of the each classifier to the accelerometer data and the image data such that the weights of one classifier of the plurality of classifier are different from the weights of another classifier of the plurality of classifiers.

2. The vehicle of claim 1, wherein the camera is secured to the vehicle and oriented so as to be forward facing.

3. The vehicle of claim 1, wherein the camera is secured to the vehicle and oriented so as to be rearward facing.

4. The vehicle of claim 1, wherein the image data comprises multiple consecutive images captured by the camera over a period of less than ten seconds.

5. The vehicle of claim 4, further comprising at least one windshield wiper and wherein the at least one function comprises a speed of the at least one windshield wiper.

6. The vehicle of claim 5, further comprising a drivetrain having one or more tires.

7. The vehicle of claim 6, further comprising a drivetrain sensor outputting friction data characterizing frictional engagement of the one or more tires with a driving surface.

8. The vehicle of claim 7, wherein the artificial neural network is trained to classify meteorological precipitation based at least in part on each of the image data, the accelerometer data, and the friction data.

9. The vehicle of claim 4, further comprising a traction control system and wherein the at least one function comprises a setting of the traction control system.

10. The vehicle of claim 1, further comprising a communication system communicating one or more classifications of the artificial neural network to hardware off-board the vehicle.

11. The vehicle of claim 1, wherein the artificial neural network is trained to classify at least three classes of meteorological precipitation, the at least three classes comprising no rain, rain below a first precipitation rate, and rain above a second precipitation rate that is equal to or greater than the first precipitation rate.

12. A method comprising:

receiving, by a computer system carried on-board a vehicle, image data captured by a camera oriented so as to be forward or rearward facing with respect to the vehicle, wherein the image data comprises at least one image captured by the camera within a period of time;

receiving, by the computer system, accelerometer data output by an accelerometer secured to a windshield of the vehicle, wherein the accelerometer data characterizes vibration of the windshield during the period of time;

receiving, by an artificial neural network run on the computer system, the image data and the accelerometer data as inputs;

generating, by the artificial neural network, classification scores based on the inputs with respect to at least three classes of meteorological precipitation, the at least three classes comprising rain, hail, and snow; and

controlling, by the computer system, at least one function of the vehicle based on the classification scores generated by the artificial neural network;

wherein generating the classification scores based on the inputs comprises;

inputting the image data and accelerometer data to a plurality of classifiers, each classifier of the plurality of classifiers being trained to detect a class of the at least three classes of meteorological precipitation corresponding to the each classifier;

weighting, by each classifier of the plurality of classifiers, the image data and accelerometer data according to weights of the each classifier such that the weights of one classifier of the plurality of classifiers are different from the weights of another classifier of the plurality of classifiers;

generating, by each classifier of the plurality of classifiers, a classification score of the classifications scores according to the weights of the each classifier, the image data, and the accelerometer data.

13. The method of claim 12, wherein:

the image date comprises multiple consecutive images captured by the camera over the period of time; and

the period of time is less 10 seconds.

14. The method of claim 13, wherein the vehicle comprises at least one windshield wiper and wherein the at least one function comprises a speed of the at least one windshield wiper.

15. The method of claim 12, wherein the vehicle comprises a drivetrain having one or more tires.

16. The method of claim 15, wherein the vehicle further comprises a drivetrain sensor outputting friction data characterizing frictional engagement of the one or more tires with a driving surface.

17. The method of claim 16, further comprising receiving, by the artificial neural network, the image data, the accelerometer data, and the friction data as the inputs.

18. The method of claim 17, wherein the vehicle further comprises a traction control system and wherein the at least one function comprises a setting of the traction control system.

19. The method of claim 12, wherein the vehicle further comprises a communication system communicating one or more classifications made by the artificial neural network to hardware off-board the vehicle.

20. The method of claim 19, wherein the hardware off-board the vehicle forms part of stationary telecommunications infrastructure or another vehicle.