Vision-based rain detection using deep learning

• US 10,049,284 B2
• Filed: 04/11/2016
• Issued: 08/14/2018
• Est. Priority Date: 04/11/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Jain, Jinesh J, Banvait, Harpreetsingh, Micks, Ashley Elizabeth, Nariyambut Murali, Vidya
• Primary Examiner(s): Nakhjavan, Shervin
• Application Number:
  US15/095,876
• Publication Number:
  US 20170293808A1
• Time in Patent Office: 855 Days
• Field of Search: None
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

A method is disclosed for using a camera on-board a vehicle to determine whether precipitation is failing near the vehicle. The method may include obtaining multiple images. Each of the multiple images may be known to photographically depict a “rain” or a “no rain” condition. An artificial neural network may be trained on the multiple images. Later, the artificial neural network may analyze one or more images captured by a first camera secured to a first vehicle. Based on that analysis, the artificial neural network may classify the first vehicle as being in “rain” or “no rain” weather.

Claims

1. A method comprising:

obtaining multiple images, each known to photographically depict a “

rain”

or a “

no rain”

condition;

training, by an artificial neural network, on the multiple images;

analyzing, by the artificial neural network after the training, a plurality of consecutive images captured by a first camera;

classifying, by the artificial neural network based on the analyzing, each image of the plurality of consecutive images as being in “

rain”

or “

no rain”

weather;

(a) determining that a major portion of the plurality of consecutive images have been classified as in “

rain”

weather; and

in response to (a), modifying operation of the vehicle.

2. The method of claim 1, wherein the multiple images are captured by one or more cameras on-board one or more vehicles.

3. The method of claim 1, wherein the first camera is on-board a first vehicle.

4. The method of claim 3, wherein the analyzing occurs on-board the first vehicle.

5. The method of claim 4, wherein the classifying occurs on-board the first vehicle.

6. The method of claim 3, wherein the training occurs off-board the first vehicle.

7. The method of claim 3, wherein the training occurs while the artificial neural network is running on off-board computer hardware located remotely with respect to the first vehicle.

8. The method of claim 3, wherein the analyzing and the classifying occurs while the artificial neural network is running on on-board computer hardware carried on-board the first vehicle.

9. The method of claim 1, wherein the first camera is secured to a first vehicle and oriented so as to be forward facing.

10. The method of claim 1, wherein the first camera is secured to a first vehicle and oriented so as to be rearward facing.

11. The method of claim 1, wherein the plurality of consecutive images comprise multiple consecutive images captured by the first camera over a period of less than ten seconds.

12. A method comprising:

obtaining a first plurality of images captured by one or more cameras on-board one or more vehicles, each image of the first plurality of images being known to photographically depict a “

rain”

or a “

no rain”

condition;

using the first plurality of images to train an artificial neural network to distinguish between photographic data corresponding to the rain condition and photographic data corresponding to the no rain condition;

analyzing, by the artificial neural network after the using, a second plurality of consecutive images captured by a first camera secured to a first vehicle;

classifying, by the artificial neural network based on the analyzing, each image of the second plurality of consecutive images as being in “

rain”

or “

no rain”

weather;

(a) determining that a major portion of the second plurality of consecutive images have been classified as in “

rain”

weather; and

in response to (a), producing an output modifying operation of a vehicle.

13. The method of claim 12, wherein the analyzing and the classifying occurs while the artificial neural network is running on on-board computer hardware carried on-board the first vehicle.

14. The method of claim 12, further comprising capturing, by the first camera, the second plurality of consecutive images.

15. The method of claim 14, wherein the analyzing and the classifying occur in real time with the capturing.

16. The method of claim 15, wherein the analyzing and the classifying occur less than ten seconds after the capturing.

17. The method of claim 12, wherein the using occurs while the artificial neural network is running on off-board computer hardware located remotely with respect to the first vehicle.

18. The method of claim 12, wherein the second plurality of consecutive images comprise multiple consecutive images captured by the first camera over a period of less than ten seconds.

19. A computer system comprising:

one or more processors;

memory operably connected to the one or more processors; and

the memory storing

an artificial neural network trained on a plurality of images captured by one or more cameras on-board one or more vehicles to distinguish between images corresponding to a “

rain”

condition and images corresponding to a “

no rain”

condition;

a plurality of consecutive images captured by a first camera secured to a first vehicle; and

software programmed to cause the one or more processors to—

feed the plurality of consecutive images to the artificial neural network for classification;

when a majority of the plurality of consecutive images are classified by the artificial neural network as corresponding to the “

rain”

condition thereby indicating a transition from the “

no rain”

to the “

rain”

condition, change a functional characteristic of the first vehicle.