Monitoring and detecting weather conditions based on images acquired from image sensor aboard mobile platforms

• US 9,465,987 B1
• Filed: 03/17/2015
• Issued: 10/11/2016
• Est. Priority Date: 03/17/2015
• Status: Active Grant

• Current Assignee: NV5 Geospatial Solutions Incorporated (L3Harris Technologies, Inc.)
• Original Assignee: Exelis Incorporated (L3Harris Technologies, Inc.)
• Inventors: Bell, Brian, Dixon, Eric
• Primary Examiner(s): PARK, SOO JIN
• Application Number:
  US14/660,510
• Time in Patent Office: 574 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A weather control station receives from a communication network data packets transmitted wirelessly to the network by a mobile platform separated from the control station. The data packets carry images captured by an image sensor aboard the mobile platform and show one or more weather conditions local to the mobile platform. The control station recovers the images from the data packets and processes the recovered images according to one or more weather condition detection algorithms to detect the one or more weather conditions, respectively. The weather station reports the one or more detected weather conditions.

Claims

1. A method of detecting weather conditions using images from a mobile platform, comprising:

at a control station;

receiving from a communication network data packets transmitted wirelessly to the network by a mobile platform separated from the control station, the data packets carrying images captured by an image sensor aboard the mobile platform and that show one or more weather conditions local to the mobile platform;

recovering the images from the data packets;

processing the images according to one or more weather condition detection algorithms to detect the one or more weather conditions, respectively, including processing to detect whether rain is present by;

segmenting the images into foreground regions representative of rain and background regions;

bandpass filtering the foreground regions spatially and temporally;

detecting spatiotemporal intensity fluctuations in the bandpass filtered foreground regions that result from an optical disruption of background illumination by rain drops;

determining a frequency and magnitude of the detected intensity fluctuations; and

correlating the frequency and magnitude of the detected intensity fluctuations with an intensity of rain fall; and

reporting the one or more detected weather conditions.

2. The method of claim 1, further comprising, at the control station:

displaying the images and the reported one or more detected weather conditions.

3. The method of claim 1, wherein the processing the images includes processing intensity values of image pixels of the images according to the one or more weather condition detection algorithms.

4. The method of claim 1, wherein:

the images include images of a road surface over which the mobile platform travels; and

the processing includes processing to detect whether the road surface is wet or dry.

5. The method of claim 4, wherein the processing to detect whether the road surface is wet or dry includes:

normalizing scene lighting in images across different scene lighting conditions;

segmenting scene features in images to isolate the road surface from other scene features in the images;

extracting statistical features indicative of changes in road wetness conditions of the road surface; and

classifying the statistical features as either wet or dry road surface conditions.

6. The method of claim 1, wherein the processing includes processing to detect weather-related changes in visibility.

7. The method of claim 1, further comprising, at the mobile platform:

capturing the images that show the one or more weather conditions with the image sensor aboard the mobile platform; and

wirelessly transmitting the images in the data packets from the mobile platform to the communication network.

8. The method of claim 7, further comprising, at the mobile platform:

prior to the wirelessly transmitting, preprocessing the images by;

stabilizing the images to compensate for scene motion introduced by motion of the mobile platform; and

compressing the images,

wherein the wirelessly transmitting includes wirelessly transmitting the images in a stabilized, image compressed form in the data packets.

9. The method of claim 8, wherein the preprocessing further includes filtering the images to remove images that are unsuitable for being processed according to the one or more weather condition detection algorithms used to detect the weather conditions.

10. The method of claim 7, wherein:

the image sensor is positioned on the mobile platform and configured so that a field of view of the image sensor encompasses a scene of a road surface on which the mobile platform travels, a horizon toward or away from which the mobile platform travels, and a portion of sky; and

the capturing images includes capturing images of the scene of the road surface, the horizon, and the portion of the sky.

11. The method of claim 1, further comprising, at the control station:

receiving from the communication network measurements of environmental conditions obtained by corresponding environmental sensors aboard the mobile platform and wirelessly transmitted to the network by the mobile platform, the measurements including measurements of one or more of temperature, humidity, atmospheric pressure, and dew point; and

verifying the one or more detected weather conditions based on the environmental measurements.

12. A apparatus comprising:

a network interface unit (NIU) to send and receive data packets to and from a communication network; and

a processor coupled to the NIU and configured to;

receive from a communication network data packets transmitted wirelessly to the network by a mobile platform separated from the apparatus, the data packets carrying images captured by an image sensor aboard the mobile platform and that show one or more weather conditions local to the mobile platform;

recover the images from the data packets;

process the images according to one or more weather condition detection algorithms to detect the one or more weather conditions, respectively, including to detect whether rain is present by;

segmenting the images into foreground regions representative of rain and background regions;

bandpass filtering the foreground regions spatially and temporally;

detecting spatiotemporal intensity fluctuations in the bandpass filtered foreground regions that result from an optical disruption of background illumination by rain drops;

determining a frequency and magnitude of the detected intensity fluctuations; and

correlating the frequency and magnitude of the detected intensity fluctuations with an intensity of rain fall; and

report the one or more detected weather conditions.

13. The apparatus of claim 12, wherein the processor is further configured to:

generate for display the images and the reported one or more detected weather conditions.

14. The apparatus of claim 12, wherein the processor is configured to process the images by processing intensity values of image pixels of the images according to the one or more weather condition detection algorithms.

15. The apparatus of claim 14, wherein the processor is configured to process the images by processing the images according to the one or more weather condition detection algorithms to detect one or more of whether a road surface on which the mobile platform travels is wet or dry, weather-related changes in visibility, and whether rain is present.

16. The apparatus of claim 12, wherein the processor is further configured to:

receive from the communication network measurements of environmental conditions obtained by corresponding environmental sensors aboard the mobile platform and wirelessly transmitted to the network by the mobile platform, the measurements including measurements of one or more of temperature, humidity, and atmospheric pressure; and

verify the one or more detected weather conditions based on the environmental measurements.

17. The apparatus of claim 12, wherein the images are one of visible light images and infrared images.

18. A system comprising:

multiple mobile platforms each respectively equipped with;

an image sensor to capture images that show weather conditions local to the mobile platform;

a mobile platform processor to packetize the images into data packets; and

a wireless transmitter to wirelessly transmit the data packets to a communication network; and

a control station to;

receive from the communication network the data packets transmitted thereto by the mobile platforms;

recover from the data packets the images from each of the image sensors and process the images according to one or more weather condition detection algorithms to detect the one or more weather conditions local to each of the mobile platforms, respectively, including to detect whether rain is present by;

segmenting the images into foreground regions representative of rain and background regions;

bandpass filtering the foreground regions spatially and temporally;

detecting spatiotemporal intensity fluctuations in the bandpass filtered foreground regions that result from an optical disruption of background illumination by rain drops;

determining a frequency and magnitude of the detected intensity fluctuations; and

correlating the frequency and magnitude of the detected intensity fluctuations with an intensity of rain fall; and

generate for display the images from each mobile platform and corresponding reports of detected weather conditions output by the weather condition detection algorithms.

19. The system of claim 18, wherein the multiple image sensors of the multiple mobile platforms include video cameras mounted to the mobile platforms.

20. The system of claim 18, wherein:

the mobile platform processor is further configured to preprocess the images by;

stabilizing the images to compensate for scene motion introduced by motion of the mobile platform; and

compressing the images; and

the wireless transmitter is configured to wirelessly transmit the images in the data packets in a stabilized, image compressed form.

21. The system of claim 18, wherein the station processor is configured to process the received images by processing intensity values of image pixels of the received images according to the one or more weather condition detection algorithms.