Estimating conditions from observations of one instrument based on training from observations of another instrument

• US 10,489,923 B2
• Filed: 12/13/2017
• Issued: 11/26/2019
• Est. Priority Date: 12/13/2017
• Status: Active Grant

• Current Assignee: Vaisala Oyj
• Original Assignee: Vaisala Oyj
• Inventors: Ruzanski, Evan, Petty, Kevin, Tan, Haiming, Jackson, Cosima
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US15/841,060
• Publication Number:
  US 20190180458A1
• Time in Patent Office: 713 Days
• CPC Class Codes: G01B 11/303 using photoelectric detection means
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06T 2207/30192 Weather; Meteorology
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

Systems and methods are disclosed to label data collected by one instrument based on data collected by another instrument, train a computer learning model with the labeled data, and then use the trained computer learning model to estimate a condition from new unlabeled data. For example, weather-related pavement conditions may be estimated from camera images according to such systems and methods. Systems and methods are also disclosed to estimate road weather safety or hazard conditions using two different types of pavement conditions, such as road state and friction or grip coefficient, estimated from unlabeled camera images.

Claims

1. A method, comprising:

accessing a first data set generated by a first instrument and a second data set generated by a second instrument different than the first instrument, wherein;

the first data set includes first observations by the first instrument of a subject under observation;

the second data set includes second observations by the second instrument of the subject;

the first observations are of a different type than the second observations and include conditions of the subject at points in time at which the first observations are made; and

each condition of the subject at a corresponding point in time at which the corresponding first observation is made is selected from a set of possible conditions;

automatically labeling the second data set based on the first data set;

training a computer learning model with the labeled second data set to recognize complex relationships between the second observations and conditions in the set of possible conditions;

analyzing, with the trained computer learning model, a third data set that includes a third observation by the second instrument of the subject; and

based on the analyzing, estimating a condition of the subject in the third observation, the estimated condition selected from the set of possible conditions.

2. The method of claim 1, wherein automatically labeling the second data set based on the first data set includes automatically labeling each second observation with the corresponding condition of the subject included in a corresponding one of the first observations made temporally near in time to the second observation.

3. The method of claim 1, wherein only one of:

the first observations made by the first instrument include liquid water equivalent value measurements made by a surface precipitation gauge of a surface included in the subject, the second observations made by the second instrument include images of the atmosphere included in the subject, and estimating the condition of the subject in the third observation includes estimating a liquid water equivalent value of the subject from a new image of the atmosphere;

the first observations made by the first instrument include light measurements made by a surface lighting sensor of the subject, the second observations made by the second instrument include images of the atmosphere included in the subject, and estimating the condition of the subject in the third observation includes estimating impending lightning activity in the atmosphere of the subject from a new image of the atmosphere;

or

the first observations made by the first instrument include point observations made by an Automated Surface Observation Station (ASOS) of an ASOS site included in the subject, the second observations made by the second instrument include images of the ASOS site made by an imaging device mounted to an aircraft that flies over the ASOS site, and estimating the condition of the subject in the third observation includes estimating an ASOS observation of the ASOS site from a new image of the ASOS site.

4. The method of claim 1, wherein:

the first observations made by the first instrument include a set of pavement conditions observed by a condition sensor of pavement included in an environment in the subject;

the second observations made by the second instrument include a set of images of the pavement in the environment;

the training comprises training the computer learning model with the set of images of the pavement in the environment, each image in the set of images labeled with a corresponding pavement condition in the set of pavement conditions, to recognize complex relationships between the set of images and the set of pavement conditions;

the analyzing comprises analyzing a new image of the pavement in the environment with the trained computer learning model; and

the estimating comprises estimating a pavement condition of the pavement in the new image based on the analyzing.

5. The method of claim 4, wherein estimating the pavement condition of the new image comprises estimating a quantitative representation of pavement condition associated with a time of capture of the new image.

6. The method of claim 5, wherein estimating the quantitative representation comprises estimating a grip or friction coefficient of the pavement.

7. The method of claim 6, wherein estimating the grip or friction coefficient of the pavement comprises solving a regression problem.

8. The method of claim 4, wherein training the computer learning model with the set of images comprises training the computer learning model with raw image data.

9. The method of claim 8, wherein training the computer learning model with raw image data comprises at least one of:

training the computer learning model with images that are unmodified from their captured states;

training the computer learning model with images that include both the pavement and other features in the environment apart from the pavement;

or

training the computer learning model without identifying or extracting features of the images prior to the training.

10. The method of claim 4, further comprising capturing each image in the set of images under ambient light conditions.

11. The method of claim 4, wherein training the computer learning model with the set of images comprises training a convolutional neural network (CNN) configured to accurately and efficiently learn patterns in digital image data of the set of images without prior manual selection and extraction of features from the digital image data.

12. A non-transitory computer-readable medium having computer-readable instructions stored thereon that are executable by a processor device to perform or control performance of the method of claim 1.

13. A system, comprising:

a first data set generated by a first instrument, the first data set including first observations by the first instrument of a subject under observation, the first observations including conditions of the subject at points in time at which the first observations are made, wherein each condition of the subject at the corresponding point in time at which the corresponding first observation is made is selected from a set of possible conditions;

a second data set generated by a second instrument different than the first instrument, the second data set including second observations by the second instrument of the subject, the first observations being of a different type than the second observations, the second data set labeled based on the first data set; and

a computer learning model that has access to the first data set and the second data set, wherein;

the computer learning model is trainable with the first data set and the second data set to recognize complex relationships between the second data set and conditions in the set of possible conditions; and

the trained computer learning model is configured to;

analyze a third data set that includes a third observation by the second instrument of the subject; and

based on the analysis, estimate a condition of the subject.

14. The system of claim 13, wherein:

the first data set generated by the first instrument includes a set of pavement conditions generated by a condition sensor of pavement included in an environment in the subject;

the second data set generated by the second instrument includes a set of images of the pavement in the environment, each image in the set of images labeled with a corresponding pavement condition in the set of pavement conditions;

the computer learning model has access to the set of images and the set of pavement conditions,

the computer learning model being trainable includes the computer learning model being trainable with the set of images and the set of pavement conditions to recognize complex relationships between the set of images and the set of pavement conditions;

the trained computer learning model being configured to analyze the third data set includes the trained computer learning model being configured to analyze a new image of the pavement in the environment; and

the trained computer learning model being configured to estimate the condition includes the trained computer learning model being configured to estimate a pavement condition of the pavement in the new image based on the analysis.

15. The system of claim 14, wherein the estimated pavement condition comprises an estimated quantitative representation of pavement condition associated with a time of capture of the new image.

16. The system of claim 15, wherein the quantitative representation comprises a grip or friction coefficient of the pavement.

17. The system of claim 14, further comprising:

the second instrument, wherein the second instrument includes a camera located in a vicinity of the pavement in the environment and configured to capture each image in the set of images, wherein each image is captured at a corresponding known time of capture;

the first instrument, wherein the first instrument includes the condition sensor located in the vicinity of the pavement and configured to measure the condition of the pavement temporally near to the corresponding known time of capture of each image; and

a processor communicatively coupled to the camera and the condition sensor and configured to automatically label each image with a corresponding temporally near measured condition as the corresponding pavement condition of the corresponding image.

18. The system of claim 17, wherein the condition sensor comprises a remote condition sensor.

19. The system of claim 17, wherein the condition sensor comprises an in situ condition sensor.

20. The system of claim 14, wherein the set of images comprises raw image data, the raw image data comprising at least one of:

images that are unmodified from their captured states;

images that include both the pavement and other features in the environment apart from the pavement;

or

unprocessed image data devoid of extracted features.

21. The system of claim 20, wherein the other features in the environment apart from the pavement include one or more of surrounding terrain, sky, foot traffic, or vehicle traffic.

22. The system of claim 14, wherein the second instrument includes a camera located in a vicinity of the pavement in the environment and configured to capture each image in the set of images under ambient light conditions.

23. The system of claim 14, wherein the computer learning model comprises a convolutional neural network (CNN) configured to accurately and efficiently learn patterns in digital image data of the set of images without prior manual selection and extraction of features from the digital image data.

24. The system of claim 13, wherein the CNN comprises a CNN run on a graphical processing unit (GPU) in a cloud-based architecture.