Deep machine learning to predict and prevent adverse conditions at structural assets

• US 10,664,750 B2
• Filed: 08/10/2016
• Issued: 05/26/2020
• Est. Priority Date: 08/10/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Greene, Michael
• Primary Examiner(s): Chen, Alan
• Assistant Examiner(s): Islam, Urmana
• Application Number:
  US15/233,414
• Publication Number:
  US 20180046910A1
• Time in Patent Office: 1,385 Days
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06V 20/188 Vegetation
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

The present disclosure provides systems and methods that use machine-learned models, such as deep neural networks, to predict and prevent adverse conditions at structural assets. One example method includes obtaining data descriptive of a plurality of images that depict at least a portion of a geographic area that contains a first structural asset. The plurality of images include at least a first image captured at a first time and a second image captured at a second time that is different than the first time. The method includes inputting data descriptive of at least the first image, the first time, the second image, and the second time into a condition prediction model. The method includes receiving, as an output of the condition prediction model, at least one prediction regarding the occurrence of an adverse condition at the first structural asset during one or more future time periods.

Claims

1. A computing system to predict and prevent instances of vegetative overgrowth on utility infrastructure, the computing system comprising:

at least one processor; and

at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the computing system to;

obtain data descriptive of a plurality of images that depict at least a portion of a geographic area that contains a first utility asset, the plurality of images comprising at least a first image and a second image, the first image and the second image being different from each other in at least one of the following attributes;

time of capture, location of capture, angle of capture, data type, and resolution;

input data descriptive of at least the first image and the second image into a machine-learned vegetation overgrowth prediction model that comprises a machine-learned object detection model and a machine-learned vegetation overgrowth neural network;

receive, as an output of the machine-learned object detection model, one or more object feature vectors that respectively describe one or more attributes of one or more objects included in the geographic area;

input the one or more object feature vectors and at least one of the plurality of images into the machine-learned vegetation overgrowth neural network; and

receive, as an output of the machine-learned vegetation overgrowth neural network, at least one prediction regarding the occurrence of a vegetative overgrowth condition at the first utility asset during one or more future time periods, the at least one prediction produced by the machine-learned vegetation overgrowth neural network based at least in part on the one or more object feature vectors and the at least one of the plurality of images.

2. The computing system of claim 1, wherein:

the first image comprises a first ground-level image;

the second image comprises a first overhead image; and

the first ground-level image is of a relatively higher resolution than the first overhead image.

3. The computing system of claim 2, wherein:

the plurality of images comprise a plurality of ground-level images; and

execution of the instructions further causes the computing system to;

analyze the plurality of ground-level images to build a plant database that contains a plurality of plant feature vectors that respectively describe attributes of a plurality of plants contained in the geographic area; and

input at least one of the plurality of plant feature vectors into the machine-learned vegetation overgrowth prediction model along with at least one of the first image and the second image.

4. The computing system of claim 1, wherein execution of the instructions further causes the computing system to:

generate a maintenance schedule for the first utility asset based at least in part on the at least one prediction regarding the occurrence of the vegetative overgrowth condition at the first utility asset during the one or more future time periods.

5. A computer-implemented method to predict and prevent adverse conditions at structural assets, the method comprising:

obtaining, by one or more computing devices, data descriptive of a plurality of images that depict at least a portion of a geographic area that contains a first structural asset, wherein the plurality of images respectively depict the geographic area at a plurality of different times, the plurality of images comprising at least a first image captured at a first time and a second image captured at a second time that is different than the first time;

inputting, by the one or more computing devices, data descriptive of at least the first image, the first time, the second image, and the second time into a condition prediction model, the condition prediction model comprising a machine-learned object detection model and a machine-learned condition prediction neural network;

receiving, as an output of the machine-learned object detection model, one or more object feature vectors that respectively describe one or more attributes of one or more objects included in the geographic area;

inputting the one or more object feature vectors and at least one of the plurality of images into the machine-learned condition prediction neural network; and

receiving, by the one or more computing devices as an output of the machine-learned condition prediction neural network, at least one prediction regarding the occurrence of an adverse condition at the first structural asset during one or more future time periods, the at least one prediction produced by the machine-learned condition prediction neural network based at least in part on the one or more object feature vectors and the at least one of the plurality of images.

6. The computer-implemented method of claim 5, wherein obtaining, by the one or more computing devices, the data descriptive of the plurality of images comprises:

obtaining, by the one or more computing devices, the first image that comprises a first ground-level image of the geographic area that contains first structural asset; and

obtaining, by the one or more computing devices, the second image that comprises a first overhead image of the geographic area that contains the first structural asset.

7. The computer-implemented method of claim 6, wherein obtaining, by the one or more computing devices, the first ground-level image comprises obtaining, by the one or more computing devices, a first set of light detection and ranging (LIDAR) data that corresponds to the first ground-level image of the first structural asset.

8. The computer-implemented method of claim 6, wherein inputting, by the one or more computing devices, the data descriptive of at least the first image, the first time, the second image, and the second time into the condition prediction model comprises:

inputting, by the one or more computing devices, the first ground-level image into the machine-learned object detection model;

and inputting, by the one or more computing devices, data descriptive of the first time, the first overhead image, and the second time into the machine-learned condition prediction neural network of the condition prediction model alongside the one or more object feature vectors.

9. The computer-implemented method of claim 8, wherein:

inputting, by the one or more computing devices, the first ground-level image into the machine-learned object detection model comprises inputting, by the one or more computing devices, the first ground-level image into a plant detection model;

receiving, by the one or more computing devices, the one or more object feature vectors comprises receiving, by the one or more computing devices as an output of the plant detection model, one or more plant feature vectors that respectively describe one or more attributes of one or more respective plants included in the geographic area that contains the first structural asset and depicted by the first ground-level image, wherein the one or more plants influence the occurrence of a vegetation overgrowth condition at the first structural asset;

inputting, by the one or more computing devices, the data into the condition prediction neural network comprises inputting, by the one or more computing devices, data descriptive of at least the one or more plant feature vectors, the first time, the first overhead image, and the second time into a vegetative overgrowth prediction neural network; and

receiving, by the one or more computing devices as the output of the condition prediction model, the at least one prediction comprises receiving, by the one or more computing devices as the output of the vegetative overgrowth prediction neural network, the at least one prediction regarding the occurrence of the vegetative overgrowth condition at the first structural asset during the one or more future time periods.

10. The computer-implemented method of claim 6, wherein:

obtaining, by the one or more computing devices, the data descriptive of the plurality of images further comprises obtaining, by the one or more computing devices, a third image that comprises a second overhead image of the geographic area that contains the first structural asset, the third image captured at a third time that is different than the second time; and

inputting, by the one or more computing devices, the data into the condition prediction model comprises inputting, by the one or more computing devices, the data descriptive of at least the first image, the first time, the second image, the second time, the third image, and the third time into the condition prediction model.

11. The computer-implemented method of claim 10, wherein inputting, by the one or more computing devices, the data descriptive of at least the first image, the first time, the second image, the second time, the third image, and the third time into the condition prediction model comprises:

determining, by the one or more computing devices, one or more quantitative measures based at least in part on the first overhead image and the second overhead image, the one or more quantitative measures respectively descriptive of one or more changes exhibited by the geographic area between the second time and the third time; and

inputting, by the one or more computing devices, data descriptive of at least the first image, the first time, the one or more quantitative measures, the second time, and the third time into a condition prediction neural network of the condition prediction model.

12. The computer-implemented method of claim 11, wherein determining, by the one or more computing devices, the one or more quantitative measures comprises determining, by the one or more computing devices, the one or more quantitative measures respectively descriptive of a rate of vegetative growth exhibited by the geographic area between the second time and the third time.

13. The computer-implemented method of claim 5, further comprising:

prior to inputting the data into the condition prediction model, training, by the one or more computing devices the condition prediction model based at least in part on a set of training data that includes a plurality of training examples, each training example comprising a time series of imagery depicting a location, each training example providing information sufficient to determine, for each image in the time series of imagery, an amount of time until a known instance of occurrence of the adverse condition at the location depicted by such image.

14. The computer-implemented method of claim 5, wherein receiving, by the one or more computing devices as the output of the machine-learned condition prediction neural network, the at least one prediction regarding the occurrence of the adverse condition at the first structural asset during the one or more future time periods comprises receiving, by the one or more computing devices as the output of the machine-learned condition prediction neural network, a plurality of confidence scores that respectively describe a confidence of the adverse condition respectively occurring at the first structural asset during a plurality of future time periods.

15. The computer-implemented method of claim 5, wherein:

obtaining, by the one or more computing devices, the data descriptive of the plurality of images comprises obtaining, by the one or more computing devices, the data descriptive of the plurality of images that depict vegetative growth around a first utility asset at the plurality of different times;

inputting, by the one or more computing devices, the data descriptive of at least the first image, the first time, the second image, and the second time into the condition prediction model comprises inputting, by the one or more computing devices, the data descriptive of at least the first image, the first time, the second image, and the second time into a vegetative overgrowth prediction model that includes the at least one neural network; and

receiving, by the one or more computing devices as the output of the condition prediction model, the at least one prediction comprises receiving, by the one or more computing devices as the output of the vegetative overgrowth prediction model, the at least one prediction regarding the occurrence of a vegetative overgrowth condition at the first utility asset during the one or more future time periods.

16. A system to predict and prevent adverse conditions at structural assets, the system comprising:

an image database that stores a plurality of images of geographic areas, wherein the plurality of images comprise at least a first ground-level image that depicts at least a portion of a first geographic area that contains a first structural asset, and wherein the plurality of images further comprise at least a first overhead image that depicts at least a portion of the first geographic area that contains the first structural asset; and

a machine learning computing system, wherein the machine learning computing system comprises at least one processor and at least one memory that stores a condition prediction model usable to predict the occurrence of adverse conditions at structural assets during one or more future time periods based on input imagery, wherein the condition prediction model comprises a machine-learned object detection model and a machine-learned condition prediction neural network, and wherein the machine learning computing system is operable to;

obtain the first ground-level image of the first geographic area;

obtain the first overhead image of the first geographic area;

input data descriptive of at least the first ground-level image into the machine-learned objection detection model;

receive, as an output of the machine-learned object detection model, one or more object feature vectors that respectively describe one or more attributes of one or more objects included in the first geographic area;

input the one or more object feature vectors and the first overhead image into the machine-learned condition prediction neural network; and

receive at least one prediction regarding the occurrence of an adverse condition at the first structural asset as an output of the machine-learned condition prediction neural network, the at least one prediction produced by the machine-learned condition prediction neural network based at least in part on the one or more object feature vectors and the first overhead image.

17. The system of claim 16, wherein the first ground-level image comprises at least one of a first ground-level panoramic image and a first set of light detection and ranging (LIDAR) data.

18. The system of claim 16, wherein:

the machine-learned object detection model comprises a plant detection model;

the one or more object feature vectors comprise one or more plant feature vectors that respectively describe one or more attributes of one or more respective plants included in the first geographic area that contains the first structural asset and depicted by the first ground-level image, wherein the one or more plants influence the occurrence of a vegetation overgrowth condition at the first structural asset; and

the at least one prediction comprises at least one prediction regarding the occurrence of a vegetative overgrowth condition at the first structural asset.

19. The system of claim 16, further comprising a client computing system, wherein the client computing system comprises at least one processor and at least one memory, and wherein the client computing system is operable to:

receive user input that identifies at least one of the first geographic area and the first structural asset; and

in response to receipt of the user input, cause the machine learning computing system to obtain at least the first ground-level image and the first overhead image from the image database and input at least the first ground-level image and the first overhead image into the condition prediction model.