SYSTEM AND METHOD FOR LABELLING AERIAL IMAGES

• US 20130343641A1
• Filed: 06/21/2013
• Published: 12/26/2013
• Est. Priority Date: 06/22/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mnih, Volodymyr, Hinton, Geoffrey E.
• Granted Patent: US 9,704,068 B2
• Time in Patent Office: Days
• US Class Current: 382/156
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/182 Network patterns, e.g. roads or rivers

Abstract

A system and method for labelling aerial images. A neural network generates predicted map data. The parameters of the neural network are trained by optimizing an objective function which compensates for noise in the map images. The function compensates both omission noise and registration noise.

Claims

1. (canceled)

2. A method performed by one or more computers, the method comprising:

training a neural network on training data, wherein the training data includes a plurality of labeled aerial images, wherein the neural network is configured to generate a respective score for each of a plurality of pixels of an input aerial image, wherein the respective scores for each of the plurality of pixels represent a probability that the pixel belongs to an object class, and wherein training the neural network on the training data comprises;

minimizing an objective function having a derivative that includes a posterior probability term that represents a likelihood that labels for pixels from the labeled aerial images differ from true labels for the pixels.

3. The method of claim 2, wherein the objective function is a negative log likelihood function.

4. The method of claim 3, wherein the derivative takes the form of, for a pixel i in a particular labeled aerial image, p(m_i=1|{tilde over (m)}_i, s)−

{circumflex over (m)}_i, wherein p(m_i=1|{tilde over (m)}_i, s) is a posterior probability that a true label m_i for the pixel i is 1 given a label for the pixel i in the particular labeled aerial image {tilde over (m)}_s and a portion s of the aerial image containing the pixel i, and wherein {circumflex over (m)}_i is a probability predicted for the pixel i by the neural network.

5. The method of claim 2, wherein the posterior probability term compensates for omission errors in the training data, an omission error occurring when a pixel in a labeled aerial image has been incorrectly labeled as not belonging to the object class.

6. The method of claim 2, wherein the object class is roads, and wherein the respective scores for each of the plurality of pixels represent a probability that the pixel is a part of an image of a road.

7. A method performed by one or more computers, the method comprising:

training a neural network on training data, wherein the training data includes a plurality of labeled aerial images, wherein the neural network is configured to generate a respective score for each of a plurality of pixels of an input aerial image, wherein the respective scores for each of the plurality of pixels represent a probability that the pixel belongs to an object class, and wherein training the neural network on the training data comprises;

minimizing an objective function to adjust values of parameters of the neural network and of a translation parameter, wherein the translation parameter accounts for potential translation errors for labels in the labeled aerial images, wherein a translation error in a labeled aerial image is a misalignment between the pixels of an object of the object class in the labeled aerial image and the labels for the object in the labeled aerial image.

8. The method of claim 7, wherein the objective function is a negative log likelihood function.

9. The method of claim 7, wherein training the neural network on the training data comprises training a model that generates the labeled aerial images from true labeled aerial images.

10. The method of claim 9, wherein the translation parameter parameterizes a translation distribution from which a value of a translation variable is sampled during the training.

11. The method of claim 10, wherein the value of the translation variable defines, for a labeled aerial image, a translation necessary to obtain the labeled aerial image from a corresponding true labeled aerial image.

12. The method of claim 7, wherein the object class is roads, and wherein the respective scores for each of the plurality of pixels represent a probability that the pixel is a part of an image of a road.

13. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

training a neural network on training data, wherein the training data includes a plurality of labeled aerial images, wherein the neural network is configured to generate a respective score for each of a plurality of pixels of an input aerial image, wherein the respective scores for each of the plurality of pixels represent a probability that the pixel belongs to an object class, and wherein training the neural network on the training data comprises;

minimizing an objective function having a derivative that includes a posterior probability term that represents a likelihood that labels for pixels from the labeled aerial images differ from true labels for the pixels.

14. The system of claim 13, wherein the objective function is a negative log likelihood function.

15. The system of claim 14 wherein the derivative takes the form of, for a pixel i in a particular labeled aerial image, p(m_i=1|{tilde over (m)}_i, s)−

{circumflex over (m)}_i, wherein p(m_i=1|{tilde over (m)}_i, s) is a posterior probability that a true label m_i for the pixel i is 1 given a label for the pixel i in the particular labeled aerial image {tilde over (m)}_s and a portion s of the aerial image containing the pixel i, and wherein {circumflex over (m)}_i is a probability predicted for the pixel i by the neural network.

16. The system of claim 13 wherein the posterior probability term compensates for omission errors in the training data, an omission error occurring when a pixel in a labeled aerial image has been incorrectly labeled as not belonging to the object class.

17. The system of claim 13 wherein the object class is roads, and wherein the respective scores for each of the plurality of pixels represent a probability that the pixel is a part of an image of a road.

18. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

training a neural network on training data, wherein the training data includes a plurality of labeled aerial images, wherein the neural network is configured to generate a respective score for each of a plurality of pixels of an input aerial image, wherein the respective scores for each of the plurality of pixels represent a probability that the pixel belongs to an object class, and wherein training the neural network on the training data comprises;

minimizing an objective function to adjust values of parameters of the neural network and of a translation parameter, wherein the translation parameter accounts for potential translation errors for labels in the labeled aerial images, wherein a translation error in a labeled aerial image is a misalignment between the pixels of an object of the object class in the labeled aerial image and the labels for the object in the labeled aerial image.

19. The system of claim 18, wherein the objective function is a negative log likelihood function.

20. The system of claim 18, wherein training the neural network on the training data comprises training a model that generates the labeled aerial images from true labeled aerial images.

21. The system of claim 20, wherein the translation parameter parameterizes a translation distribution from which a value of a translation variable is sampled during the training.

22. The system of claim 21, wherein the value of the translation variable defines, for a labeled aerial image, a translation necessary to obtain the labeled aerial image from a corresponding true labeled aerial image.

23. The system of claim 18, wherein the object class is roads, and wherein the respective scores for each of the plurality of pixels represent a probability that the pixel is a part of an image of a road.