Training a neural network to detect objects in images

• US 9,514,389 B1
• Filed: 06/17/2016
• Issued: 12/06/2016
• Est. Priority Date: 11/01/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Erhan, Dumitru, Szegedy, Christian, Anguelov, Dragomir
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US15/185,613
• Time in Patent Office: 172 Days
• Field of Search: 382/156, 382/157, 382/158, 382/159, 382/278, 706/15, 706/16, 706/25, 128/925
• US Class Current: 1/1
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to detect object in images. One of the methods includes receiving a training image and object location data for the training image; providing the training image to a neural network and obtaining bounding box data for the training image from the neural network, wherein the bounding box data comprises data defining a plurality of candidate bounding boxes in the training image and a respective confidence score for each candidate bounding box in the training image; determining an optimal set of assignments using the object location data for the training image and the bounding box data for the training image, wherein the optimal set of assignments assigns a respective candidate bounding box to each of the object locations; and training the neural network on the training image using the optimal set of assignments.

Claims

1. A system for detecting objects in images, the system comprising:

an object detection neural network implemented by one or more computers, the object detection neural network comprising;

one or more hidden neural network layers configured to;

receive an input image, and

process the input image to generate a hidden layer output; and

an output layer configured to;

receive the hidden layer output,

apply a first transformation to the hidden layer output to generate a first output that defines a predetermined number of bounding boxes in the input image, and

apply one or more second transformations to the hidden layer output to generate a second output that includes a respective confidence score for each of the bounding boxes that represents a likelihood that the bounding box contains an image of an object.

2. The system of claim 1, wherein the first transformation is a linear transformation.

3. The system of claim 1, wherein the one or more second transformations comprise a linear transformation followed by a non-linear transformation.

4. The system of claim 1, wherein the first output comprises, for each of the bounding boxes, first values that identify coordinates of a first vertex of the bounding box in the input image and second values that identify coordinates of a second vertex of the bounding box in the input image.

5. The system of claim 1, wherein the one or more hidden neural network layers comprise one or more convolutional neural network layers and one or more fully-connected neural network layers.

6. The system of claim 1, wherein the predetermined number is an integer greater than one.

7. The system of claim 1, wherein the output layer and the one or more hidden layers have been trained through backpropagation.

8. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement an object detection neural network, the object detection neural network comprising:

one or more hidden neural network layers configured to;

receive an input image, and

process the input image to generate a hidden layer output; and

an output layer configured to;

receive the hidden layer output,

apply a first transformation to the hidden layer output to generate a first output that defines a predetermined number of bounding boxes in the input image, and

apply one or more second transformations to the hidden layer output to generate a second output that includes a respective confidence score for each of the bounding boxes that represents a likelihood that the bounding box contains an image of an object.

9. The computer storage media of claim 8, wherein the first transformation is a linear transformation.

10. The computer storage media of claim 8, wherein the one or more second transformations comprise a linear transformation followed by a non-linear transformation.

11. The computer storage media of claim 8, wherein the first output comprises, for each of the bounding boxes, first values that identify coordinates of a first vertex of the bounding box in the input image and second values that identify coordinates of a second vertex of the bounding box in the input image.

12. The computer storage media of claim 8, wherein the one or more hidden neural network layers comprise one or more convolutional neural network layers and one or more fully-connected neural network layers.

13. The computer storage media of claim 8, wherein the predetermined number is an integer greater than one.

14. The computer storage media of claim 8, wherein the output layer and the one or more hidden layers have been trained through backpropagation.

15. A method performed by one or more computers, the method comprising:

receiving an input image;

processing the input image using one or more hidden neural network layers that are configured to;

receive the input image, and

process the input image to generate a hidden layer output; and

processing the hidden layer output using an output layer that is configured to;

receive the hidden layer output,

apply a first transformation to the hidden layer output to generate a first output that defines a predetermined number of bounding boxes in the input image, and

apply one or more second transformations to the hidden layer output to generate a second output that includes a respective confidence score for each of the bounding boxes that represents a likelihood that the bounding box contains an image of an object.

16. The method of claim 15, wherein the first transformation is a linear transformation.

17. The method of claim 15, wherein the one or more second transformations comprise a linear transformation followed by a non-linear transformation.

18. The method of claim 15, wherein the first output comprises, for each of the bounding boxes, first values that identify coordinates of a first vertex of the bounding box in the input image and second values that identify coordinates of a second vertex of the bounding box in the input image.

19. The method of claim 15, wherein the one or more hidden neural network layers comprise one or more convolutional neural network layers and one or more fully-connected neural network layers.

20. The method of claim 15, wherein the predetermined number is an integer greater than one.