Object detection using deep neural networks

• US 9,275,308 B2
• Filed: 05/27/2014
• Issued: 03/01/2016
• Est. Priority Date: 05/31/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Szegedy, Christian, Erhan, Dumitru, Toshev, Alexander Toshkov
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US14/288,194
• Publication Number:
  US 20150170002A1
• Time in Patent Office: 644 Days
• Field of Search: 382/156, 382/159, 382/103
• US Class Current: 1/1
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for detecting objects in images. One of the methods includes receiving an input image. A full object mask is generated by providing the input image to a first deep neural network object detector that produces a full object mask for an object of a particular object type depicted in the input image. A partial object mask is generated by providing the input image to a second deep neural network object detector that produces a partial object mask for a portion of the object of the particular object type depicted in the input image. A bounding box is determined for the object in the image using the full object mask and the partial object mask.

Claims

1. A computer-implemented method comprising:

receiving, by one or more computers, an input image;

generating, by one or more computers, a full object mask by providing the input image to a first deep neural network object detector that produces a full object mask for an object of a particular object type depicted in the input image, wherein the full object mask identifies regions of the input image that correspond to the object and regions of the input image that do not correspond to the object;

generating, by one or more computers, a partial object mask by providing the input image to a second deep neural network object detector that produces a partial object mask for a portion of the object of the particular object type depicted in the input image; and

determining, by one or more computers, a bounding box for the object in the image using the full object mask and the partial object mask.

2. The method of claim 1, wherein the portion of the object corresponds to the bottom portion, the top portion, the left portion, or the right portion of the object.

3. The method of claim 1, wherein the bounding box has a partial bounding box corresponding to the partial object mask, and wherein determining the bounding box for the object in the image using the full object mask and the partial object mask comprises determining a bounding box that has a best fit, among a plurality of candidate bounding boxes, of the full bounding box with the full object mask and the partial bounding box with the partial object mask.

4. The method of claim 1, wherein determining a bounding box for the object in the image using the full object mask and the partial object mask comprises:

computing a score for each of a plurality of candidate bounding boxes based on a first measure of overlap between the bounding box and the full object mask and a second measure of overlap between the bounding box and the partial object mask; and

determining a bounding box having a highest score.

5. The method of claim 4, wherein the score for a candidate bounding box bb is given by:

6. The method of claim 4, further comprising:

generating a second full object mask by providing a portion of the image corresponding to the bounding box to the first deep neural network object detector;

generating a second partial object mask by providing the portion of the image corresponding to the bounding box to the second deep neural network object detector;

computing a score for each of a second plurality of candidate bounding boxes based on a third measure of overlap between each bounding box and the second full object mask and a fourth measure of overlap between each bounding box and the second partial object mask; and

determining a refined bounding box of the second plurality of candidate bounding boxes having a highest score.

7. The method of claim 1, further comprising:

determining full and partial object masks for subwindows of each of multiple subwindow scales; and

merging object masks determined at same values of the subwindow scales.

8. The method of claim 7, wherein merging the object masks determined at the multiple values of the scale s comprises averaging the object masks.

9. The method of claim 1, wherein determining full and partial object masks for subwindows of each of multiple subwindow scales comprises determining full and partial object masks for no more than 50 subwindows.

10. The method of claim 1, further comprising:

generating a predicted object type by providing the input image to a third deep neural network classifier that produces a predicted object type for a portion of the image corresponding to the bounding box;

determining that the predicted object type does not match the particular object type; and

removing the bounding box from consideration.

11. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving an input image;

generating a full object mask by providing the input image to a first deep neural network object detector that produces a full object mask for an object of a particular object type depicted in the input image, wherein the full object mask identifies regions of the input image that correspond to the object and regions of the input image that do not correspond to the object;

generating a partial object mask by providing the input image to a second deep neural network object detector that produces a partial object mask for a portion of the object of the particular object type depicted in the input image; and

determining a bounding box for the object in the image using the full object mask and the partial object mask.

12. The system of claim 11, wherein the portion of the object corresponds to the bottom portion, the top portion, the left portion, or the right portion of the object.

13. The system of claim 11, wherein the bounding box has a partial bounding box corresponding to the partial object mask, and wherein determining the bounding box for the object in the image using the full object mask and the partial object mask comprises determining a bounding box that has a best fit, among a plurality of candidate bounding boxes, of the full bounding box with the full object mask and the partial bounding box with the partial object mask.

14. The system of claim 11, wherein determining a bounding box for the object in the image using the full object mask and the partial object mask comprises:

computing a score for each of a plurality of candidate bounding boxes based on a first measure of overlap between the bounding box and the full object mask and a second measure of overlap between the bounding box and the partial object mask; and

determining a bounding box having a highest score.

15. The system of claim 14, wherein the score for a candidate bounding box bb is given by:

16. The system of claim 14, wherein the operations further comprise:

generating a second full object mask by providing a portion of the image corresponding to the bounding box to the first deep neural network object detector;

generating a second partial object mask by providing the portion of the image corresponding to the bounding box to the second deep neural network object detector;

computing a score for each of a second plurality of candidate bounding boxes based on a third measure of overlap between each bounding box and the second full object mask and a fourth measure of overlap between each bounding box and the second partial object mask; and

determining a refined bounding box of the second plurality of candidate bounding boxes having a highest score.

17. The system of claim 11, wherein the operations further comprise:

determining full and partial object masks for subwindows of each of multiple subwindow scales; and

merging object masks determined at same values of the subwindow scales.

18. The system of claim 17, wherein merging the object masks determined at the multiple values of the scale s comprises averaging the object masks.

19. The system of claim 11, wherein determining full and partial object masks for subwindows of each of multiple subwindow scales comprises determining full and partial object masks for no more than 50 subwindows.

20. The system of claim 11, wherein the operations further comprise:

generating a predicted object type by providing the input image to a third deep neural network classifier that produces a predicted object type for a portion of the image corresponding to the bounding box;

determining that the predicted object type does not match the particular object type; and

removing the bounding box from consideration.

21. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving an input image;

generating a full object mask by providing the input image to a first deep neural network object detector that produces a full object mask for an object of a particular object type depicted in the input image, wherein the full object mask identifies regions of the input image that correspond to the object and regions of the input image that do not correspond to the object;

generating a partial object mask by providing the input image to a second deep neural network object detector that produces a partial object mask for a portion of the object of the particular object type depicted in the input image; and

determining a bounding box for the object in the image using the full object mask and the partial object mask.