Neural network for object detection in images

• US 10,346,723 B2
• Filed: 11/01/2016
• Issued: 07/09/2019
• Est. Priority Date: 11/01/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Han, Wei, Yang, Jianchao, Zhang, Ning, Li, Jia
• Primary Examiner(s): Vu, Kim Y
• Assistant Examiner(s): Vanchy, Jr., Michael J
• Application Number:
  US15/340,675
• Publication Number:
  US 20180121762A1
• Time in Patent Office: 980 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

Systems, devices, media, and methods are presented for identifying and categorically labeling objects within a set of images. The systems and methods receive an image depicting an object of interest, detect at least a portion of the object of interest within the image using a multilayer object model, determine context information, and identify the object of interest included in two or more bounding boxes.

Claims

1. A device implemented method for image recognition, the method comprising:

accessing, using one or more processors of the device coupled to a memory of the device, an image depicting an object of interest and a background within a field of view;

detecting, by the one or more processors using a set of detection layers of the multilayer object model, at least a portion of the object of interest within the image;

detecting, by the one or more processors using a set of detection layers of the multilayer object model, at least a portion of the background within the image;

extracting, by the one or more processors using a set of image representation layers of the multilayer object model comprising a lower image representation layer and a higher image representation layer, context information from the portion of the background, wherein a layer output of the higher image representation layer includes the extracted context information; and

identifying, by the one or more processors, the object of interest from the detected portion of the object of interest and the context information, using the set of image representation layers of the multilayer object model, passing the layer output of the higher image representation layer including the extracted context information backward to the lower image representation layer.

2. The method of claim 1, further comprising generating, by the one or more processors configured by the multilayer object model, a set of bounding boxes within the image, and wherein generating the set of bounding boxes comprises:

identifying a set of coordinates within the image, the set of coordinates including an indication of one or more boundaries for the image;

determining a set of sizes and a set of aspect ratios for the set of bounding boxes;

determining a distribution of bounding boxes to encompass each coordinate of the set of coordinates in at least one bounding box of the set of bounding boxes; and

generating the set of bounding boxes to distribute the set of bounding boxes uniformly over the image, wherein each bounding box of the set of bounding boxes is generated with a size included in the set of sizes and an aspect ratio included in the set of aspect ratios.

3. The method of claim 2, wherein the set of bounding boxes includes at least one bounding box having a first size and a first aspect ratio and at least one bounding box having a second size and a second aspect ratio, and wherein the first size is distinct from the second size and the first aspect ratio is distinct from the second aspect ratio.

4. The method of claim 1, wherein detecting the portion of the object of interest using the set of detection layers of the multilayer object model further comprises:

generating, by the one or more processors configured by the multilayer object model, a set of bounding boxes within the image;

detecting part of the portion of the object of interest in a first bounding box of the set of bounding boxes using a first detection layer, the first detection layer associated with a first scale corresponding to the first bounding box; and

detecting part of the portion of the object of interest in a second bounding box of the set of bounding boxes using a second detection layer, the second detection layer associated with a second scale corresponding to the second bounding box.

5. The method of claim 4, wherein the first detection layer generates a first confidence score and a first set of coordinates for the part of the object of interest depicted within the first bounding box and the second detection layer generates a second confidence score and a second set of coordinates for the part of the object of interest depicted within the second bounding box.

6. The method of claim 1, wherein the layer output of the higher image representation layer is passed backward to the lower image representation layer using a deconvolution layer of the multilayer object model.

7. The method of claim 1 further comprising training the multilayer object model by:

accessing a set of training images, each training image depicting a known object of interest;

identifying a set of bounding boxes within the set of training images, each bounding box having a set of coordinates identifying a location within a training image, a resolution, and a label;

determining the resolution of a bounding box exceeds a specified box resolution;

rescaling the resolution of the bounding box to match the specified box resolution by identifying a center point of the bounding box and cropping portions of the bounding box outside of the specified box resolution with respect to the center point;

initializing one or more model parameters; and

iteratively adjusting the one or more model parameters to until a change in averaged loss function values resulting from iterations of the one or more model parameters falls below a change threshold.

8. The method of claim 1, further comprising training the multilayer object model by:

accessing a set of training images, each training image depicting a known object of interest; and

detecting the known objects of interest within the set of training images using the multilayer object model, the detection performed with one or more layers of the multilayer object model set at a first resolution, and one or more layers of the multilayer object model set at a second resolution.

9. The method of claim 8, wherein detecting the known objects of interest further comprises:

iteratively adjusting one or more model parameters until a change in averaged loss function values falls below a change threshold, the averaged loss function obtained for two or more instances of a training image of the set of training images, each of the two or more instances of the training image having distinct resolutions.

10. The method of claim 1, further comprising training the set of detection layers of the multilayer object model by:

accessing a set of training images, each training image depicting a known object of interest and containing at least one bounding box comprising a tuple indicating coordinates of the bounding box within a training image and a classification label for the known object of interest; and

iteratively initializing one or more image representation layers of the set of image representation layers and one or more detection layers of the set of detection layers and adjusting one or more parameters of the one or more image representation layers and the one or more detection layers until a change in averaged loss function values resulting from iterations of the one or more model parameters falls below a change threshold.

11. A system comprising:

one or more processors; and

a processor-readable storage device coupled to the one or more processors, the processor-readable storage device storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

accessing an image depicting an object of interest and a background within a field of view;

detecting, by the one or more processors using a set of detection layers of the multilayer object model, at least a portion of the object of interest within the image;

detecting, by the one or more processors using a set of detection layers of the multilayer object model, at least a portion of the background within the image;

extracting, by the one or more processors using a set of image representation layers of the multilayer object model comprising a lower image representation layer and a higher image representation layer, context information from the portion of the background, wherein a layer output of the higher image representation layer includes the extracted context information; and

identifying, by the one or more processors, the object of interest from the detected portion of the object of interest and the context information, using the set of image representation layers of the multilayer object model, by passing the layer output of the higher image representation layer including the extracted context information backward to the lower image representation layer.

12. The system of claim 11, wherein the operations further comprise:

generating, by the one or more processors configured by the multilayer object model, a set of bounding boxes within the image, and wherein generating the set of bounding boxes comprises;

identifying a set of coordinates within the image, the set of coordinates including an indication of one or more boundaries for the image;

determining a set of sizes and a set of aspect ratios for the set of bounding boxes;

determining a distribution of bounding boxes to encompass each coordinate of the set of coordinates in at least one bounding box of the set of bounding boxes; and

generating the set of bounding boxes to distribute the set of bounding boxes uniformly over the image, wherein each bounding box of the set of bounding boxes is generated with a size included in the set of sizes and an aspect ratio included in the set of aspect ratios.

13. The system of claim 11, wherein detecting the portion of the object of interest using the set of detection layers of the multilayer object model further comprises:

detecting part of the portion of the object of interest in a first bounding box of the two or more bounding boxes using a first detection layer, the first detection layer associated with a first scale corresponding to the first bounding box; and

detecting part of the portion of the object of interest in a second bounding box of the two or more bounding boxes using the second detection layer, the second detection layer associated with a second scale corresponding to the second bounding box.

14. The system of claim 11, wherein the layer output of the higher image representation layer is passed backward to the lower image representation layer using a deconvolution layer of the multilayer object model.

15. The system of claim 11 further comprising training the multilayer object model by:

accessing a set of training images, each training image depicting a known object of interest;

identifying a set of bounding boxes within the set of training images, each bounding box having a set of coordinates identifying a location within a training image, a resolution, and a label;

determining the resolution of a bounding box exceeds a specified box resolution;

rescaling the resolution of the bounding box to match the specified box resolution by identifying a center point of the bounding box and cropping portions of the bounding box outside of the specified box resolution with respect to the center point;

initializing one or more model parameters; and

iteratively adjusting the one or more model parameters until a change in averaged loss function values resulting from iterations of the one or more model parameters falls below a change threshold.

16. A processor-readable storage device storing processor-executable instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

accessing an image depicting an object of interest and a background within a field of view;

detecting, by the one or more processors using a set of detection layers of the multilayer object model, at least a portion of the object of interest within the image;

detecting, by the one or more processors using a set of detection layers of the multilayer object model, at least a portion of the background within the image;

extracting, by the one or more processors using a set of image representation layers of the multilayer object model comprising a lower image representation layer and a higher image representation layer, context information from the portion of the background, wherein a layer output of the higher image representation layer includes the extracted context information; and

identifying, by the one or more processors, the object of interest from the detected portion of the object of interest and the context information, using the set of image representation layers of the multilayer object model, by passing the layer output of the higher image representation layer including the extracted context information backward to the lower image representation layer.

17. The processor-readable storage device of claim 16, wherein the operations further comprise:

generating, by the one or more processors configured by the multilayer object model, a set of bounding boxes within the image, and wherein generating the set of bounding boxes comprises;

identifying a set of coordinates within the image, the set of coordinates including an indication of one or more boundaries for the image;

determining a set of sizes and a set of aspect ratios for the set of bounding boxes;

determining a distribution of bounding boxes to encompass each coordinate of the set of coordinates in at least one bounding box of the set of bounding boxes; and

generating the set of bounding boxes to distribute the set of bounding boxes uniformly over the image, wherein each bounding box of the set of bounding boxes is generated with a size included in the set of sizes and an aspect ratio included in the set of aspect ratios.

18. The processor-readable storage device of claim 16, wherein detecting the portion of the object of interest using the set of detection layers of the multilayer object model further comprises:

detecting part of the portion of the object of interest in a first bounding box of the two or more bounding boxes using the first detection layer, the first detection layer associated with a first scale corresponding to the first bounding box; and

detecting part of the portion of the object of interest in a second bounding box of the two or more bounding boxes using the second detection layer, the second detection layer associated with a second scale corresponding to the second bounding box.

19. The processor-readable storage device of claim 16, wherein the layer output of the higher image representation layer is passed backward to the lower image representation layer using a deconvolution layer of the multilayer object model.

20. The processor-readable storage device of claim 16 further comprising training the multilayer object model by:

accessing a set of training images, each training image depicting a known object of interest;

identifying a set of bounding boxes within the set of training images, each bounding box having a set of coordinates identifying a location within a training image, a resolution, and a label;

determining the resolution of a bounding box exceeds a specified box resolution;

rescaling the resolution of the bounding box to match the specified box resolution by identifying a center point of the bounding box and cropping portions of the bounding box outside of the specified box resolution with respect to the center point;

initializing one or more model parameters; and

iteratively adjusting the one or more model parameters until a change in averaged loss function values resulting from iterations of the one or more model parameters falls below a change threshold.