OBJECT DETECTION AND CLASSIFICATION IN IMAGES

• US 20170206431A1
• Filed: 01/20/2016
• Published: 07/20/2017
• Est. Priority Date: 01/20/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Girshick, Ross, Ren, Shaoqing, Sun, Jian, He, Kaiming
• Granted Patent: US 9,858,496 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 30/248 involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q" G06V30/242 takes precedence

Abstract

Systems, methods, and computer-readable media for providing fast and accurate object detection and classification in images are described herein. In some examples, a computing device can receive an input image. The computing device can process the image, and generate a convolutional feature map. In some configurations, the convolutional feature map can be processed through a Region Proposal Network (RPN) to generate proposals for candidate objects in the image. In various examples, the computing device can process the convolutional feature map with the proposals through a Fast Region-Based Convolutional Neural Network (FRCN) proposal classifier to determine a class of each object in the image and a confidence score associated therewith. The computing device can then provide a requestor with an output including the object classification and/or confidence score.

Claims

1. A method comprising:

receiving an input image;

generating a convolutional feature map;

identifying, by a first type of neural network, a candidate object in the input image;

determining, by a second type of neural network, a category of the candidate object, wherein the first type of neural network and the second type of neural network share at least one algorithm; and

assigning a confidence score to the category of the candidate object.

7. A method as claim 1 recites, wherein the convolutional feature map is generated by a Zeiler and Fergus model or a Simonyan and Zisserman model deep convolutional neural network.

8. A method as claim 1 recites, further comprising training the convolutional feature map, the first type of neural network, and the second type of neural network using at least one of:

stochastic gradient descent;

or

back-propagation.

2. A method as 1 recites, wherein the first type of neural network comprises a translation invariant component configured to:

classify an anchor based on overlap with a ground-truth box; and

predict a shift and a scale of the anchor.

3. A method as 1 recites, wherein the identifying the candidate object in the input image comprises:

generating one or more anchors at a point of the input image;

determining an overlap of individual ones of the one or more anchors to a ground-truth box;

assigning a label to each anchor of the one or more anchors based at least in part on the overlap;

assigning a score to the label based at least in part on the overlap; and

identifying the candidate object at the point based at least in part on the score.

4. A method as claim 3 recites, wherein the identifying the candidate object in the input image further comprises:

identifying an anchor corresponding to a highest score, the highest score corresponding to a percentage of the overlap;

shifting the anchor corresponding to the highest score to better define the candidate object; and

scaling the anchor corresponding to the highest score to better define the candidate object.

6. A method as claim 3 recites, wherein the label is positive when the overlap exceeds a threshold level.

5. A method as 3 recites, wherein the generating the one or more anchors at the point of the input image comprises generating a set of anchor boxes, the set anchor boxes having three scales and three aspect ratios.

9. A system comprising:

a processor; and

a computer-readable medium including instructions for an object detection and classification network, for actuation by the processor, the object detection and classification network comprising;

an initial processing module configured to input an image and generate a convolutional feature map;

an object proposal module configured to generate a proposal corresponding to a candidate object in the image;

a proposal classifier module configured to assign a category associated with the candidate object, wherein the object proposal module and the proposal classifier module share at least one convolutional layer.

10. A system as claim 9 recites, wherein the proposal classifier module is further configured to assign a confidence score to the classification.

11. A system as claim 9 recites, wherein the object proposal module comprises a translation invariant component configured to:

classify an anchor based on overlap with a ground-truth box; and

predict a shift and a scale of the anchor.

12. A system as claim 9 recites, wherein the object proposal module is further configured to:

generate one or more anchors at a point of the image;

determine an overlap of each anchor of the one or more anchors to a ground-truth box;

assign a label to each anchor of the one or more anchor based at least in part on the overlap;

assign a score to the label based at least in part on the overlap;

select an anchor with a highest score; and

generate the proposal based at least in part on the highest score.

13. A system as claim 9 recites, wherein the object proposal module is further configured to:

identify an anchor corresponding to a highest score, the highest score corresponding to a percentage of the overlap;

shift the anchor corresponding to the highest score to better define the candidate object;

or

scale the anchor corresponding to the highest score to better define the candidate object.

14. A system as claim 9 recites, further comprising a machine learning module configured to:

train one or more parameters of the initial processing module and the object proposal module to generate one or more proposals on a training image; and

train one or more parameters of the proposal classifier module to assign a category to each of the one or more proposals on the training image.

15. A system as claim 14 recites, wherein the machine learning module is further configured to train the one or more parameters of the initial processing module, the object proposal module, and the proposal classifier module using one or more of:

stochastic gradient descent;

or

back-propagation.

16. A computer storage medium having thereon computer-executable instructions, the computer-executable instructions responsive to execution configuring a device to perform operations comprising:

receiving an input image;

generating a convolutional feature map;

identifying, by a neural network, a candidate object in the input image;

determining, by a proposal classifier sharing an algorithm with the neural network, a category of the candidate object; and

assigning, by the proposal classifier, a confidence score to the category of the candidate object.

17. A computer storage medium as claim 16 recites, wherein the neural network is a region proposal network and the proposal classifier is a component of a fast region based convolutional neural network.

18. A computer storage medium as claim 16 recites, further comprising:

generating one or more anchors at a point of the input image;

determining an overlap of individual ones of the one or more anchors to a ground-truth box;

assigning a label to each anchor of the one or more anchors based at least in part on the overlap;

assigning a score to the label based at least in part on the overlap; and

identifying the candidate object at the point based at least in part on the score.

19. A computer storage medium as claim 16 recites, wherein the neural network comprises a translation invariant component configured to:

classify an anchor based on overlap with a ground-truth box; and

predict a shift and a scale of the anchor.

20. A computer storage medium as claim 16 recites, further comprising training the convolution feature map, the neural network, and the proposal classifier using at least one of:

stochastic gradient descent;

or

back-propagation.