Subcategory-aware convolutional neural networks for object detection

• US 9,965,719 B2
• Filed: 11/03/2016
• Issued: 05/08/2018
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Choi, Wongun, Lin, Yuanqing, Xiang, Yu, Savarese, Silvio
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US15/342,823
• Publication Number:
  US 20170124415A1
• Time in Patent Office: 551 Days
• Field of Search: 382100, 382103-107, 382155-160, 382162, 382168, 382173, 382181, 382190, 382219, 382224, 382232, 382254, 382276, 382286-291, 382305, 382312, 382205, 706 15
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

A computer-implemented method for detecting objects by using subcategory-aware convolutional neural networks (CNNs) is presented. The method includes generating object region proposals from an image by a region proposal network (RPN) which utilizes subcategory information, and classifying and refining the object region proposals by an object detection network (ODN) that simultaneously performs object category classification, subcategory classification, and bounding box regression. The image is an image pyramid used as input to the RPN and the ODN. The RPN and the ODN each include a feature extrapolating layer to detect object categories with scale variations among the objects.

Claims

1. A computer-implemented method for detecting objects by using subcategory-aware convolutional neural networks (CNNs), the method comprising:

generating object region proposals from an image by a region proposal network (RPN) which utilizes subcategory information; and

classifying and refining the object region proposals by an object detection network (ODN) that simultaneously performs object category classification, subcategory classification, and bounding box regression,

wherein the RPN and the ODN each include a feature extrapolating layer to detect object categories with scale variations among the objects.

2. The method of claim 1, wherein the image is an image pyramid used as input to the RPN and the ODN.

3. The method of claim 1, wherein the RPN includes a subcategory convolution layer such that each filter in the convolution layer corresponds to an object subcategory.

4. The method of claim 3, wherein the subcategory convolution layer outputs heats maps indicating a presence of subcategories at a specific location and a specific scale.

5. The method of claim 4, wherein the heat maps are used to create a region of interest (RoI) generating layer for generating object candidates by thresholding the heat maps.

6. The method of claim 5, wherein the RPN terminates at two layers, one layer that outputs softmax probability estimates over object subcategories and another layer that refines RoI location with a bounding box regressor.

7. A system for detecting objects by using subcategory-aware convolutional neural networks (CNNs), the system comprising:

a memory; and

a hardware computer processor operatively coupled to the memory, the processor being configured for;

generating object region proposals from an image by a region proposal network (RPN) which utilizes subcategory information; and

classifying and refining the object region proposals by an object detection network (ODN) that simultaneously performs object category classification, subcategory classification, and bounding box regression,

wherein the RPN and the ODN each include a feature extrapolating layer to detect object categories with scale variations among the objects.

8. The system of claim 7, wherein the image is an image pyramid used as input to the RPN and the ODN.

9. The system of claim 7, wherein the RPN includes a subcategory convolution layer such that each filter in the convolution layer corresponds to an object subcategory.

10. The system of claim 9, wherein the subcategory convolution layer outputs heats maps indicating a presence of subcategories at a specific location and a specific scale.

11. The system of claim 10, wherein the heat maps are used to create a region of interest (RoI) generating layer for generating object candidates by thresholding the heat maps.

12. The system of claim 11, wherein the RPN terminates at two layers, one layer that outputs softmax probability estimates over object subcategories and another layer that refines RoI location with a bounding box regressor.

13. A non-transitory computer-readable storage medium comprising a computer-readable program for detecting objects by using subcategory-aware convolutional neural networks (CNNs), wherein the computer-readable program when executed on a computer, using a hardware computer processor coupled to the non-transitory computer-readable storage medium, causes the computer to perform the steps of:

generating object region proposals from an image by a region proposal network (RPN) which utilizes subcategory information; and

classifying and refining the object region proposals by an object detection network (ODN) that simultaneously performs object category classification, subcategory classification, and bounding box regression,

wherein the RPN and the ODN each include a feature extrapolating layer to detect object categories with scale variations among the objects.

14. The non-transitory computer-readable storage medium of claim 13, wherein the image is an image pyramid used as input to the RPN and the ODN.

15. The non-transitory computer-readable storage medium of claim 13, wherein the RPN includes a subcategory convolution layer such that each filter in the convolution layer corresponds to an object subcategory.

16. The non-transitory computer-readable storage medium of claim 15, wherein the subcategory convolution layer outputs heats maps indicating a presence of subcategories at a specific location and a specific scale.

17. The non-transitory computer-readable storage medium of claim 16, wherein the heat maps are used to create a region of interest (RoI) generating layer for generating object candidates by thresholding the heat maps.