Learning method and learning device for object detector based on CNN using 1×1 convolution to be used for hardware optimization, and testing method and testing device using the same
First Claim
1. A method for learning parameters of an object detector based on a CNN, comprising steps of:
- (a) a learning device, if at least one training image is acquired, (i) instructing one or more convolutional layers to generate at least one initial feature map by applying one or more convolution operations to the training image, (ii) instructing an RPN to generate one or more proposals corresponding to each of one or more objects in the training image by using the initial feature map, and (iii) (iii-1) instructing a pooling layer to apply one or more pooling operations to each region, corresponding to each of the proposals, on the initial feature map, to thereby generate pooled feature maps per each of the proposals, and instructing a first transposing layer to concatenate each of pixels, per each of the proposals, in each of corresponding same locations on the pooled feature maps per each of the proposals, to thereby generate an integrated feature map, or (iii-2) instructing the pooling layer to apply the pooling operations to each region, corresponding to each of the proposals, on the initial feature map, to thereby generate the pooled feature maps per each of the proposals, and instructing the pooling layer to concatenate each of the pixels, per each of the proposals, in each of the corresponding same locations on the pooled feature maps per each of the proposals, to thereby generate the integrated feature map;
(b) the learning device instructing a first 1×
1 convolutional layer to apply a 1×
1 convolution operation to the integrated feature map, to thereby generate a first adjusted feature map whose volume is adjusted, and instructing a second 1×
1 convolutional layer to apply the 1×
1 convolution operation to the first adjusted feature map, to thereby generate a second adjusted feature map whose volume is adjusted; and
(c) the learning device (c1) (i) instructing a second transposing layer to divide the second adjusted feature map by each of the pixels, to thereby generate pixel-wise feature maps per each of the proposals, and instructing a classifying layer to generate object class information on each of the proposals by using the pixel-wise feature maps per each of the proposals, or (ii) instructing the classifying layer to divide the second adjusted feature map by each of the pixels, to thereby generate the pixel-wise feature maps per each of the proposals, and instructing the classifying layer to generate the object class information on each of the proposals by using the pixel-wise feature maps per each of the proposals, (c2) instructing a detecting layer to generate object detection information corresponding to the objects in the training image by referring to the object class information and the pixel-wise feature maps per each of the proposals, and (c3) instructing a detection loss layer to calculate one or more object detection losses by referring to the object detection information and its corresponding GT, to thereby learn at least part of parameters of the second 1×
1 convolutional layer, the first 1×
1 convolutional layer, and the convolutional layers by backpropagating the object detection losses.
1 Assignment
0 Petitions
Accused Products
Abstract
A method for learning parameters of an object detector based on a CNN is provided to be used for hardware optimization which satisfies KPI. The method includes steps of: a learning device instructing a first transposing layer or a pooling layer to generate an integrated feature map by concatenating pixels per each proposal; and instructing a second transposing layer or a classifying layer to divide volume-adjusted feature map, generated by using the integrated feature map, by pixel, and instructing the classifying layer to generate object class information. By this method, size of a chip can be decreased as convolution operations and fully connected layer operations can be performed by a same processor. Accordingly, there are advantages such as no need to build additional lines in a semiconductor manufacturing process, power saving, more space to place other modules instead of an FC module in a die, and the like.
-
Citations
28 Claims
-
1. A method for learning parameters of an object detector based on a CNN, comprising steps of:
-
(a) a learning device, if at least one training image is acquired, (i) instructing one or more convolutional layers to generate at least one initial feature map by applying one or more convolution operations to the training image, (ii) instructing an RPN to generate one or more proposals corresponding to each of one or more objects in the training image by using the initial feature map, and (iii) (iii-1) instructing a pooling layer to apply one or more pooling operations to each region, corresponding to each of the proposals, on the initial feature map, to thereby generate pooled feature maps per each of the proposals, and instructing a first transposing layer to concatenate each of pixels, per each of the proposals, in each of corresponding same locations on the pooled feature maps per each of the proposals, to thereby generate an integrated feature map, or (iii-2) instructing the pooling layer to apply the pooling operations to each region, corresponding to each of the proposals, on the initial feature map, to thereby generate the pooled feature maps per each of the proposals, and instructing the pooling layer to concatenate each of the pixels, per each of the proposals, in each of the corresponding same locations on the pooled feature maps per each of the proposals, to thereby generate the integrated feature map; (b) the learning device instructing a first 1×
1 convolutional layer to apply a 1×
1 convolution operation to the integrated feature map, to thereby generate a first adjusted feature map whose volume is adjusted, and instructing a second 1×
1 convolutional layer to apply the 1×
1 convolution operation to the first adjusted feature map, to thereby generate a second adjusted feature map whose volume is adjusted; and(c) the learning device (c1) (i) instructing a second transposing layer to divide the second adjusted feature map by each of the pixels, to thereby generate pixel-wise feature maps per each of the proposals, and instructing a classifying layer to generate object class information on each of the proposals by using the pixel-wise feature maps per each of the proposals, or (ii) instructing the classifying layer to divide the second adjusted feature map by each of the pixels, to thereby generate the pixel-wise feature maps per each of the proposals, and instructing the classifying layer to generate the object class information on each of the proposals by using the pixel-wise feature maps per each of the proposals, (c2) instructing a detecting layer to generate object detection information corresponding to the objects in the training image by referring to the object class information and the pixel-wise feature maps per each of the proposals, and (c3) instructing a detection loss layer to calculate one or more object detection losses by referring to the object detection information and its corresponding GT, to thereby learn at least part of parameters of the second 1×
1 convolutional layer, the first 1×
1 convolutional layer, and the convolutional layers by backpropagating the object detection losses. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A method for testing an object detector based on a CNN, comprising steps of:
-
(a) on condition that a learning device (1) (i) has instructed one or more convolutional layers to generate at least one initial feature map for training by applying one or more convolution operations to at least one training image, (ii) has instructed an RPN to generate one or more proposals for training corresponding to each of one or more objects for training in the training image by using the initial feature map for training, and (iii) (iii-1) has instructed a pooling layer to apply one or more pooling operations to each region, corresponding to each of the proposals for training, on the initial feature map for training, to thereby generate pooled feature maps for training per each of the proposals for training, and has instructed a first transposing layer to concatenate each of pixels, per each of the proposals for training, in each of corresponding same locations on the pooled feature maps for training per each of the proposals for training, to thereby generate an integrated feature map for training, or (iii-2) has instructed the pooling layer to apply the pooling operations to each region, corresponding to each of the proposals for training, on the initial feature map for training, to thereby generate the pooled feature maps for training per each of the proposals for training, and has instructed the pooling layer to concatenate each of the pixels, per each of the proposals for training, in each of the corresponding same locations on the pooled feature maps for training per each of the proposals for training, to thereby generate the integrated feature map for training, (2) has instructed a first 1×
1 convolutional layer to apply a 1×
1 convolution operation to the integrated feature map for training, to thereby generate a first adjusted feature map for training whose volume is adjusted, and has instructed a second 1×
1 convolutional layer to apply the 1×
1 convolution operation to the first adjusted feature map for training, to thereby generate a second adjusted feature map for training whose volume is adjusted, and (3) (3-1) (i) has instructed a second transposing layer to divide the second adjusted feature map for training by each of the pixels, to thereby generate pixel-wise feature maps for training per each of the proposals for training, and has instructed a classifying layer to generate object class information for training on each of the proposals for training by using the pixel-wise feature maps for training per each of the proposals for training, or (ii) has instructed the classifying layer to divide the second adjusted feature map for training by each of the pixels, to thereby generate the pixel-wise feature maps for training per each of the proposals for training, and has instructed the classifying layer to generate the object class information for training on each of the proposals for training by using the pixel-wise feature maps for training per each of the proposals for training, (3-2) has instructed a detecting layer to generate object detection information for training corresponding to the objects for training in the training image by referring to the object class information for training and the pixel-wise feature maps for training per each of the proposals for training, and (3-3) has instructed a detection loss layer to calculate one or more object detection losses by referring to the object detection information for training and its corresponding GT, to thereby learn at least part of parameters of the second 1×
1 convolutional layer, the first 1×
1 convolutional layer, and the convolutional layers by backpropagating the object detection losses;
a testing device, if at least one test image is acquired, (i) instructing the convolutional layers to generate at least one initial feature map for testing by applying the convolution operations to the test image, (ii) instructing the RPN to generate one or more proposals for testing corresponding to each of one or more objects for testing in the test image by using the initial feature map for testing, and (iii) (iii-1) instructing the pooling layer to apply the pooling operations to each region, corresponding to each of the proposals for testing, on the initial feature map for testing, to thereby generate pooled feature maps for testing per each of the proposals for testing, and instructing a first transposing layer to concatenate each of pixels, per each of the proposals for testing, in each of corresponding same locations on the pooled feature maps for testing per each of the proposals for testing, to thereby generate an integrated feature map for testing, or (iii-2) instructing the pooling layer to apply the pooling operations to each region, corresponding to each of the proposals for testing, on the initial feature map for testing, to thereby generate the pooled feature maps for testing per each of the proposals for testing, and instructing the pooling layer to concatenate each of the pixels, per each of the proposals for testing, in each of the corresponding same locations on the pooled feature maps for testing per each of the proposals for testing, to thereby generate the integrated feature map for testing;(b) the testing device instructing the first 1×
1 convolutional layer to apply the 1×
1 convolution operation to the integrated feature map for testing, to thereby generate a first adjusted feature map for testing whose volume is adjusted, and instructing the second 1×
1 convolutional layer to apply the 1×
1 convolution operation to the first adjusted feature map for testing, to thereby generate a second adjusted feature map for testing whose volume is adjusted; and(c) the testing device (c1) (i) instructing the second transposing layer to divide the second adjusted feature map for testing by each of the pixels, to thereby generate pixel-wise feature maps for testing per each of the proposals for testing, and instructing the classifying layer to generate object class information for testing on each of the proposals for testing by using the pixel-wise feature maps for testing per each of the proposals for testing, or (ii) instructing the classifying layer to divide the second adjusted feature map for testing by each of the pixels, to thereby generate the pixel-wise feature maps for testing per each of the proposals for testing, and instructing the classifying layer to generate the object class information for testing on each of the proposals for testing by using the pixel-wise feature maps for testing per each of the proposals for testing, and (c2) instructing the detecting layer to generate object detection information for testing corresponding to the objects for testing in the test image by referring to the object class information for testing and the pixel-wise feature maps for testing per each of the proposals for testing. - View Dependent Claims (9, 10, 11, 12, 13, 14)
-
-
15. A learning device for learning parameters of an object detector based on a CNN, comprising:
-
at least one memory that stores instructions; and at least one processor configured to execute the instructions to;
perform processes of (I) (i) instructing one or more convolutional layers to generate at least one initial feature map by applying one or more convolution operations to at least one training image, (ii) instructing an RPN to generate one or more proposals corresponding to each of one or more objects in the training image by using the initial feature map, and (iii) (iii-1) instructing a pooling layer to apply one or more pooling operations to each region, corresponding to each of the proposals, on the initial feature map, to thereby generate pooled feature maps per each of the proposals, and instructing a first transposing layer to concatenate each of pixels, per each of the proposals, in each of corresponding same locations on the pooled feature maps per each of the proposals, to thereby generate an integrated feature map, or (iii-2) instructing the pooling layer to apply the pooling operations to each region, corresponding to each of the proposals, on the initial feature map, to thereby generate the pooled feature maps per each of the proposals, and instructing the pooling layer to concatenate each of the pixels, per each of the proposals, in each of the corresponding same locations on the pooled feature maps per each of the proposals, to thereby generate the integrated feature map, (II) instructing a first 1×
1 convolutional layer to apply a 1×
1 convolution operation to the integrated feature map, to thereby generate a first adjusted feature map whose volume is adjusted, and instructing a second 1×
1 convolutional layer to apply the 1×
1 convolution operation to the first adjusted feature map, to thereby generate a second adjusted feature map whose volume is adjusted, and (III) (III-1) (i) instructing a second transposing layer to divide the second adjusted feature map by each of the pixels, to thereby generate pixel-wise feature maps per each of the proposals, and instructing a classifying layer to generate object class information on each of the proposals by using the pixel-wise feature maps per each of the proposals, or (ii) instructing the classifying layer to divide the second adjusted feature map by each of the pixels, to thereby generate the pixel-wise feature maps per each of the proposals, and instructing the classifying layer to generate the object class information on each of the proposals by using the pixel-wise feature maps per each of the proposals, (III-2) instructing a detecting layer to generate object detection information corresponding to the objects in the training image by referring to the object class information and the pixel-wise feature maps per each of the proposals, and (III-3) instructing a detection loss layer to calculate one or more object detection losses by referring to the object detection information and its corresponding GT, to thereby learn at least part of parameters of the second 1×
1 convolutional layer, the first 1×
1 convolutional layer, and the convolutional layers by backpropagating the object detection losses. - View Dependent Claims (16, 17, 18, 19, 20, 21)
-
-
22. A testing device for testing an object detector based on a CNN, comprising:
-
at least one memory that stores instructions; and at least one processor, on condition that a learning device (1) (i) has instructed one or more convolutional layers to generate at least one initial feature map for training by applying one or more convolution operations to at least one training image, (ii) has instructed an RPN to generate one or more proposals for training corresponding to each of one or more objects for training in the training image by using the initial feature map for training, and (iii) (iii-1) has instructed a pooling layer to apply one or more pooling operations to each region, corresponding to each of the proposals for training, on the initial feature map for training, to thereby generate pooled feature maps for training per each of the proposals for training, and has instructed a first transposing layer to concatenate each of pixels, per each of the proposals for training, in each of corresponding same locations on the pooled feature maps for training per each of the proposals for training, to thereby generate an integrated feature map for training, or (iii-2) has instructed the pooling layer to apply the pooling operations to each region, corresponding to each of the proposals for training, on the initial feature map for training, to thereby generate the pooled feature maps for training per each of the proposals for training, and has instructed the pooling layer to concatenate each of the pixels, per each of the proposals for training, in each of the corresponding same locations on the pooled feature maps for training per each of the proposals for training, to thereby generate the integrated feature map for training, (2) has instructed a first 1×
1 convolutional layer to apply a 1×
1 convolution operation to the integrated feature map for training, to thereby generate a first adjusted feature map for training whose volume is adjusted, and has instructed a second 1×
1 convolutional layer to apply the 1×
1 convolution operation to the first adjusted feature map for training, to thereby generate a second adjusted feature map for training whose volume is adjusted, and (3) (3-1) (i) has instructed a second transposing layer to divide the second adjusted feature map for training by each of the pixels, to thereby generate pixel-wise feature maps for training per each of the proposals for training, and has instructed a classifying layer to generate object class information for training on each of the proposals for training by using the pixel-wise feature maps for training per each of the proposals for training, or (ii) has instructed the classifying layer to divide the second adjusted feature map for training by each of the pixels, to thereby generate the pixel-wise feature maps for training per each of the proposals for training, and has instructed the classifying layer to generate the object class information for training on each of the proposals for training by using the pixel-wise feature maps for training per each of the proposals for training, (3-2) has instructed a detecting layer to generate object detection information for training corresponding to the objects for training in the training image by referring to the object class information for training and the pixel-wise feature maps for training per each of the proposals for training, and (3-3) has instructed a detection loss layer to calculate one or more object detection losses by referring to the object detection information for training and its corresponding GT, to thereby learn at least part of parameters of the second 1×
1 convolutional layer, the first 1×
1 convolutional layer, and the convolutional layers by backpropagating the object detection losses;
configured to execute the instructions to;
perform processes of (I) (i) instructing the convolutional layers to generate at least one initial feature map for testing by applying the convolution operations to at least one test image, (ii) instructing the RPN to generate one or more proposals for testing corresponding to each of one or more objects for testing in the test image by using the initial feature map for testing, and (iii) (iii-1) instructing the pooling layer to apply the pooling operations to each region, corresponding to each of the proposals for testing, on the initial feature map for testing, to thereby generate pooled feature maps for testing per each of the proposals for testing, and instructing a first transposing layer to concatenate each of pixels, per each of the proposals for testing, in each of corresponding same locations on the pooled feature maps for testing per each of the proposals for testing, to thereby generate an integrated feature map for testing, or (iii-2) instructing the pooling layer to apply the pooling operations to each region, corresponding to each of the proposals for testing, on the initial feature map for testing, to thereby generate the pooled feature maps for testing per each of the proposals for testing, and instructing the pooling layer to concatenate each of the pixels, per each of the proposals for testing, in each of the corresponding same locations on the pooled feature maps for testing per each of the proposals for testing, to thereby generate the integrated feature map for testing, (II) instructing the first 1×
1 convolutional layer to apply the 1×
1 convolution operation to the integrated feature map for testing, to thereby generate a first adjusted feature map for testing whose volume is adjusted, and instructing the second 1×
1 convolutional layer to apply the 1×
1 convolution operation to the first adjusted feature map for testing, to thereby generate a second adjusted feature map for testing whose volume is adjusted, and (III) (III-1) (i) instructing the second transposing layer to divide the second adjusted feature map for testing by each of the pixels, to thereby generate pixel-wise feature maps for testing per each of the proposals for testing, and instructing the classifying layer to generate object class information for testing on each of the proposals for testing by using the pixel-wise feature maps for testing per each of the proposals for testing, or (ii) instructing the classifying layer to divide the second adjusted feature map for testing by each of the pixels, to thereby generate the pixel-wise feature maps for testing per each of the proposals for testing, and instructing the classifying layer to generate the object class information for testing on each of the proposals for testing by using the pixel-wise feature maps for testing per each of the proposals for testing, and (III-2) instructing the detecting layer to generate object detection information for testing corresponding to the objects for testing in the test image by referring to the object class information for testing and the pixel-wise feature maps for testing per each of the proposals for testing. - View Dependent Claims (23, 24, 25, 26, 27, 28)
-
Specification