Learning method and learning device for pooling ROI by using masking parameters to be used for mobile devices or compact networks via hardware optimization, and testing method and testing device using the same

• US 10,325,179 B1
• Filed: 01/23/2019
• Issued: 06/18/2019
• Est. Priority Date: 01/23/2019
• Status: Active Grant

• Current Assignee: Stradvision, Inc.
• Original Assignee: Stradvision, Inc.
• Inventors: Kim, Kye-Hyeon, Kim, Yongjoong, Kim, Insu, Kim, Hak-Kyoung, Nam, Woonhyun, Boo, SukHoon, Sung, Myungchul, Yeo, Donghun, Ryu, Wooju, Jang, Taewoong, Jeong, Kyungjoong, Je, Hongmo, Cho, Hojin
• Primary Examiner(s): Seth, Manav
• Application Number:
  US16/254,982
• Time in Patent Office: 146 Days
• CPC Class Codes: G06F 18/2115 by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2163 Partitioning the feature space
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: H04N 1/64 Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor H04N19/00 takes precedence

Abstract

A method for pooling at least one ROI by using one or more masking parameters is provided. The method is applicable to mobile devices, compact networks, and the like via hardware optimization. The method includes steps of: (a) a computing device, if an input image is acquired, instructing a convolutional layer of a CNN to generate a feature map corresponding to the input image; (b) the computing device instructing an RPN of the CNN to determine the ROI corresponding to at least one object included in the input image by using the feature map; (c) the computing device instructing an ROI pooling layer of the CNN to apply each of pooling operations correspondingly to each of sub-regions in the ROI by referring to each of the masking parameters corresponding to each of the pooling operations, to thereby generate a masked pooled feature map.

Claims

1. A method for pooling at least one ROI by using one or more masking parameters, comprising steps of:

(a) a computing device, if an input image is acquired, instructing a convolutional layer of a CNN to generate a feature map corresponding to the input image;

(b) the computing device instructing an RPN of the CNN to determine the ROI corresponding to at least one object included in the input image by using the feature map; and

(c) the computing device instructing an ROI pooling layer of the CNN to apply each of pooling operations correspondingly to each of sub-regions in the ROI by referring to each of the masking parameters corresponding to each of the pooling operations, to thereby generate a masked pooled feature map.

2. The method of claim 1, wherein, each of the masking parameters is determined as either a first value or a second value, and

wherein, at the step of (c), the computing device instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions in the ROI by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map.

3. The method of claim 2, wherein, if a specific masking parameter among the masking parameters is the first value, the computing device instructs the ROI pooling layer not to apply its corresponding pooling operation to its corresponding sub-region in the ROI.

4. The method of claim 1, further comprising steps of:

(d) the computing device instructing an output layer of the CNN to generate CNN output values by applying neural network operations to the masked pooled feature map; and

(e) the computing device instructing a loss layer of the CNN to generate loss values by referring to the CNN output values and their corresponding GTs, and learning the masking parameters of the ROI pooling layer by backpropagating the loss values.

5. The method of claim 4, wherein, at the step of (e), the computing device further learns at least part of parameters included in the convolutional layer and the output layer by backpropagating the loss values.

6. The method of claim 1, wherein, each of the masking parameters is determined as either a first value or a second value,

wherein, at the step of (c), the computing device instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions in the ROI by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map,

wherein, at the step of (c), the computing device instructs the ROI pooling layer to output each of representative feature values, as each component of the masked pooled feature map, and

wherein each of the representative feature values is each of averages of feature values, included in each of the corresponding sub-regions in the ROI, whose corresponding masking parameters are the second value.

7. The method of claim 1, wherein, each of the masking parameters is determined as either a first value or a second value,

wherein, at the step of (c), the computing device instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions in the ROI by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map,

wherein, at the step of (c), the computing device instructs the ROI pooling layer to output each of representative feature values, as each component of the masked pooled feature map, and

wherein each of the representative feature values is each of largest ones among feature values, included in each of the corresponding sub-regions in the ROI, whose corresponding masking parameters are the second value.

8. A testing method for pooling at least one ROI by using one or more masking parameters, comprising steps of:

(a) on condition that (1) a computing device has instructed a convolutional layer of a CNN to generate a feature map for training corresponding to a training image, (2) the computing device has instructed an RPN of the CNN to determine the ROI for training corresponding to at least one object for training included in the training image by using the feature map for training, (3) the computing device has instructed an ROI pooling layer of the CNN to apply each of pooling operations correspondingly to each of sub-regions for training in the ROI for training by referring to each of the masking parameters corresponding to each of the pooling operations, to thereby generate a masked pooled feature map for training, (4) the computing device has instructed an output layer of the CNN to generate CNN output values for training by applying neural network operations to the masked pooled feature map for training, and (5) the computing device has instructed a loss layer of the CNN to generate loss values by referring to the CNN output values for training and their corresponding GTs, and learning the masking parameters of the ROI pooling layer by backpropagating the loss values;

a testing device, if a test image is acquired, instructing the convolutional layer of the CNN to generate a feature map for testing corresponding to the test image;

(b) the testing device instructing the RPN of the CNN to determine the ROI for testing corresponding to at least one object for testing included in the test image by using the feature map for testing;

(c) the testing device instructing the ROI pooling layer of the CNN to apply each of the pooling operations correspondingly to each of sub-regions for testing in the ROI for testing by referring to each of the masking parameters corresponding to each of the pooling operations, to thereby generate a masked pooled feature map for testing; and

(d) the testing device instructing the output layer of the CNN to generate CNN output values for testing by applying the neural network operations to the masked pooled feature map for testing.

9. The testing method of claim 8, wherein, each of the masking parameters is determined as either a first value or a second value, and

wherein, at the step of (c), the testing device instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions for testing in the ROI for testing by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map for testing.

10. The method of claim 9, wherein, if a specific masking parameter among the masking parameters is the first value, the computing device instructs the ROI pooling layer not to apply its corresponding pooling operation to its corresponding sub-region for testing in the ROI for testing.

11. The method of claim 8, wherein, each of the masking parameters is determined as either a first value or a second value,

wherein, at the step of (c), the testing device instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions for testing in the ROI for testing by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map for testing,

wherein, at the step of (c), the testing device instructs the ROI pooling layer to output each of representative feature values for testing, as each component of the masked pooled feature map for testing, and

wherein each of the representative feature values for testing is each of averages of feature values for testing, included in each of the corresponding sub-regions for testing in the ROI for testing, whose corresponding masking parameters are the second value.

12. The method of claim 8, wherein, each of the masking parameters is determined as either a first value or a second value,

wherein, at the step of (c), the testing device instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions for testing in the ROI for testing by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map for testing,

wherein, at the step of (c), the testing device instructs the ROI pooling layer to output each of representative feature values for testing, as each component of the masked pooled feature map for testing, and

wherein each of the representative feature values for testing is each of largest ones among feature values for testing, included in each of the corresponding sub-regions for testing in the ROI for testing, whose corresponding masking parameters are the second value.

13. A computing device for pooling at least one ROI by using one or more masking parameters, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to;

perform processes of (I) instructing a convolutional layer of a CNN to generate a feature map corresponding to an input image;

(II) instructing an RPN of the CNN to determine the ROI corresponding to at least one object included in the input image by using the feature map; and

(III) instructing an ROI pooling layer of the CNN to apply each of pooling operations correspondingly to each of sub-regions in the ROI by referring to each of the masking parameters corresponding to each of the pooling operations, to thereby generate a masked pooled feature map.

14. The computing device of claim 13, wherein, each of the masking parameters is determined as either a first value or a second value, and

wherein, at the process of (III), the processor instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions in the ROI by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map.

15. The computing device of claim 14, wherein, if a specific masking parameter among the masking parameters is the first value, the processor instructs the ROI pooling layer not to apply its corresponding pooling operation to its corresponding sub-region in the ROI.

16. The computing device of claim 13, wherein the processor further performs processes of:

(IV) instructing an output layer of the CNN to generate CNN output values by applying neural network operations to the masked pooled feature map;

(V) instructing a loss layer of the CNN to generate loss values by referring to the CNN output values and their corresponding GTs, and learning the masking parameters of the ROI pooling layer by backpropagating the loss values.

17. The computing device of claim 16, wherein, at the process of (V), the processor further learns at least part of parameters included in the convolutional layer and the output layer by backpropagating the loss values.

18. The computing device of claim 13, wherein, each of the masking parameters is determined as either a first value or a second value,

wherein, at the process of (III), the processor instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions in the ROI by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map,

wherein, at the process of (III), the processor instructs the ROI pooling layer to output each of representative feature values, as each component of the masked pooled feature map, and

wherein each of the representative feature values is each of averages of feature values, included in each of the corresponding sub-regions in the ROI, whose corresponding masking parameters are the second value.

19. The computing device of claim 13, wherein, each of the masking parameters is determined as either a first value or a second value,

wherein, at the process of (III), the processor instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions in the ROI by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map,

wherein, at the process of (III), the processor instructs the ROI pooling layer to output each of representative feature values, as each component of the masked pooled feature map, and

wherein each of the representative feature values is each of largest ones among feature values, included in each of the corresponding sub-regions in the ROI, whose corresponding masking parameters are the second value.

20. A testing device for pooling at least one ROI by using one or more masking parameters, comprising:

at least one memory that stores instructions; and

at least one processor, on condition that (1) a computing device has instructed a convolutional layer of a CNN to generate a feature map for training corresponding to a training image, (2) the computing device has instructed an RPN of the CNN to determine the ROI for training corresponding to at least one object for training included in the training image by using the feature map for training, (3) the computing device has instructed an ROI pooling layer of the CNN to apply each of pooling operations correspondingly to each of sub-regions for training in the ROI for training by referring to each of the masking parameters corresponding to each of the pooling operations, to thereby generate a masked pooled feature map for training, (4) the computing device has instructed an output layer of the CNN to generate CNN output values for training by applying neural network operations to the masked pooled feature map for training, and (5) the computing device has instructed a loss layer of the CNN to generate loss values by referring to the CNN output values for training and their corresponding GTs, and learning the masking parameters of the ROI pooling layer by backpropagating;

the loss values;

configured to execute the instructions to;

perform processes of (I) instructing the convolutional layer of the CNN to generate a feature map for testing corresponding to a test image;

(II) instructing the RPN of the CNN to determine the ROI for testing corresponding to at least one object for testing included in the test image by using the feature map for testing;

(III) instructing the ROI pooling layer of the CNN to apply each of the pooling operations correspondingly to each of sub-regions for testing in the ROI for testing by referring to each of the masking parameters corresponding to each of the pooling operations, to thereby generate a masked pooled feature map for testing; and

(IV) instructing the output layer of the CNN to generate CNN output values for testing by applying the neural network operations to the masked pooled feature map for testing.

21. The testing device of claim 20, wherein, each of the masking parameters is determined as either a first value or a second value, and

wherein, at the process of (III), the processor instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions for testing in the ROI for testing by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map for testing.

22. The testing device of claim 21, wherein, if a specific masking parameter among the masking parameters is the first value, the processor instructs the ROI pooling layer not to apply its corresponding pooling operation to its corresponding sub-region for testing in the ROI for testing.

23. The testing device of claim 20, wherein, each of the masking parameters is determined as either a first value or a second value,

wherein, at the process of (III), the processor instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions for testing in the ROI for testing by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map for testing,

wherein, at the process of (III), the processor instructs the ROI pooling layer to output each of representative feature values for testing, as each component of the masked pooled feature map for testing, and

wherein each of the representative feature values for testing is each of averages of feature values for testing, included in each of the corresponding sub-regions for testing in the ROI for testing, whose corresponding masking parameters are the second value.

24. The testing device of claim 20, wherein, each of the masking parameters is determined as either a first value or a second value,

wherein, at the process of (III), the processor instructs the ROI pooling layer to apply each of the pooling operations correspondingly to each of the sub-regions for testing in the ROI for testing by referring to information on whether each of the masking parameters is the first value or the second value, to thereby generate the masked pooled feature map for testing,

wherein, at the process of (III), the processor instructs the ROI pooling layer to output each of representative feature values for testing, as each component of the masked pooled feature map for testing, and

wherein each of the representative feature values for testing is each of largest ones among feature values for testing, included in each of the corresponding sub-regions for testing in the ROI for testing, whose corresponding masking parameters are the second value.