Learning method and learning device for adjusting parameters of CNN in which residual networks are provided for meta learning, and testing method and testing device using the same

• US 10,496,899 B1
• Filed: 01/25/2019
• Issued: 12/03/2019
• Est. Priority Date: 01/25/2019
• Status: Active Grant

• Current Assignee: Stradvision, Inc.
• Original Assignee: Stradvision, Inc.
• Inventors: Kim, Kye-Hyeon, Kim, Yongjoong, Kim, Insu, Kim, Hak-Kyoung, Nam, Woonhyun, Boo, SukHoon, Sung, Myungchul, Yeo, Donghun, Ryu, Wooju, Jang, Taewoong, Jeong, Kyungjoong, Je, Hongmo, Cho, Hojin
• Primary Examiner(s): Strege, John B
• Application Number:
  US16/258,047
• Time in Patent Office: 312 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

A CNN-based method for meta learning, i.e., learning to learning, by using a learning device including convolutional layers capable of applying convolution operations to an image or its corresponding input feature maps to generate output feature maps, and residual networks capable of feed-forwarding the image or its corresponding input feature maps to next convolutional layer through bypassing the convolutional layers or its sub-convolutional layers is provided. The CNN-based method includes steps of: the learning device (a) selecting a specific residual network to be dropped out among the residual networks; (b) feeding the image into a transformed CNN where the specific residual network is dropped out, and outputting a CNN output; and (c) calculating losses by using the CNN output and its corresponding GT, and adjusting parameters of the transformed CNN. Further, the CNN-based method can be also applied to layer-wise dropout, stochastic ensemble, virtual driving, and the like.

Claims

1. A convolutional neural network (CNN)-based learning method by using a learning device including (i) one or more convolutional layers capable of applying one or more convolution operations to an input image or its corresponding one or more encoded input feature maps to thereby sequentially generate one or more encoded output feature maps and (ii) one or more residual networks capable of feed-forwarding the input image or its corresponding encoded input feature maps to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of sub-convolutional layers included in at least one of the convolutional layers, comprising steps of:

(a) the learning device, if the input image is acquired, performing a process of randomly selecting a specific residual network to be dropped out among the residual networks;

(b) the learning device (i) feeding the input image into at least one transformed CNN in which the specific residual network is dropped out, to thereby generate adjusted encoded output feature maps and (ii) generating a CNN output by using the adjusted encoded output feature maps; and

(c) the learning device calculating one or more losses by using the CNN output and its corresponding GT and adjusting at least one parameter of the transformed CNN by backpropagating the losses;

wherein, assuming that the number of the convolutional layers is N and each of the N convolutional layers has L sub-convolutional layers, each of the N convolutional layers includes M sub-residual networks having each different bypassing route for bypassing at least one of the L sub-convolutional layers; and

wherein, at the step of (a), at least one specific sub-residual network to be dropped out is randomly selected among the N×

M sub-residual networks.

2. The CNN-based learning method of claim 1, wherein the steps of (a) to (c) are repeated for the same input image.

3. A convolutional neural network (CNN)-based learning method by using a learning device including (i) one or more convolutional layers capable of applying one or more convolution operations to an input image or its corresponding one or more encoded input feature maps to thereby sequentially generate one or more encoded output feature maps and (ii) one or more residual networks capable of feed-forwarding the input image or its corresponding encoded input feature maps to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of sub-convolutional layers included in at least one of the convolutional layers, comprising steps of:

(a) the learning device, if the input image is acquired, performing a process of randomly selecting a specific residual network to be dropped out among the residual networks;

(b) the learning device (i) feeding the input image into at least one transformed CNN in which the specific residual network is dropped out, to thereby generate adjusted encoded output feature maps and (ii) generating a CNN output by using the adjusted encoded output feature maps; and

(c) the learning device calculating one or more losses by using the CNN output and its corresponding GT and adjusting at least one parameter of the transformed CNN by backpropagating the losses;

wherein the learning device further includes;

(i) one or more deconvolutional layers capable of applying one or more deconvolutional operations to at least part of the adjusted encoded output feature maps or its corresponding adjusted decoded input feature maps to thereby sequentially generate one or more adjusted decoded output feature maps and (ii) one or more intermediate layers, which are located between at least one of the convolutional layers and at least one of the deconvolutional layers, capable of applying one or more convolution operations to one or more inputs fed thereto and then feeding one or more outputs therefrom into at least one of the deconvolutional layers,

wherein, at the step of (a), the learning device performs a process of randomly selecting the specific residual network and at least one specific intermediate layer to be dropped out among the residual networks and the intermediate layers, and

wherein, at the step of (b), the learning device controls the transformed CNN such that the specific residual network and the specific intermediate layer are dropped out.

4. The CNN-based learning method of claim 3, wherein the learning device further includes additional residual networks capable of feed-forwarding the adjusted decoded input feature maps to its corresponding next deconvolutional layer through bypassing at least one of the plurality of deconvolutional layers,

wherein, assuming that (i) each of the N convolutional layers includes L sub-convolutional layers and (ii) each of the N deconvolutional layers includes L sub-deconvolutional layers, and the number of the intermediate layers is K, (1) each of the N convolutional layers includes M sub-residual networks having each different bypassing route through which at least one of the L sub-convolutional layers is bypassed and (2) each of the N deconvolutional layers includes M sub-residual networks having each different bypassing route through which at least one of the L sub-deconvolutional layers is bypassed, and

wherein, at the step of (a), at least one specific intermediate layer and at least one specific sub-residual network to be dropped out are randomly selected among the K intermediate layers and the 2×

N×

M sub-residual networks, respectively.

5. The CNN-based learning method of claim 3, wherein at least one of the intermediate layers is a dilated convolutional layer.

6. The CNN-based learning method of claim 5, wherein, at the step of (c), the learning device backpropagates the losses to thereby adjust at least one parameter of the deconvolutional layers, the intermediate layers, and the convolutional layers.

7. A method for testing a test image based on a convolutional neural network (CNN), comprising steps of:

(a) on condition that, assuming that a learning device includes (i) one or more convolutional layers capable of applying one or more convolution operations to a training image or its corresponding one or more encoded input feature maps for training to thereby sequentially generate one or more encoded output feature maps for training and (ii) one or more residual networks capable of feed-forwarding the training image or its corresponding encoded input feature maps for training to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of sub-convolutional layers included in at least one of the convolutional layers, (1) the learning device has performed a process of randomly selecting a specific residual network to be dropped out among the residual networks, (2) the learning device (i) has fed the training image into at least one transformed CNN in which the specific residual network is dropped out, to thereby generate adjusted encoded output feature maps for training and (ii) has generated a CNN output for training by using the adjusted encoded output feature maps for training, and (3) the learning device has calculated one or more losses by using the CNN output for training and its corresponding GT and has adjusted at least one parameter of the transformed CNN by backpropagating the losses, a testing device acquiring the test image; and

(b) the testing device generating one or more encoded output feature maps for testing based on the test image and generating a CNN output for testing by using the encoded output feature maps for testing;

wherein, assuming that the number of the convolutional layers is N and each of the N convolutional layers has L sub-convolutional layers, each of the N convolutional layers includes M sub-residual networks having each different bypassing route for bypassing at least one of the L sub-convolutional layers, and

wherein, at the process of (1), at least one specific sub-residual network to be dropped out is randomly selected among the N×

M sub-residual networks.

8. The method of claim 7, wherein the processes of (1) to (3) are repeated for the same training image.

9. A method for testing a test image based on a convolutional neural network (CNN), comprising steps of:

(a) on condition that, assuming that a learning device includes (i) one or more convolutional layers capable of applying one or more convolution operations to a training image or its corresponding one or more encoded input feature maps for training to thereby sequentially generate one or more encoded output feature maps for training and (ii) one or more residual networks capable of feed-forwarding the training image or its corresponding encoded input feature maps for training to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of sub-convolutional layers included in at least one of the convolutional layers, (1) the learning device has performed a process of randomly selecting a specific residual network to be dropped out among the residual networks, (2) the learning device (i) has fed the training image into at least one transformed CNN in which the specific residual network is dropped out, to thereby generate adjusted encoded output feature maps for training and (ii) has generated a CNN output for training by using the adjusted encoded output feature maps for training, and (3) the learning device has calculated one or more losses by using the CNN output for training and its corresponding GT and has adjusted at least one parameter of the transformed CNN by backpropagating the losses, a testing device acquiring the test image; and

(b) the testing device generating one or more encoded output feature maps for testing based on the test image and generating a CNN output for testing by using the encoded output feature maps for testing;

wherein, on condition that the test device includes (i) the convolutional layers capable of applying the convolution operations to the test image or its corresponding one or more encoded input feature maps for testing to thereby sequentially generate the encoded output feature maps for testing and (ii) the residual networks capable of feed-forwarding the test image or its corresponding encoded input feature maps for testing to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of the sub-convolutional layers included in at least one of the convolutional layers, at the step of (b), the testing device (b-i) performs a process of randomly selecting a certain residual network to be dropped out among the residual networks, and (b-ii) feeds the test image into the transformed CNN in which the certain residual network is dropped out, to thereby generate adjusted encoded output feature maps for testing, and then generates the CNN output for testing by using the adjusted encoded output feature maps for testing.

10. The method of claim 9, wherein the testing device further includes:

(i) one or more deconvolutional layers capable of applying one or more deconvolutional operations to at least part of the adjusted encoded output feature maps for testing or its corresponding adjusted decoded input feature maps for testing to thereby sequentially generate one or more adjusted decoded output feature maps for testing and (ii) one or more intermediate layers, which are located between at least one of the convolutional layers and at least one of the deconvolutional layers, capable of applying one or more convolution operations to one or more inputs fed thereto and then feeding one or more outputs therefrom into at least one of the deconvolutional layers,

wherein, at the step of (b-i), the testing device performs a process of randomly selecting the certain residual network and at least one certain intermediate layer to be dropped out among the residual networks and the intermediate layers, and

wherein, at the step of (b-ii), the testing device controls the transformed CNN such that the certain residual network and the certain intermediate layer are dropped out.

11. The method of claim 10, wherein the testing device further includes additional residual networks capable of feed-forwarding the adjusted decoded input feature maps for testing to its corresponding next deconvolutional layer through bypassing at least one of the plurality of deconvolutional layers,

wherein, assuming that (i) each of the N convolutional layers includes L sub-convolutional layers and (ii) each of the N deconvolutional layers includes L sub-deconvolutional layers, and the number of the intermediate layers is K, (1) each of the N convolutional layers includes M sub-residual networks having each different bypassing route through which at least one of the L sub-convolutional layers is bypassed and (2) each of the N deconvolutional layers includes M sub-residual networks having each different bypassing route through which at least one of the L sub-deconvolutional layers is bypassed, and

wherein, at the step of (b-i), at least one certain intermediate layer and at least one certain sub-residual network to be dropped out are randomly selected among the K intermediate layers and the 2×

N×

M sub-residual networks, respectively.

12. The method of claim 10, wherein at least one of the intermediate layers is a dilated convolutional layer.

13. The method of claim 11, wherein, at the process of (3), the learning device has backpropagated the losses to thereby adjust at least one parameter of the deconvolutional layers, the intermediate layers, and the convolutional layers.

14. A learning device in which (i) one or more convolutional layers capable of applying one or more convolution operations to an input image or its corresponding one or more encoded input feature maps to thereby sequentially generate one or more encoded output feature maps and (ii) one or more residual networks capable of feed-forwarding the input image or its corresponding encoded input feature maps to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of sub-convolutional layers included in at least one of the convolutional layers are included, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to;

perform processes of (I) randomly selecting a specific residual network to be dropped out among the residual networks, (II) (i) feeding the input image into at least one transformed CNN in which the specific residual network is dropped out, to thereby generate adjusted encoded output feature maps and (ii) generating a CNN output by using the adjusted encoded output feature maps, and (III) calculating one or more losses by using the CNN output and its corresponding GT and adjusting at least one parameter of the transformed CNN by backpropagating the losses;

wherein, assuming that the number of the convolutional layers is N and each of the N convolutional layers has L sub-convolutional layers, each of the N convolutional layers includes M sub-residual networks having each different bypassing route for bypassing at least one of the L sub-convolutional layers, and

wherein, at the process of (I), at least one specific sub-residual network to be dropped out is randomly selected among the N×

M sub-residual networks.

15. The learning device of claim 14, wherein the processes of (I) to (III) are repeated for the same input image.

16. A learning device in which (i) one or more convolutional layers capable of applying one or more convolution operations to an input image or its corresponding one or more encoded input feature maps to thereby sequentially generate one or more encoded output feature maps and (ii) one or more residual networks capable of feed-forwarding the input image or its corresponding encoded input feature maps to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of sub-convolutional layers included in at least one of the convolutional layers are included, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to;

perform processes of (I) randomly selecting a specific residual network to be dropped out among the residual networks, (II) (i) feeding the input image into at least one transformed CNN in which the specific residual network is dropped out, to thereby generate adjusted encoded output feature maps and (ii) generating a CNN output by using the adjusted encoded output feature maps, and (III) calculating one or more losses by using the CNN output and its corresponding GT and adjusting at least one parameter of the transformed CNN by backpropagating the losses;

wherein the learning device further includes;

(i) one or more deconvolutional layers capable of applying one or more deconvolutional operations to at least part of the adjusted encoded output feature maps or its corresponding adjusted decoded input feature maps to thereby sequentially generate one or more adjusted decoded output feature maps and (ii) one or more intermediate layers, which are located between at least one of the convolutional layers and at least one of the deconvolutional layers, capable of applying one or more convolution operations to one or more inputs fed thereto and then feeding one or more outputs therefrom into at least one of the deconvolutional layers,

wherein, at the process of (I), the processor performs a process of randomly selecting the specific residual network and at least one specific intermediate layer to be dropped out among the residual networks and the intermediate layers, and

wherein, at the process of (II), the processor controls the transformed CNN such that the specific residual network and the specific intermediate layer are dropped out.

17. The learning device of claim 16, wherein the learning device further includes additional residual networks capable of feed-forwarding the adjusted decoded input feature maps to its corresponding next deconvolutional layer through bypassing at least one of the plurality of deconvolutional layers,

wherein, assuming that (i) each of the N convolutional layers includes L sub-convolutional layers and (ii) each of the N deconvolutional layers includes L sub-deconvolutional layers, and the number of the intermediate layers is K, (1) each of the N convolutional layers includes M sub-residual networks having each different bypassing route through which at least one of the L sub-convolutional layers is bypassed and (2) each of the N deconvolutional layers includes M sub-residual networks having each different bypassing route through which at least one of the L sub-deconvolutional layers is bypassed, and

wherein, at the process of (I), at least one specific intermediate layer and at least one specific sub-residual network to be dropped out are randomly selected among the K intermediate layers and the 2×

N×

M sub-residual networks, respectively.

18. The learning device of claim 16, wherein at least one of the intermediate layers is a dilated convolutional layer.

19. The learning device of claim 18, wherein, at the process of (III), the processor backpropagates the losses to thereby adjust at least one parameter of the deconvolutional layers, the intermediate layers, and the convolutional layers.

20. A testing device for testing a test image based on a convolutional neural network (CNN), comprising:

at least one memory that stores instructions; and

at least one processor, on condition that, assuming that a learning device includes (i) one or more convolutional layers capable of applying one or more convolution operations to a training image or its corresponding one or more encoded input feature maps for training to thereby sequentially generate one or more encoded output feature maps for training and (ii) one or more residual networks capable of feed-forwarding the training image or its corresponding encoded input feature maps for training to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of sub-convolutional layers included in at least one of the convolutional layers, (1) the learning device has performed a process of randomly selecting a specific residual network to be dropped out among the residual networks, (2) the learning device (i) has fed the training image into at least one transformed CNN in which the specific residual network is dropped out, to thereby generate adjusted encoded output feature maps for training and (ii) has generated a CNN output for training by using the adjusted encoded output feature maps for training, and (3) the learning device has calculated one or more losses by using the CNN output for training and its corresponding GT and has adjusted at least one parameter of the transformed CNN by backpropagating the losses;

configured to execute the instructions to;

perform processes of generating one or more encoded output feature maps for testing based on the test image and generating a CNN output for testing by using the encoded output feature maps for testing;

wherein, assuming that the number of the convolutional layers is N and each of the N convolutional layers has L sub-convolutional layers, each of the N convolutional layers includes M sub-residual networks having each different bypassing route for bypassing at least one of the L sub-convolutional layers, and

wherein, at the process of (1), at least one specific sub-residual network to be dropped out is randomly selected among the N×

M sub-residual networks.

21. The testing device of claim 20, wherein the processes of (1) to (3) are repeated for the same training image.

22. A testing device for testing a test image based on a convolutional neural network (CNN), comprising:

at least one memory that stores instructions; and

at least one processor, on condition that, assuming that a learning device includes (i) one or more convolutional layers capable of applying one or more convolution operations to a training image or its corresponding one or more encoded input feature maps for training to thereby sequentially generate one or more encoded output feature maps for training and (ii) one or more residual networks capable of feed-forwarding the training image or its corresponding encoded input feature maps for training to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of sub-convolutional layers included in at least one of the convolutional layers, (1) the learning device has performed a process of randomly selecting a specific residual network to be dropped out among the residual networks, (2) the learning device (i) has fed the training image into at least one transformed CNN in which the specific residual network is dropped out, to thereby generate adjusted encoded output feature maps for training and (ii) has generated a CNN output for training by using the adjusted encoded output feature maps for training, and (3) the learning device has calculated one or more losses by using the CNN output for training and its corresponding GT and has adjusted at least one parameter of the transformed CNN by backpropagating the losses;

configured to execute the instructions to;

perform processes of generating one or more encoded output feature maps for testing based on the test image and generating a CNN output for testing by using the encoded output feature maps for testing;

wherein, on condition that the test device includes (i) the convolutional layers capable of applying the convolution operations to the test image or its corresponding one or more encoded input feature maps for testing to thereby sequentially generate the encoded output feature maps for testing and (ii) the residual networks capable of feed-forwarding the test image or its corresponding encoded input feature maps for testing to its corresponding next convolutional layer through bypassing at least one of the convolutional layers or at least one of the sub-convolutional layers included in at least one of the convolutional layers, the processor performs processes of (I) randomly selecting a certain residual network to be dropped out among the residual networks, and (II) feeding the test image into the transformed CNN in which the certain residual network is dropped out, to thereby generate adjusted encoded output feature maps for testing, and then generating the CNN output for testing by using the adjusted encoded output feature maps for testing.

23. The testing device of claim 22, wherein the testing device further includes:

(i) one or more deconvolutional layers capable of applying one or more deconvolutional operations to at least part of the adjusted encoded output feature maps for testing or its corresponding adjusted decoded input feature maps for testing to thereby sequentially generate one or more adjusted decoded output feature maps for testing and (ii) one or more intermediate layers, which are located between at least one of the convolutional layers and at least one of the deconvolutional layers, capable of applying one or more convolution operations to one or more inputs fed thereto and then feeding one or more outputs therefrom into at least one of the deconvolutional layers,

wherein, at the process of (I), the processor randomly selects the certain residual network and at least one certain intermediate layer to be dropped out among the residual networks and the intermediate layers, and

wherein, at the process of (II), the processor controls the transformed CNN such that the certain residual network and the certain intermediate layer are dropped out.

24. The testing device of claim 23, wherein the testing device further includes additional residual networks capable of feed-forwarding the adjusted decoded input feature maps for testing to its corresponding next deconvolutional layer through bypassing at least one of the plurality of deconvolutional layers,

wherein, assuming that (i) each of the N convolutional layers includes L sub-convolutional layers and (ii) each of the N deconvolutional layers includes L sub-deconvolutional layers, and the number of the intermediate layers is K, (1) each of the N convolutional layers includes M sub-residual networks having each different bypassing route through which at least one of the L sub-convolutional layers is bypassed and (2) each of the N deconvolutional layers includes M sub-residual networks having each different bypassing route through which at least one of the L sub-deconvolutional layers is bypassed, and

wherein, at the process of (I), at least one certain intermediate layer and at least one certain sub-residual network to be dropped out are randomly selected among the K intermediate layers and the 2×

N×

M sub-residual networks, respectively.

25. The testing device of claim 23, wherein at least one of the intermediate layers is a dilated convolutional layer.

26. The testing device of claim 24, wherein, at the process of (3), the learning device has backpropagated the losses to thereby adjust at least one parameter of the deconvolutional layers, the intermediate layers, and the convolutional layers.