Method and device of neural network operations using a grid generator for converting modes according to classes of areas to satisfy level 4 of autonomous vehicles

• US 10,311,336 B1
• Filed: 01/22/2019
• Issued: 06/04/2019
• Est. Priority Date: 01/22/2019
• Status: Active Grant

• Current Assignee: Stradvision, Inc.
• Original Assignee: Stradvision, Inc.
• Inventors: Kim, Kye-Hyeon, Kim, Yongjoong, Kim, Insu, Kim, Hak-Kyoung, Nam, Woonhyun, Boo, SukHoon, Sung, Myungchul, Yeo, Donghun, Ryu, Wooju, Jang, Taewoong, Jeong, Kyungjoong, Je, Hongmo, Cho, Hojin
• Primary Examiner(s): Seth, Manav
• Application Number:
  US16/254,525
• Time in Patent Office: 133 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2163 Partitioning the feature space
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A method of neural network operations by using a grid generator is provided for converting modes according to classes of areas to satisfy level 4 of autonomous vehicles. The method includes steps of: (a) a computing device instructing a pair detector to acquire information on locations and classes of pairs for testing by detecting the pairs for testing; (b) the computing device instructing the grid generator to generate section information by referring to the information on the locations of the pairs for testing; (c) the computing device instructing a neural network to determine parameters for testing by referring to parameters for training which have been learned by using information on pairs for training; and (d) the computing device instructing the neural network to apply the neural network operations to a test image by using each of the parameters for testing to thereby generate one or more neural network outputs.

Claims

1. A method for neural network operations by using a grid generator, comprising steps of:

(a) a computing device, if a test image is acquired, instructing a pair detector to acquire information on locations and classes of one or more pairs for testing by detecting the pairs for testing, including objects for testing and non-objects for testing corresponding to the objects for testing, on the test image;

(b) the computing device instructing the grid generator to generate section information, which includes information on a plurality of subsections in the test image, by referring to the information on the locations of the pairs for testing;

(c) the computing device instructing a neural network to determine parameters for testing, to be used for applying the neural network operations to each of at least part of the subsections including each of the pairs for testing, by referring to parameters for training which have been learned by using information on pairs for training whose corresponding information on the classes is same as or similar to that on the pairs for testing; and

(d) the computing device instructing the neural network to apply the neural network operations to the test image by using each of the parameters for testing, corresponding to each of said at least part of the subsections, to thereby generate one or more neural network outputs.

2. The method of claim 1, wherein, at the step of (b), the grid generator divides the test image by using a dynamic template, and wherein the dynamic template is provided by adjusting one or more boundaries included therein, such that (i) at least one of the pairs for testing is included in each of said at least part of the subsections of the dynamic template.

3. The method of claim 2, wherein the grid generator manages information on the dynamic template, and wherein the dynamic template includes at least part of (i) at least one first boundary in a first direction and (ii) at least one second boundary in a second direction, some of which are adjustable.

4. The method of claim 2, wherein the dynamic template includes row groups and one or more column groups, wherein the row groups have an upper row group, a middle row group including at least part of the column groups, and a lower row group, and wherein (i) at least one of the pairs for testing is included in each of said at least part of the subsections formed by the row groups and the column groups.

5. The method of claim 1, before the step (a), comprising a step of:

(a0) the computing device instructing the neural network to learn the parameters for training by using information on each of the pairs for training included in one or more training images.

6. The method of claim 5, wherein, at the step of (a0), the neural network learns the parameters for training by using information on each of the pairs for training included in each of at least part of partial areas, from a first partial area to a k-th partial area, included in the training images, and store information on classes of the pairs for training coupled with their corresponding parameters for training.

7. The method of claim 6, wherein, on condition that there are first location information to k-th location information corresponding to the first partial area to the k-th partial area in each of the training images, the training images, in which respective relative variations of at least part of the first location information to the k-th location information are equal to or smaller than a predetermined threshold value, are selected.

8. The method of claim 6, wherein, at the step of (c), (i) on condition that area of a specific object for testing is larger than that of a specific non-object for testing, which is included in a specific subsection along with the specific object for testing, the computing device determines a specific pair for testing, including the specific non-object for testing and the specific object for testing, as having first information on classes, and instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one first specific parameter for training among the parameters for training and (ii) on condition that the area of the specific object for testing is same as or smaller than that of the specific non-object, the computing device determines a specific pair for testing, including the specific non-object for testing and the specific object for testing, as having second class information on classes and instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one second specific parameter for training among the parameters for training.

9. The method of claim 1, wherein, at the step of (d), the computing device instructs the neural network to generate the neural network outputs by applying one or more convolutional operations and one or more deconvolutional operations to the test image with the parameters for testing.

10. The method of claim 9, wherein the test image represents road driving circumstances, the objects for testing represent objects existing on a road, and the neural network outputs are used for autonomous driving.

11. A computing device for neural network operations by using a grid generator, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to;

perform processes of (I) instructing a pair detector to acquire information on locations and classes of one or more pairs for testing by detecting the pairs for testing, including objects for testing and non-objects for testing corresponding to the objects for testing, on a test image, (II) instructing the grid generator to generate section information, which includes information on a plurality of subsections in the test image, by referring to the information on the locations of the pairs for testing, (III) instructing a neural network to determine parameters for testing, to be used for applying the neural network operations to each of at least part of the subsections including each of the pairs for testing, by referring to parameters for training which have been learned by using information on pairs for training whose corresponding information on the classes is same as or similar to that on the pairs for testing, and (IV) instructing the neural network to apply the neural network operations to the test image by using each of the parameters for testing, corresponding to each of said at least part of the subsections, to thereby generate one or more neural network outputs.

12. The computing device of claim 11, wherein, at the process of (II), the grid generator divides the test image by using a dynamic template, and wherein the dynamic template is provided by adjusting one or more boundaries included therein, such that (i) at least one of the pairs for testing is included in each of said at least part of the subsections of the dynamic template.

13. The computing device of claim 12, wherein the grid generator manages information on the dynamic template, and wherein the dynamic template includes at least part of (i) at least one first boundary in a first direction and (ii) at least one second boundary in a second direction, some of which are adjustable.

14. The computing device of claim 12, wherein the dynamic template includes row groups and one or more column groups, wherein the row groups have an upper row group, a middle row group including at least part of the column groups, and a lower row group, and wherein (i) at least one of the pairs for testing is included in each of said at least part of the subsections formed by the row groups and the column groups.

15. The computing device of claim 11, before the process (I), wherein the processor further performs a process of:

(0) instructing the neural network to learn the parameters for training by using information on each of the pairs for training included in one or more training images.

16. The computing device of claim 15, wherein, at the process of (0), the neural network learns the parameters for training by using information on each of the pairs for training included in each of at least part of partial areas, from a first partial area to a k-th partial area, included in the training images, and store information on classes of the pairs for training coupled with their corresponding parameters for training.

17. The computing device of claim 16, wherein, on condition that there are first location information to k-th location information corresponding to the first partial area to the k-th partial area in each of the training images, the training images, in which respective relative variations of at least part of the first location information to the k-th location information are equal to or smaller than a predetermined threshold value, are selected.

18. The computing device of claim 16, wherein, at the process of (III), (i) on condition that area of a specific object for testing is larger than that of a specific non-object for testing, which is included in a specific subsection along with the specific object for testing, the processor determines a specific pair for testing, including the specific non-object for testing and the specific object for testing, as having first information on classes, and instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one first specific parameter for training among the parameters for training and (ii) on condition that the area of the specific object for testing is same as or smaller than that of the specific non-object, the processor determines a specific pair for testing, including the specific non-object for testing and the specific object for testing, as having second class information on classes and instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one second specific parameter for training among the parameters for training.

19. The computing device of claim 11, wherein, at the process of (IV), the processor instructs the neural network to generate the neural network outputs by applying one or more convolutional operations and one or more deconvolutional operations to the test image with the parameters for testing.

20. The computing device of claim 19, wherein the test image represents road driving circumstances, the objects for testing represent objects existing on a road, and the neural network outputs are used for autonomous driving.